State evaluation system and method based on multi-modal data dynamic monitoring

Abstract

The invention provides a state evaluation system and method based on multi-modal data dynamic monitoring. The system comprises a multi-modal data acquisition module (a high-temperature-resistant camera and a radar sensor), a data processing module (a visual identification all-in-one machine and a control all-in-one machine) and a dynamic regulation and control module. The visual identification all-in-one machine performs material blocking detection through a YOLOv10 model, the control all-in-one machine fuses visual and radar data, the state of a material layer is judged in real time, and the rotating speed of the feeder is automatically adjusted. The system is connected with all the modules through the industrial Ethernet, and efficient cooperation and information transmission are ensured. The method comprises the steps of data acquisition, preprocessing, material blocking detection model training, model deployment and real-time judgment, and can accurately identify a material blocking area and make a response according to material level changes. The detection accuracy and stability are improved through multi-modal data fusion, the stability of the production process is ensured through an automatic regulation and control function, the system reliability is further improved through an alarm mechanism, and the system has high adaptability and wide application prospects.

Description

Technical Field

[0001] The present invention relates to the field of monitoring, and in particular to a state assessment system and method based on dynamic monitoring of multimodal data. Background Art

[0002] In the modern steel industry, sintering is a crucial step in ironmaking. Its core goal is to produce high-quality sintered ore through a reasonable raw material ratio and uniform material bed distribution. However, in actual production, slow trolley speeds or poor raw material flow often lead to material accumulation in front of the baffle, resulting in uneven material bed distribution, and even causing equipment blockage and sintering quality issues.

[0003] Currently, the material distribution system of the sintering trolley mainly relies on manual adjustment: when the material layer is thin, the operator increases the feed rate by increasing the speed of the roller feeder; when the material layer is too high or blockage occurs, the operator reduces the speed of the roller feeder to reduce the feed rate. This manual adjustment method has the following significant problems: Inefficiency: Manual adjustments rely on the operator's experience and judgment, making it difficult to respond to changes in material thickness in real time, resulting in low production efficiency.

[0004] Poor reliability: Manual operation is easily affected by subjective factors, resulting in uneven distribution of material layers, which in turn affects the quality of sintered ore.

[0005] Safety issues: In complex industrial environments with high temperatures and high dust, manual adjustments increase the safety risks for operators.

[0006] Some control systems have implemented intelligent material distribution, but they primarily rely on a single sensor to monitor the material bed condition. Due to the complex high-temperature and high-dust environment of the sintering workshop, the detection effectiveness of a single sensor is often limited. For example, while radar sensors have strong anti-interference capabilities, their accuracy in detecting piled materials in complex backgrounds is limited. While cameras can provide high-resolution visual information, image quality is also affected by high temperatures and strong light conditions. Therefore, a single sensor cannot achieve high-precision real-time monitoring. Summary of the Invention

[0007] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a state assessment system and method based on dynamic monitoring of multimodal data.

[0008] To achieve the above objectives, the present invention adopts the following technical solutions: a state assessment system based on dynamic monitoring of multimodal data, the state assessment system includes Multimodal data acquisition module, including a high-temperature resistant camera and a radar sensor, used to collect material layer images and data at different material level heights respectively; The data processing module includes a visual recognition integrated machine and a control integrated machine; the visual recognition integrated machine runs a material blockage detection model to process image data, and the control integrated machine integrates visual signals and radar data to determine the state of the material layer; And the dynamic control module automatically adjusts the speed of the round roller feeder according to the state of the material layer.

[0009] Preferably, the radar sensors are arranged on both sides of the front end of the baffle, including a high-level laser radar arranged at the upper material position and a low-level laser radar arranged at the lower material position; the high-temperature resistant camera is installed on the upper side of the sintering trolley baffle.

[0010] Preferably, the data processing module and the multimodal data acquisition module are connected by network signals via industrial Ethernet; the high-temperature resistant camera and radar sensor are connected to the visual recognition all-in-one machine and the control all-in-one machine via industrial Ethernet; the visual recognition all-in-one machine receives the pre-processing of video data and the operation of the blockage recognition model and transmits the recognition results to the control all-in-one machine; the control all-in-one machine receives the visual blockage signal and the radar signal and fuses the two modal data, and adjusts the speed of the round roller feeder according to preset rules.

[0011] Preferably, the dynamic control module is connected to each device via industrial Ethernet.

[0012] Preferably, the control all-in-one machine triggers an alarm mechanism when the visual blockage is inconsistent with the radar data.

[0013] A method for a condition assessment system based on dynamic monitoring of multimodal data, the method specifically comprises the following steps: S1: Data collection and data preprocessing: Collect running video data of the sintering trolley under different working conditions and perform data preprocessing on the collected video data, including denoising, frame rate adjustment and image enhancement; S2: Data segmentation and labeling; Segment the preprocessed data according to time series or specific conditions, and label the segmented data; S3: Blockage detection model training: Use the YOLOv10 target detection model to detect material blockages; divide the labeled data in step S2 into material blockage areas and normal areas, use the labeled data for supervised learning, and then perform cross-validation, adjust hyperparameters, and optimize model performance; S4: Model deployment: Prune and quantize the model trained in step S3 to adapt to the performance of the actual hardware, and deploy the processed model on the visual recognition all-in-one machine; S5: Real-time judgment: When the visual detection model detects a blocking signal and the radar detects that the material level is higher than the upper material level, it is judged that the material level is too high; On the contrary, when the visual inspection does not detect the blocking signal and the radar detects that the material level is lower than the lower material level, it is judged that the material level is too low.

[0014] Compared with existing technologies, this invention offers the following advantages: By fusing visual signals with radar data, this system effectively overcomes the limitations of single sensors in complex operating conditions. Visual signals provide precise images of the material layer, while radar sensors accurately collect material level data at varying heights. The combination of these two ensures comprehensive monitoring of the material layer's status. This multimodal data fusion significantly improves the accuracy and stability of blockage detection.

[0015] Secondly, the system uses the YOLOv10 object detection model for blockage detection. By training and optimizing video data, it can quickly identify blockage areas and, combined with radar sensor data, determine the state of the material layer in real time. This model training and deployment method effectively improves the efficiency and accuracy of blockage detection and reduces the delays and errors associated with traditional manual inspections.

[0016] This system not only determines the material layer status in real time but also automatically adjusts the roller feeder's speed based on the results. This automatic control capability ensures the continuity and stability of the production process, effectively avoiding interruptions caused by material layer anomalies. Furthermore, when visual and radar data conflict, the integrated control system triggers an alarm mechanism, providing timely warnings and avoiding potential issues caused by data conflicts, further enhancing system reliability.

[0017] All data acquisition modules, visual recognition integrated devices, control integrated devices, and dynamic control modules within the system are connected via industrial Ethernet, ensuring efficient collaboration and information transmission between devices. This highly efficient data communication network significantly improves the system's real-time performance and responsiveness, enabling effective response to various emergencies in a dynamic production environment.

[0018] The system design of this invention boasts strong adaptability in both hardware and software. In particular, during model deployment, through pruning and quantization, the optimized model can adapt to the performance requirements of different hardware, enabling stable system operation under diverse operating conditions. Furthermore, the system boasts excellent scalability, allowing for the addition of additional sensors or algorithmic improvements as needed to further enhance performance.

[0019] The present invention is based on a state assessment system for dynamic monitoring of multimodal data. By combining real-time monitoring, intelligent analysis, automatic control and early warning mechanisms, it solves the accuracy, real-time and reliability problems of traditional monitoring systems in material layer state assessment. It has significant technical advantages and application prospects, and has broad application value, especially in the fields of industrial automation and intelligent manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the installation diagram of the system equipment of the present invention; Figure 2 This is a diagram of the network architecture of the system equipment of the present invention; Figure 3 The figure is a decision flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the present invention is described in detail below with reference to the embodiments.

[0022] Please refer to Figure 1 The present invention provides a state assessment system based on dynamic monitoring of multimodal data, the state assessment system includes The multimodal data acquisition module includes a high-temperature resistant camera and a radar sensor, which are used to collect material layer images and data at different material level heights respectively.

[0023] Cameras and radar sensors collect images of the material layer and material level data, respectively. Image data provides visual information about the material layer, while radar data provides information such as the distance and speed of objects. The combination of the two makes monitoring more comprehensive.

[0024] The use of high-temperature resistant cameras enables the system to work effectively in high-temperature environments, adapt to various extreme conditions, and ensure the accuracy and reliability of data.

[0025] The data processing module includes a visual recognition all-in-one machine and a control all-in-one machine; the visual recognition all-in-one machine runs a material blockage detection model to process image data, and the control all-in-one machine fuses visual signals with radar data and determines the state of the material layer.

[0026] The all-in-one visual recognition machine detects material blockages and can analyze image data in real time to identify and accurately determine if a blockage is present. Furthermore, the all-in-one control machine integrates visual information with radar data to provide a more comprehensive assessment of the material layer's condition.

[0027] The fusion of visual signals and radar data improves the system's ability to accurately judge the state of the material layer, reduces the blind spots that may exist in a single sensor, and enhances the stability and robustness of the system.

[0028] And the dynamic control module automatically adjusts the speed of the round roller feeder according to the state of the material layer.

[0029] The dynamic control module automatically adjusts the roller feeder's speed based on the material layer's condition. This allows for real-time response to changes in the material layer, preventing production stagnation and material blockages caused by unsatisfactory material conditions. This feature improves production efficiency and reduces equipment failures and the need for manual intervention.

[0030] Preferably, the radar sensors are arranged on both sides of the front end of the baffle, including a high-level laser radar arranged at the upper material position and a low-level laser radar arranged at the lower material position; the high-temperature resistant camera is installed on the upper side of the sintering trolley baffle.

[0031] High-level LiDAR and low-level LiDAR sensors, located at the top and bottom of the material pile, respectively, effectively monitor changes throughout the entire pile. This setup ensures real-time feedback on pile height and position, helping to improve operational accuracy and safety.

[0032] The high-temperature resistant camera is installed on the upper side of the sintering trolley baffle, effectively avoiding direct exposure to high temperatures. This allows the camera to continue to work stably in high-temperature environments, ensuring long-term and reliable operation of the system.

[0033] By setting up laser radar and high-temperature resistant cameras, the material flow conditions and material accumulation on the trolley can be monitored in real time to avoid material blockage or overflow, thereby reducing safety hazards and material waste during operations.

[0034] Real-time monitoring and detection can quickly identify problems and provide feedback, reducing human intervention and maintenance time, thereby improving the overall efficiency of the production line. Placing sensors and cameras in different locations not only increases data redundancy but also facilitates multi-dimensional monitoring of the environment under different operating conditions, enhancing the system's fault tolerance.

[0035] Preferably, the data processing module and the multimodal data acquisition module are connected by network signals via industrial Ethernet; the high-temperature resistant camera and radar sensor are connected to the visual recognition all-in-one machine and the control all-in-one machine via industrial Ethernet; the visual recognition all-in-one machine receives the pre-processing of video data and the operation of the blockage recognition model and transmits the recognition results to the control all-in-one machine; the control all-in-one machine receives the visual blockage signal and the radar signal and fuses the two modal data, and adjusts the speed of the round roller feeder according to preset rules.

[0036] Preferably, the dynamic control module is connected to each device via industrial Ethernet.

[0037] Industrial Ethernet provides high-bandwidth and low-latency data transmission, enabling rapid data exchange between devices and ensuring real-time performance. For example, the dynamic control module can quickly receive monitoring data from radar sensors and cameras and instantly adjust the speed of the roller feeder.

[0038] Industrial Ethernet supports communication standards and protocols across multiple devices, enabling seamless connectivity among diverse devices (such as all-in-one visual recognition devices, all-in-one control devices, sensors, actuators, and more), simplifying collaboration between them. This interconnectivity improves overall system coordination and operational efficiency.

[0039] Preferably, the control all-in-one machine triggers an alarm mechanism when the visual blockage is inconsistent with the radar data.

[0040] Vision and radar systems can be affected by various factors, such as lighting conditions, material properties, and sensor errors. By implementing a conflict detection mechanism that triggers an alarm when the data from the two systems is inconsistent, we can effectively avoid misjudgments caused by relying on a single sensor, thereby improving the accuracy and reliability of the overall system.

[0041] Vision systems and radar sensors have different monitoring methods and viewing angles, and they may provide different feedback in different situations. For example, radar may not accurately detect a material jam at certain angles, while a vision system can detect anomalies through image analysis. By triggering an alarm, the system can promptly identify potential problems and intervene, preventing minor issues from developing into major failures.

[0042] A method for a condition assessment system based on dynamic monitoring of multimodal data, the method specifically comprises the following steps: S1: Data collection and data preprocessing: The operation video data of the sintering trolley is collected under different working conditions, and the collected video data is preprocessed, including denoising, frame rate adjustment and image enhancement.

[0043] Video data of the sintering trolley's operation is collected under different operating conditions to ensure coverage of various possible working situations. The video data can reflect the state changes of the trolley during operation, including problems such as blockage.

[0044] Video data requires processing such as denoising, frame rate adjustment, and image enhancement. Denoising helps eliminate background noise and improve image quality; frame rate adjustment ensures data consistency across different scenarios; and image enhancement strengthens target features, making subsequent analysis more accurate.

[0045] S2: Data segmentation and labeling; The preprocessed data is segmented according to time series or specific conditions, and the segmented data is labeled.

[0046] By segmenting video data based on time series or specific conditions, long videos can be processed into multiple small segments, facilitating subsequent analysis and annotation. The segmented data can accurately reflect key moments in the trolley's operation, making it easier to track anomalies.

[0047] The segmented data is labeled to identify areas with blockage and areas without any other issues. This is the foundation for supervised learning. The quality of the labeled data directly impacts training results, so accuracy and consistency of the labeling are crucial.

[0048] S3: Blockage detection model training: The YOLOv10 target detection model is used for blockage detection. The labeled data in step S2 is divided into blockage areas and normal areas. The labeled data is used for supervised learning, and then cross-validation is performed to adjust hyperparameters and optimize model performance.

[0049] YOLOv10 can quickly and accurately identify objects in images. In this method, YOLOv10 is used to identify material blockages in a sintering trolley. By training on labeled data, the model can learn the characteristics of material blockages under different working conditions.

[0050] The model is trained using supervised learning and optimized using labeled data. During training, cross-validation is used to verify model performance and ensure good generalization. Hyperparameters (such as the learning rate and loss function) are adjusted to further improve the model's accuracy and robustness.

[0051] S4: Model deployment: The model trained in step S3 is pruned and quantized to adapt to the performance of the actual hardware, and the processed model is deployed on the visual recognition all-in-one machine.

[0052] To enable the model to run efficiently on actual hardware, the trained model is pruned (reducing unimportant neurons and connections) and quantized (reducing the model's storage requirements). This allows the model to adapt to the performance of the hardware device and reduces computing resource consumption.

[0053] The optimized model will be deployed on an actual all-in-one visual recognition machine, enabling real-time blockage detection. This deployment allows the system to be directly applied in production environments for condition monitoring and assessment.

[0054] S5: Real-time judgment: When the visual detection model detects a blocking signal and the radar detects that the material level is higher than the upper material level, it is judged that the material level is too high; Conversely, when visual detection does not detect a blockage signal and radar detection detects that the material level is below the lower material level, it is determined that the material level is too low.

[0055] The visual detection model determines whether there is a blockage signal, and then combines the radar detection material level information to make a judgment.

[0056] This method realizes real-time state monitoring, can find problems in the production process and feedback in time, avoids the delay of manual inspection, and improves the monitoring efficiency.

[0057] By monitoring the running state of the sintering trolley in real time, problems such as blockage can be found in time, which can effectively avoid production stagnation, reduce unnecessary downtime, and improve production efficiency. Combined with visual data and radar data, the detection accuracy can be greatly improved. Visual data provides accurate feedback on the operation process of the sintering trolley, while radar data provides real-time information about the material level. The combination of the two enables the system to make more accurate state judgments. This system can reduce the frequency and intensity of manual inspection, reduce the probability of human error, and improve safety through automated detection and judgment. The YOLOv10 model can adapt to changes in different working conditions and has good adaptability to different blockage conditions and production environments. At the same time, the model can be optimized through cross-validation to improve its generalization ability. The pruned and quantized model can efficiently utilize hardware resources when running on actual hardware, reducing the system's requirements for computing resources, making this technology widely applicable in actual production environments. This method can be further extended to other production equipment or fields, through similar multi-modal data analysis, to monitor equipment status and predict faults, and has strong universality and expandability.

[0058] The present application has been described by the above-mentioned related embodiments, however, the above-mentioned embodiments are only examples of implementing the present application. It must be pointed out that the disclosed embodiments do not limit the scope of the present application. On the contrary, changes and modifications made without departing from the spirit and scope of the present application are within the scope of the patent protection of the present application.

Claims

1. A condition assessment system based on dynamic monitoring of multimodal data, characterized by: The state assessment system includes Multimodal data acquisition module, including a high-temperature resistant camera and a radar sensor, used to collect material layer images and data at different material level heights respectively; The data processing module includes a visual recognition integrated machine and a control integrated machine; the visual recognition integrated machine runs a material blockage detection model to process image data, and the control integrated machine integrates visual signals and radar data to determine the state of the material layer; And the dynamic control module automatically adjusts the speed of the round roller feeder according to the state of the material layer.

2. A condition assessment system based on dynamic monitoring of multimodal data according to claim 1, characterized in that: The radar sensors are arranged on both sides of the front end of the baffle, including a high-level laser radar arranged at the upper material position and a low-level laser radar arranged at the lower material position; the high-temperature resistant camera is installed on the upper side of the sintering trolley baffle.

3. The state assessment system based on dynamic monitoring of multimodal data according to claim 1, characterized in that: The data processing module and the multimodal data acquisition module are connected by network signals via industrial Ethernet; the high-temperature resistant camera and radar sensor are connected to the visual recognition all-in-one machine and the control all-in-one machine via industrial Ethernet; the visual recognition all-in-one machine receives the pre-processing of video data and the operation of the blockage recognition model and transmits the recognition results to the control all-in-one machine; the control all-in-one machine receives the visual blockage signal and the radar signal and fuses the two modal data, and adjusts the speed of the round roller feeder according to preset rules.

4. The state assessment system based on dynamic monitoring of multimodal data according to claim 1, characterized in that: The dynamic control module is connected to each device via industrial Ethernet.

5. The state assessment system based on dynamic monitoring of multimodal data according to claim 3, characterized in that: The control all-in-one machine triggers an alarm mechanism when the visual blockage is inconsistent with the radar data.

6. A method for a condition assessment system based on dynamic monitoring of multimodal data according to claim 1, characterized in that: The specific steps of the method are as follows: S1: Data collection and data preprocessing: Collect running video data of the sintering trolley under different working conditions and perform data preprocessing on the collected video data, including denoising, frame rate adjustment and image enhancement; S2: Data segmentation and labeling; Segment the preprocessed data according to time series or specific conditions, and label the segmented data; S3: Blockage detection model training: Use the YOLOv10 target detection model to detect material blockages; divide the labeled data in step S2 into material blockage areas and normal areas, use the labeled data for supervised learning, and then perform cross-validation, adjust hyperparameters, and optimize model performance; S4: Model deployment: Prune and quantize the model trained in step S3 to adapt to the performance of the actual hardware, and deploy the processed model on the visual recognition all-in-one machine; S5: Real-time judgment: When the visual detection model detects a blocking signal and the radar detects that the material level is higher than the upper material level, it is judged that the material level is too high; On the contrary, when the visual inspection does not detect the blocking signal and the radar detects that the material level is lower than the lower material level, it is judged that the material level is too low.

Images