Sintering machine trolley fault online diagnosis method and system based on multi-modal vision

Abstract

The application discloses a kind of based on multimodal vision's sintering machine trolley key component fault online diagnosis method and system, belong to sintering machine equipment state monitoring technical field.The method includes: through displacement encoder real-time monitoring trolley movement distance, when reaching preset step Δs, trigger multimodal acquisition unit synchronous acquisition visible light image, infrared thermogram and structured light image;According to trolley real-time movement speed, the image is motion compensated alignment, eliminate spatial position deviation;From the image after alignment, extract geometric feature, thermal feature and texture feature and carry out fusion, generate fusion feature vector;According to fusion feature vector, calculate abnormal intensity index, carry out hierarchical evaluation and alarm to fault.The application guarantees spatial sampling consistency by fixed-distance trigger and motion compensation mechanism, realizes the joint analysis of texture, temperature and geometric feature by multimodal feature fusion, realizes fault quantitative evaluation and closed-loop management by abnormal intensity index.

Description

Technical Field

[0001] This invention relates to the field of sintering machine trolley technology, specifically to a method and system for online fault diagnosis of sintering machine trolleys based on multimodal vision. Background Technology

[0002] The sintering machine trolley is the core equipment for carrying and transporting materials in the sintering process, and its operating status directly affects the output, quality, and safety of sintered ore. The trolley operates in a harsh environment of high temperature, high dust, and high vibration for extended periods, making critical components such as grate bars, wall panels, and wheels prone to wear, misalignment, cracks, and wheel loss. Grate bars often warp or break under thermal stress and mechanical impact; wall panels may tilt or loosen under thermal fatigue; and wheels are prone to deviation, jamming, or even wheel loss under prolonged load and dust erosion. If these problems are not identified early, they will lead to abnormal material layer distribution, increased ventilation resistance, and increased energy consumption, and in severe cases, equipment failure and downtime.

[0003] Currently, on-site inspections mainly rely on manual inspections or single infrared thermal imaging monitoring. Manual methods suffer from long inspection cycles, high labor intensity, and strong subjectivity, making continuous monitoring impossible while equipment is running. While single-modal thermal imagers offer the advantage of non-contact temperature measurement, they are prone to misjudgment or missed detection under dust obstruction, light fluctuations, and temperature interference, only reflecting temperature changes and unable to distinguish structural anomalies. Some video monitoring solutions struggle with geometric measurement and precise positioning due to uneven sampling and blurry images. These methods generally suffer from low detection accuracy, poor environmental adaptability, and slow response, making it difficult to meet the online diagnostic needs under continuous production conditions in sintering machines. Summary of the Invention

[0004] (a) Technical problems to be solved The technical problem to be solved by the present invention is to provide a method and system for online diagnosis of sintering machine trolley faults based on multimodal vision. This method addresses the challenges in the detection of key components of the sintering machine trolley, such as inconsistent image space due to trolley speed fluctuations, insufficient single-modal detection information leading to low fault identification accuracy, delayed response and unstable results due to manual interpretation, and the lack of quantitative diagnosis and hierarchical early warning mechanisms, which make it difficult to form an intelligent closed-loop management system.

[0005] (II) Technical Solution To solve the above-mentioned technical problems, the technical solution provided by this invention is: an online fault diagnosis method for key components of a sintering machine trolley based on multimodal vision, comprising the following steps: Acquisition Steps: The trolley's movement distance is monitored in real time by a displacement encoder. When the movement distance reaches a preset step size Δs, the multimodal acquisition unit is triggered to simultaneously acquire visible light images, infrared thermal images, and structured light images of key components of the trolley. Alignment Steps: Based on the trolley's real-time movement speed, the acquired visible light images, infrared thermal images, and structured light images are aligned by motion compensation to eliminate spatial position deviations caused by sampling time differences, resulting in spatially aligned multimodal images. Feature Extraction and Fusion Steps: Geometric features, thermal features, and texture features are extracted from the aligned multimodal images, and the extracted features are fused to generate a fused feature vector. Diagnosis Steps: Based on the fused feature vector, an anomaly intensity index is calculated to quantify the degree of target anomaly, and the fault is graded and assessed and alarmed based on this index.

[0006] As an improvement, the alignment step specifically includes: calculating the displacement at the sampling time by integrating the real-time velocity function v(t) fed back by the displacement encoder, and using interpolation or resampling algorithms to correct the image data of different modalities to a unified spatial coordinate system, thereby achieving accurate alignment of multimodal images.

[0007] As an improvement, the feature extraction and fusion step further includes: extracting geometric features from the structured light image, the geometric features including height difference, tilt angle and spacing variation; extracting thermal features from the infrared thermal image, the thermal features including temperature gradient and abnormal hot spots; extracting texture features from the visible light image, the texture features including crack edges and surface roughness; and after normalizing the geometric features, thermal features and texture features, fusing them through an attention weighting mechanism to generate the fused feature vector.

[0008] As an improvement, the formula for calculating the anomaly intensity index is as follows:

[0009] in, Standardized scores for each modal feature, For the corresponding modal weights, satisfying Based on the ESI value range, anomalies are classified into three levels: minor, moderate, and severe, and are linked with the maintenance work order system to achieve graded alarms and automatic work order dispatch.

[0010] As an improvement, in the diagnostic step, the abnormality intensity index is divided into three levels: slight, moderate and severe, according to a preset first threshold and a second threshold. When a severe abnormality is determined, a maintenance work order is automatically generated and pushed to the equipment management platform.

[0011] An online fault diagnosis system for key components of a sintering machine trolley based on multimodal vision includes: The multi-modal acquisition module adopts a multi-modal array composed of an industrial camera (RGB), an infrared thermal imager (IR), and a structured light sensor (SL). It is arranged above or on the side of the sintering machine trolley channel, and the field of view covers the entire width of the trolley. The system performs synchronous acquisition through a GigE interface and is equipped with a dust and heat protection cover to ensure the long-term stable operation of the equipment in a high-temperature and dusty environment; the synchronous trigger and motion compensation module includes a displacement encoder and a motion compensation controller. The displacement encoder is used to monitor the moving distance of the trolley in real time and generate a trigger signal when the preset step length Δs is reached. The motion compensation controller is used to perform spatial alignment on the multi-modal images according to the trolley speed; the multi-modal feature fusion and analysis module is used to extract geometric, thermal, and texture features from the aligned images and fuse them to generate a fused feature vector; the anomaly evaluation and alarm module is used to calculate the anomaly intensity index according to the fused feature vector and perform fault grading, alarm, and work order linkage according to this index.

[0012] As an improvement, the synchronous trigger and motion compensation module corrects the acquisition time through interpolation according to the real-time speed fed back by the displacement encoder , ensuring that each sampling point is evenly distributed in physical space, thereby eliminating the spatial mismatch caused by speed fluctuations: .

[0013] As an improvement, the multi-modal feature fusion and analysis module adopts a feature-level fusion strategy. First, it extracts the feature vectors of the three types of modalities through an independent convolutional network , and , and then uses attention weighted fusion:

[0014] where is an adaptive weight, which is dynamically learned from the training data.

[0015] As an improvement, in the anomaly evaluation and alarm module, the system performs classification and regression analysis on the fused features, obtains the anomaly intensity index through the ESI model, and determines the anomaly level according to the threshold : ESI < T1 is normal; T1 ≤ ESI < T2 is a minor anomaly; ESI ≥ T2 is a serious anomaly; When it is determined to be a serious anomaly, the system automatically generates a maintenance work order and pushes it to the equipment management platform to achieve intelligent closed-loop maintenance.

[0016] As an improvement, the multimodal acquisition unit, the synchronous triggering and motion compensation module, the multimodal feature fusion analysis module, and the anomaly assessment and alarm module communicate via Ethernet, and the data storage adopts a ring cache structure to achieve millisecond-level acquisition and second-level diagnostic output.

[0017] (III) Beneficial Effects The advantages of this invention compared to existing technologies are as follows: This invention solves the problem of inconsistent sampling space in traditional time-triggered methods by establishing a fixed-distance triggering and kinematic compensation mechanism, ensuring the correspondence of acquired images in physical space; it achieves joint analysis of texture, temperature, and geometric features by fusing multi-source information such as visible light, infrared thermal imaging, and structured light, overcoming the deficiency of insufficient information in a single modality; it quantifies multi-dimensional features into a unified fault intensity index by designing an anomaly intensity index model, enabling comparable and hierarchical identification of different types of anomalies; and it achieves closed-loop management from detection and evaluation to handling by establishing an intelligent hierarchical alarm and work order linkage mechanism, significantly improving the automation and intelligence level of the system. Attached Figure Description

[0018] Figure 1 This is the overall structural diagram of the online fault diagnosis method and system for sintering machine trolley based on multimodal vision according to the present invention.

[0019] Figure 2 This is a system flowchart illustrating the online fault diagnosis method and system structure of the sintering machine trolley based on multimodal vision according to the present invention. Detailed Implementation

[0020] The invention will now be described in further detail with reference to specific embodiments, but this should not be construed as limiting the scope of the subject matter of the invention to the following embodiments.

[0021] like Figure 1 As shown, the online fault diagnosis method for key components of a sintering machine trolley based on multimodal vision includes the following steps: Acquisition Steps: The trolley's movement distance is monitored in real-time using a displacement encoder. When the distance reaches a preset step size Δs, the multimodal acquisition unit is triggered to simultaneously acquire visible light images, infrared thermal images, and structured light images of key components of the trolley. Alignment Steps: Based on the trolley's real-time speed, the acquired visible light images, infrared thermal images, and structured light images are aligned using motion compensation to eliminate spatial position deviations caused by sampling time differences, resulting in spatially aligned multimodal images. Specifically, this includes: calculating the displacement at the sampling time using the real-time velocity function v(t) fed back by the displacement encoder through integration, and using interpolation or resampling algorithms to correct the image data of different modalities to a unified spatial coordinate system, achieving accurate alignment of the multimodal images. Feature Extraction and Fusion Steps: From the aligned multimodal images... Geometric features, thermal features, and texture features are extracted from the image, and the extracted features are fused to generate a fused feature vector. The process further includes: extracting geometric features from the structured light image, including height difference, tilt angle, and spacing variation; extracting thermal features from the infrared thermal image, including temperature gradient and abnormal hot spots; and extracting texture features from the visible light image, including crack edges and surface roughness. After normalizing the geometric, thermal, and texture features, they are fused using an attention weighting mechanism to generate the fused feature vector. The diagnostic step involves calculating an anomaly intensity index to quantify the degree of target anomaly based on the fused feature vector, and then classifying and alarming the fault according to this index. The formula for calculating the anomaly intensity index is:

[0022] in, Standardized scores for each modal feature, For the corresponding modal weights, satisfying Based on the ESI value range, anomalies are classified into three levels: minor, moderate, and severe, and are linked with the maintenance work order system to achieve graded alarms and automatic work order dispatch.

[0023] In the diagnostic steps, the abnormality intensity index is divided into three levels: slight, moderate and severe, according to the preset first threshold and second threshold. When a severe abnormality is determined, a maintenance work order is automatically generated and pushed to the equipment management platform.

[0024] like Figure 2 As shown, the online fault diagnosis system for key components of the sintering machine trolley based on multimodal vision includes: Multi-modal acquisition module, which adopts a multi-modal array composed of an industrial camera (RGB), an infrared thermal imager (IR) and a structured light sensor (SL), is arranged above or on the side of the sintering machine trolley channel, and the field of view covers the entire width of the trolley. The system performs synchronous acquisition through a GigE interface and is equipped with a dust and heat protection cover to ensure the long-term stable operation of the equipment in a high-temperature and dusty environment; Synchronous trigger and motion compensation module, including a displacement encoder and a motion compensation controller. The displacement encoder is used to monitor the moving distance of the trolley in real time and generate a trigger signal when the preset step size Δs is reached. The motion compensation controller is used to spatially align the multi-modal images according to the trolley speed. The synchronous trigger and motion compensation module , corrects the acquisition time by interpolation method , ensures that each sampling point is evenly distributed in physical space, thereby eliminating the spatial mismatch caused by speed fluctuations: Multi-modal feature fusion analysis module, which is used to extract geometric, thermal and texture features from the aligned images and fuse them to generate a fused feature vector. The multi-modal feature fusion analysis module adopts a feature-level fusion strategy. First, it extracts the feature vectors of the three types of modalities through an independent convolutional network 、 and , and then uses attention weighted fusion:

[0025] Among them, is an adaptive weight, which is dynamically learned from training data. Abnormality evaluation and alarm module, which is used to calculate the abnormality intensity index according to the fused feature vector, and perform fault classification, alarm and work order linkage according to this index. In the abnormality evaluation and alarm module, the system performs classification and regression analysis on the fused features, obtains the abnormality intensity index through the ESI model, and determines the abnormality level according to the threshold : ESI<T1 is normal; T1≤ESI<T2 is a minor abnormality; ESI≥T2 is a serious abnormality; When it is determined as a serious abnormality, the system automatically generates a maintenance work order and pushes it to the equipment management platform to achieve intelligent closed-loop maintenance.

[0026] The multi-modal acquisition unit, the synchronous trigger and motion compensation module, the multi-modal feature fusion analysis module and the abnormality evaluation and alarm module communicate with each other through Ethernet, and the data storage adopts a circular buffer structure to achieve millisecond-level acquisition and second-level diagnostic output.

[0027] In practical applications, the entire system achieves multi-module communication via Ethernet, and data storage adopts a ring cache structure to ensure millisecond-level acquisition and second-level diagnostic output during continuous trolley operation, thereby realizing true online intelligent fault diagnosis without shutdown.

[0028] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

[0029] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents. In short, if those skilled in the art are inspired by these claims and design similar structural methods and embodiments without departing from the inventive spirit of the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A method for online fault diagnosis of key components of a sintering machine trolley based on multimodal vision, characterized in that, It includes the following steps: Acquisition step: The movement distance of the trolley is monitored in real time by a displacement encoder. When the movement distance reaches the preset step length Δs, the multi-modal acquisition unit is triggered to synchronously acquire the visible light image, infrared thermal image and structured light image of the key components of the trolley; Alignment Step: According to the real-time movement speed of the trolley, perform motion compensation alignment on the acquired visible light image, infrared thermal image and structured light image to eliminate the spatial position deviation caused by the sampling time difference, and obtain the spatially aligned multi-modal images; Feature extraction and fusion step: Geometric features, thermal features and texture features are respectively extracted from the aligned multi-modal images, and the extracted features are fused to generate a fused feature vector; Diagnosis step: According to the fused feature vector, calculate the abnormal intensity index for quantifying the abnormal degree of the target, and perform hierarchical evaluation and alarm on the fault according to this index.

2. The online fault diagnosis method for key components of a sintering machine trolley based on multimodal vision according to claim 1, characterized in that, The alignment step specifically includes: According to the real-time speed function v(t) fed back by the displacement encoder, calculate the displacement at the sampling moment through integration, and use the interpolation or resampling algorithm to correct the image data of different modalities to the same spatial coordinates to achieve precise alignment of the multi-modal images.

3. The online fault diagnosis method for key components of a sintering machine trolley based on multimodal vision according to claim 1, characterized in that, The feature extraction and fusion step further includes: Extract geometric features from the structured light image, and the geometric features include height difference, inclination angle and spacing change; Extract thermal features from the infrared thermal image, and the thermal features include temperature gradient and abnormal hot spots; Extract texture features from the visible light image, and the texture features include crack edges and surface roughness; After normalizing the geometric features, thermal features and texture features, fuse them through an attention weighting mechanism to generate the fused feature vector.

4. The online fault diagnosis method for key components of a sintering machine trolley based on multimodal vision according to claim 1, characterized in that, The calculation formula of the abnormal intensity index is: ， in, Standardized scores for each modal feature, For the corresponding modal weights, satisfying Based on the ESI value range, anomalies are classified into three levels: minor, moderate, and severe, and are linked with the maintenance work order system to achieve graded alarms and automatic work order dispatch.

5. The online fault diagnosis method for key components of a sintering machine trolley based on multimodal vision according to claim 1, characterized in that, In the diagnosis step, the abnormal intensity index is divided into three levels: mild, moderate and severe according to the preset first threshold and second threshold. When it is determined as a severe abnormality, a maintenance work order is automatically generated and pushed to the equipment management platform.

6. A multimodal vision-based online fault diagnosis system for key components of a sintering machine trolley for implementing the method of any one of claims 1 to 5, characterized in that, It includes: A multi-modal acquisition module, which adopts a multi-modal array composed of an industrial camera (RGB), an infrared thermal imager (IR) and a structured light sensor (SL), is arranged above or on the side of the sintering machine trolley channel, and the field of view covers the entire width of the trolley. The system performs synchronous acquisition through a GigE interface and is equipped with a dust and heat protection cover to ensure the long-term stable operation of the equipment in a high-temperature and dusty environment; A synchronous trigger and motion compensation module, including a displacement encoder and a motion compensation controller. The displacement encoder is used to monitor the movement distance of the trolley in real time and generate a trigger signal when the preset step length Δs is reached. The motion compensation controller is used to perform spatial alignment on the multi-modal images according to the trolley speed; A multi-modal feature fusion analysis module, which is used to extract geometric, thermal and texture features from the aligned images and fuse them to generate a fused feature vector; Abnormal Evaluation and alarm module, which is used to calculate the abnormal intensity index according to the fused feature vector, and perform fault classification, alarm and work order linkage according to this index.

7. The online diagnostic system according to claim 6, characterized in that, The synchronous triggering and motion compensation module uses the real-time speed feedback from the displacement encoder. The acquisition time was corrected by interpolation. This ensures that each sampling point is evenly distributed in physical space, thereby eliminating spatial mismatch caused by velocity fluctuations. 。 8. The online diagnostic system according to claim 6, characterized in that, The multimodal feature fusion analysis module employs a feature-level fusion strategy, first extracting feature vectors for the three modalities through independent convolutional networks. , and Then use attention-weighted fusion: ， in, The weights are adaptive and are learned dynamically from the training data.

9. The online diagnostic system according to claim 6, characterized in that, In the anomaly assessment and alarm module, the system classifies and performs regression analysis on the fused features, obtains anomaly intensity indices through the ESI model, and determines the anomaly intensity based on a threshold. Determine the level of abnormality: ESI < T1 is normal; T1 ≤ ESI < T2 is a mild abnormality; ESI ≥ T2 is a severe abnormality; When a serious anomaly is detected, the system automatically generates a maintenance work order and pushes it to the equipment management platform to achieve intelligent closed-loop maintenance.

10. The online diagnostic system according to claim 6, characterized in that, The multimodal acquisition unit, synchronous triggering and motion compensation module, multimodal feature fusion analysis module, and anomaly assessment and alarm module communicate via Ethernet, and the data storage adopts a ring cache structure to achieve millisecond-level acquisition and second-level diagnostic output.

Images