Multi-modal data conversion and fusion method for fault diagnosis system

Abstract

The invention discloses a multi-modal data conversion and fusion method for a fault diagnosis system. The method comprises the following steps: S1, spatial registration of multi-modal data: calculating a homography transformation matrix H between different data acquisition units according to data acquired by the different data acquisition units through a reference feature point matching method; s2, time synchronization of the multi-modal data: taking the timestamp of one data acquisition unit as a reference, and performing time synchronization on the data of other data acquisition units through a recent principle; s3, data normalization: mapping the data to a similar numerical range; and S4, data fusion: carrying out data fusion through a feature level fusion strategy after data normalization. According to the multi-modal data conversion and fusion method for the fault diagnosis system, the timeliness, accuracy and objectivity of data used by the fault diagnosis system can be effectively improved, and the accuracy of fault recognition and the accuracy of fault evaluation of the fault diagnosis system are improved.

Description

Technical Field

[0001] This invention relates to the field of fault diagnosis technology in coal mines, and specifically to a method for multimodal data conversion and fusion in fault diagnosis systems. Background Technology

[0002] The underground environment in coal mines is complex, and the collected data usually covers multiple modes such as infrared, visible light, and gas. How to analyze and process various data for use in fault diagnosis systems is related to the accuracy of fault identification and the precision of assessment.

[0003] Application number CN202411669740.4 discloses a method for detecting the status of underground equipment based on multimodal data fusion, relating to the field of underground equipment detection technology in coal mines. The method includes: acquiring textual information about the start / stop status of underground equipment and monitoring video information; inputting the textual information about the start / stop status into a pre-trained text-image generation model to obtain a first image; acquiring video frame information from the monitoring video information; synthesizing the first image and the video frame information to generate an image set; and using a visual detection algorithm to detect the image set to obtain the first start / stop status detection result of the underground equipment. This disclosure comprehensively judges the start / stop status of underground equipment by combining multiple information sources, enriching data sources and improving the accuracy of equipment status detection. Furthermore, when there are problems with video information, the start / stop status textual information can be used to supplement the data, which can, to a certain extent, avoid the problem of undetectable or inaccurate detection of equipment start / stop status due to lost video information, thus ensuring construction safety.

[0004] Application number CN202410798749.9 discloses a fire early warning method and system for coal mines, as well as a storage medium. This invention deploys a multimodal sensor network within the mine. After data cleaning, time synchronization, and fusion processing, multimodal feature data is extracted, and a fire risk score is calculated using a fire risk assessment model. The fire risk feature data is uploaded to a blockchain network for encrypted storage and monitored in real time via smart contracts, automatically triggering early warning signals and initiating emergency response procedures. Combining virtual reality and augmented reality technologies, a virtual model of the mine is constructed, dynamically displaying fire risk areas and flame spread paths. The control center initiates emergency response procedures based on the early warning signal and provides real-time feedback on emergency handling through virtual reality and augmented reality devices, enabling fire emergency drills and capability enhancement. This solution effectively improves the accuracy of fire early warning and the efficiency of emergency response, reducing casualties and property losses caused by fires. Summary of the Invention

[0005] This invention aims to overcome the technical problem of low efficiency caused by the diversification of data used in existing fault diagnosis systems. It provides a method for multimodal data conversion and fusion in fault diagnosis systems, which can effectively improve the timeliness, accuracy and objectivity of the data used in fault diagnosis systems, and improve the accuracy of fault identification and the precision of fault assessment.

[0006] This invention provides a method for multimodal data conversion and fusion in a fault diagnosis system, comprising the following steps: S1. Spatial registration of multimodal data: The homography transformation matrix H between different data acquisition units is calculated by matching reference feature points. S2. Time synchronization of multimodal data: Based on the timestamp of one data acquisition unit, the data of other data acquisition units are synchronized in time according to the nearest principle. S3. Data normalization: Mapping data to a similar range of values; S4. Data Fusion: After normalizing the data, data fusion is performed using a feature-level fusion strategy.

[0007] Because of the physical discrepancies in the installation positions of the infrared thermal imaging module and the visible light camera, the fields of view of the images they acquire do not completely overlap. In order to accurately superimpose the thermal spectrum information onto the visible light image, spatial registration is required. At the same time, the sampling frequencies of different sensors may be different. In order to ensure that the data fed into the neural network is a snapshot of the device state at the same moment, time synchronization is required.

[0008] The multimodal data conversion and fusion method for a fault diagnosis system described in this invention, as a preferred embodiment, further includes the following steps in step S1: S11. Offline calibration: Take a set of calibration board images containing feature points under controlled conditions; extract matching feature point pairs (x1, x2) from the data using the SIFT or ORB algorithm. S12, Matrix Calculation: Calculate the homography matrix H using the RANSAC algorithm: ; Where: [x1, y1] is the data I1 collected by the first acquisition unit, and [x2, y2] is the data I2 collected by the second acquisition unit; S13, Image Transformation: During device operation, the real-time acquired data I1 undergoes perspective transformation using the homography matrix H to generate data I1' aligned with data I2. ; Where size(I2) is the size of data I2.

[0009] warpPerspective is a function in OpenCV used for perspective transformation of images. It maps an image from one viewpoint to another using a 3×3 transformation matrix and is often used to correct tilted images or simulate 3D projection effects.

[0010] The multimodal data conversion and fusion method for a fault diagnosis system described in this invention, in a preferred embodiment, further includes the following steps in step S1: S14. Data scaling: Scaling data I2 and data I1' to a uniform resolution W.

[0011] The multimodal data conversion and fusion method for a fault diagnosis system described in this invention, as a preferred embodiment, further includes the following steps in step S2: S21. Reference Determination: The timest of the data I2 collected by the second acquisition unit is used as the reference data I. 2t ; S22, Data Selection: For each frame of reference data I 2t The data I1 collected by the first acquisition unit and the data I3 collected by the third acquisition unit, which are closest in timestamp to I, are selected as the generated data I. 1t and I 3t The data selection method for generation is as follows: ; ; Where t1 is the timestamp of data I1, and t3 is the timestamp of data I3; S23, Time Synchronization: Combine the selected data with the reference data to form a data tuple (I 1t I 2t I 3t ), thus completing the time synchronization.

[0012] The multimodal data conversion and fusion method for a fault diagnosis system described in this invention, in a preferred embodiment, further includes the following steps in step S3: S31. Image data normalization: Normalize the pixel values ​​of data I2 and data I1' to the interval [0,1] or [-1,1]. S32. Gas concentration data normalization: For each element g of the data I3 collected by the third acquisition unit, linear normalization is performed according to the sensor's range [g0, g1] to obtain normalized data g': ; Where g0 is the lower limit of the sensor's range, and g1 is the upper limit of the sensor's range.

[0013] The multimodal data transformation and fusion method for fault diagnosis systems described in this invention, as a preferred embodiment, employs a feature-level fusion strategy that concatenates the feature vectors F1, F2, and F3 extracted from each modality along their feature dimensions to form a fused feature vector F0. ; ; Wherein: F1 is the feature vector of data I1' after image transformation of data I1 acquired by the first acquisition unit, F2 is the feature vector of data I2 acquired by the second acquisition unit, and F3 is the feature vector of data I3 acquired by the second acquisition unit.

[0014] The multimodal data transformation and fusion method for fault diagnosis systems described in this invention, as a preferred embodiment, employs a feature-level fusion strategy that involves concatenating the feature vectors F1, F2, and F3 extracted from each modality along their feature dimensions after assigning weight coefficients to them, resulting in a fused feature vector F0. ; Wherein: F1 is the feature vector of data I1' after image transformation of data I1 collected by the first acquisition unit, F2 is the feature vector of data I2 collected by the second acquisition unit, and F3 is the feature vector of data I3 collected by the second acquisition unit; α1, α2, and α3 are the weight coefficients of feature vectors F1, F2, and F3, respectively.

[0015] The multimodal data conversion and fusion method for a fault diagnosis system described in this invention, as a preferred embodiment, involves the first acquisition unit acquiring data I1 as an infrared image, the second acquisition unit acquiring data I2 as a visible light image, and the third acquisition unit acquiring data I3 as a gas concentration vector.

[0016] The present invention has the following advantages: This invention improves the timeliness, accuracy, and objectivity of data used in fault diagnosis systems by integrating multi-dimensional information such as infrared thermal imaging, visible light images, and gas sensing, thereby enhancing the accuracy of fault identification and the precision of fault assessment. Attached Figure Description

[0017] Figure 1 This is a flowchart of a multimodal data transformation and fusion method for a fault diagnosis system. Figure 2 This is a flowchart illustrating the spatial registration process of multimodal data in a multimodal data transformation and fusion method for a fault diagnosis system. Figure 3 This is a flowchart illustrating the time synchronization of multimodal data in a multimodal data transformation and fusion method for a fault diagnosis system. Figure 4This is a flowchart of data normalization for a multimodal data transformation and fusion method used in a fault diagnosis system. Detailed Implementation

[0018] Example 1 like Figure 1 As shown, a method for multimodal data conversion and fusion in a fault diagnosis system includes the following steps: S1. Spatial Registration of Multimodal Data: The homography transformation matrix H between different data acquisition units is calculated using a reference feature point matching method. The data acquisition units include a first acquisition unit, a second acquisition unit, and a third acquisition unit. The data I1 acquired by the first acquisition unit is an infrared image; the data I2 acquired by the second acquisition unit is a visible light image; and the data I3 acquired by the third acquisition unit is a gas concentration vector. Figure 2 As shown, step S1 further includes the following steps: S11. Offline calibration: Take a set of calibration board images containing feature points (such as a checkerboard pattern) under controlled conditions; extract matching feature point pairs (x1, x2) from the data using the SIFT or ORB algorithm. S12, Matrix Calculation: Calculate the homography matrix H using the RANSAC (Random Sample Consensus) algorithm. ; Where: [x1, y1] is the data I1 collected by the first acquisition unit, and [x2, y2] is the data I2 collected by the second acquisition unit; S13, Image Transformation: During device operation, the real-time acquired data I1 undergoes perspective transformation using the homography matrix H to generate data I1' aligned with data I2. ; Where size(I2) is the size of data I2; S14. Data scaling: In order to match the input size requirements of the neural network, data I2 and data I1' are scaled to a uniform resolution W; S2. Time synchronization of multimodal data: Using the timestamp of one data acquisition unit as a reference, time synchronization is performed on the data of other data acquisition units according to the nearest principle; for example... Figure 3 As shown, step S2 further includes the following steps: S21. Reference Determination: The timest of the data I2 collected by the second acquisition unit is used as the reference data I. 2t ; S22, Data Selection: For each frame of reference data I 2tThe data I1 collected by the first acquisition unit and the data I3 collected by the third acquisition unit, which are closest in timestamp to I, are selected as the generated data I. 1t and I 3t The data selection method for generation is as follows: ; ; Where t1 is the timestamp of data I1, and t3 is the timestamp of data I3; S23, Time Synchronization: Combine the selected data with the reference data to form a data tuple (I 1t I 2t I 3t ), to complete the time synchronization; S3. Data Normalization: Mapping data to a similar range of values; for example... Figure 4 As shown, step S3 further includes the following steps: S31. Image data normalization: Normalize the pixel values ​​of data I2 and data I1' to the interval [0,1] or [-1,1]. S32. Gas concentration data normalization: For each element g of the data I3 collected by the third acquisition unit, linear normalization is performed according to the sensor's range [g0, g1] to obtain normalized data g': ; Where: g0 is the lower limit of the sensor's range, and g1 is the upper limit of the sensor's range; S4. Data Fusion: After normalizing the data, data fusion is performed using a feature-level fusion strategy; two main feature-level fusion strategies are supported, which can be selected during model design: early-stage fusion (concatenation fusion) and late-stage fusion (attention fusion). (1) Selecting early-stage fusion (concatenation fusion): Concatenating the feature vectors F1, F2, and F3 extracted from each modality into a fused feature vector F0 along the feature dimension. ; Wherein: F1 is the feature vector of data I1' after image transformation of data I1 acquired by the first acquisition unit, F2 is the feature vector of data I2 acquired by the second acquisition unit, and F3 is the feature vector of data I3 acquired by the second acquisition unit.

[0019] (2) Selecting late-stage fusion (attention fusion): Introducing an attention mechanism, the feature vectors F1, F2, and F3 extracted from each modality are concatenated along the feature dimension after being assigned weight coefficients to form a fused feature vector F0. ; ; Wherein: F1 is the feature vector of data I1' after image transformation of data I1 collected by the first acquisition unit, F2 is the feature vector of data I2 collected by the second acquisition unit, and F3 is the feature vector of data I3 collected by the second acquisition unit; α1, α2, and α3 are the weight coefficients of feature vectors F1, F2, and F3, respectively.

[0020] The fault diagnosis system assigns different weights to each feature vector in different detection scenarios. For example, when detecting thermal anomalies, infrared feature I1' is given a higher weight; when judging structural deformation, visible light I2 is given a higher weight.

[0021] The above description is illustrative only and not restrictive of the present invention. Those skilled in the art will understand that any modifications, variations or equivalents that can be made without departing from the spirit and scope defined by the claims will fall within the protection scope of the present invention.

Claims

1. A method for multimodal data conversion and fusion in a fault diagnosis system, characterized in that: Includes the following steps: S1. Spatial registration of multimodal data: The homography transformation matrix H between different data acquisition units is calculated by matching reference feature points. S2. Time synchronization of multimodal data: Based on the timestamp of one of the data acquisition units, the data of the other data acquisition units are synchronized in time according to the nearest principle; S3. Data normalization: Mapping the data to a similar numerical range; S4. Data Fusion: After normalizing the data, data fusion is performed using a feature-level fusion strategy.

2. The method for multimodal data conversion and fusion in a fault diagnosis system according to claim 1, characterized in that: Step S1 further includes the following steps: S11. Offline calibration: Take a set of calibration board images containing feature points under controlled conditions; extract matching feature point pairs (x1, x2) from the data using the SIFT algorithm or ORB algorithm. S12, Matrix Calculation: Calculate the homography matrix H using the RANSAC algorithm: ; Where: [x1, y1] is the data I1 collected by the first acquisition unit, and [x2, y2] is the data I2 collected by the second acquisition unit; S13. Image Transformation: During device operation, the real-time acquired data I1 is subjected to perspective transformation through the homography matrix H to generate data I1' aligned with the data I2. ; Where size(I2) is the size of the data I2.

3. The method for multimodal data conversion and fusion in a fault diagnosis system according to claim 2, characterized in that: Step S1 also includes the following steps: S14. Data scaling: Scaling the data I2 and the data I1' to a uniform resolution W.

4. The method for multimodal data conversion and fusion in a fault diagnosis system according to claim 2, characterized in that: The data I1 collected by the first acquisition unit is an infrared image, and the data I2 collected by the second acquisition unit is a visible light image.

5. The method for multimodal data conversion and fusion in a fault diagnosis system according to claim 1, characterized in that: Step S2 further includes the following steps: S21. Reference Determination: The timest of the data I2 collected by the second acquisition unit is used as the reference data I. 2t ; S22, Data Selection: For each frame of the reference data I 2t The data I1 collected by the first acquisition unit and the data I3 collected by the third acquisition unit, which are closest to the timestamp mentioned above, are selected as the generated data I. 1t and I 3t The data selection method for generation is as follows: ; ; Where t1 is the timestamp of data I1, and t3 is the timestamp of data I3; S23, Time Synchronization: Combine the selected data with the reference data to form a data tuple (I 1t I 2t I 3t ), thus completing the time synchronization.

6. The multimodal data conversion and fusion method for a fault diagnosis system according to claim 5, characterized in that: The data I1 collected by the first acquisition unit is an infrared image, the data I2 collected by the second acquisition unit is a visible light image, and the data I3 collected by the third acquisition unit is a gas concentration vector.

7. The method for multimodal data conversion and fusion in a fault diagnosis system according to claim 2, characterized in that: Step S3 further includes the following steps: S31. Image data normalization: Normalize the pixel values ​​of the data I2 and the data I1' to the interval [0,1] or [-1,1]. S32. Gas concentration data normalization: For each element g of the data I3 collected by the third acquisition unit, linear normalization is performed according to the sensor's range [g0, g1] to obtain normalized data g': ; Wherein: g0 is the lower limit of the sensor's range, and g1 is the upper limit of the sensor's range.

8. The multimodal data conversion and fusion method for a fault diagnosis system according to claim 7, characterized in that: The data I3 collected by the third acquisition unit is a gas concentration vector.

9. A method for multimodal data conversion and fusion in a fault diagnosis system according to claim 2, characterized in that: The feature-level fusion strategy involves concatenating the feature vectors F1, F2, and F3 extracted from each modality along the feature dimension to form a fused feature vector F0. ; Wherein: F1 is the feature vector of the data I1 collected by the first acquisition unit after image transformation, F2 is the feature vector of the data I2 collected by the second acquisition unit, and F3 is the feature vector of the data I3 collected by the second acquisition unit; The data I1 collected by the first acquisition unit is an infrared image, the data I2 collected by the second acquisition unit is a visible light image, and the data I3 collected by the third acquisition unit is a gas concentration vector.

10. A method for multimodal data conversion and fusion in a fault diagnosis system according to claim 2, characterized in that: The feature-level fusion strategy involves concatenating the feature vectors F1, F2, and F3 extracted from each modality into a fused feature vector F0 after assigning weight coefficients. ; ; Wherein: F1 is the feature vector of data I1' after image transformation of data I1 collected by the first acquisition unit, F2 is the feature vector of data I2 collected by the second acquisition unit, and F3 is the feature vector of data I3 collected by the second acquisition unit; α1, α2, and α3 are the weight coefficients of feature vectors F1, F2, and F3, respectively. The data I1 collected by the first acquisition unit is an infrared image, the data I2 collected by the second acquisition unit is a visible light image, and the data I3 collected by the third acquisition unit is a gas concentration vector.

Images