An industrial equipment health state monitoring method and computer equipment based on multi-modal data fusion

Abstract

This invention provides a method and computer device for monitoring the health status of industrial equipment based on multimodal data fusion, belonging to the field of equipment fault diagnosis technology. First, low-frequency and high-frequency modal data of the industrial equipment under operating conditions are collected and preprocessed. The preprocessed low-frequency data is fed into a temporal extraction network to generate low-frequency feature vectors, and the high-frequency data is input into a two-dimensional convolutional neural network to obtain high-frequency feature vectors. A spatiotemporal dual-stream attention module containing operating condition guidance flow and acoustic-vibration complementary flow is constructed. Then, temperature features, enhanced vibration features, and acoustic-vibration resonance features are concatenated into channels, compressed using global average pooling to obtain a context fusion vector, which is fed into a two-layer fully connected perceptron. Dynamic weighting based on weights yields the final fused health feature. This feature is then fed into a fully connected layer and a classifier to output the equipment health status and specific fault type. This invention improves the accuracy of fault diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of equipment fault diagnosis technology. Specifically, this invention relates to a method and computer equipment for monitoring the health status of industrial equipment based on multimodal data fusion. Background Technology

[0002] In the fields of intelligent manufacturing and industrial automation, the stable and reliable operation of industrial equipment is crucial for ensuring continuous production line operation, reducing downtime losses, and controlling maintenance costs. During long-term operation, equipment is prone to hidden faults such as wear, spalling, deformation, and fatigue damage. If these faults are not monitored and warned of in a timely manner, they can easily lead to equipment downtime, production interruptions, or even safety accidents. Therefore, high-precision and robust equipment health monitoring and fault diagnosis technologies are core supporting technologies for achieving predictive maintenance of industrial equipment and improving the level of intelligence and safety in industrial production.

[0003] Chinese Patent 101592590A provides an indirect diagnostic technology for rotating blade faults. It includes a sensor group installed on a stationary component of a blade mill to sense vibrations and impacts caused by aerodynamic forces and hard friction between the rotor blades and the stationary component; a speed sensor to detect the operating frequency of the blade mill; and a blade fault diagnosis device that reflects blade fault information based on the motion reference law provided by the speed sensor signal and the signals from the vibration sensor group. The signals from the sensor group containing at least two vibration and impact sensors, and the signal from a speed sensor corresponding to each rotor type, are connected to the blade fault diagnosis device. A blade fault diagnosis expert system software within the microprocessor of the blade fault diagnosis device uses the vibration and speed signals to perform blade fault diagnosis. Through the combination of computer technology, electrical sensors, and the fault diagnosis expert system software, FFT analysis is performed on the lateral vibration and impact P, the axial vibration and impact Y, and the torsional vibration NZ of the blade mill's suspension or support device during operation. This analysis is then compared with the rotor's rotational speed frequency FN to identify blade faults.

[0004] Existing technologies for diagnosing equipment faults using multi-source heterogeneous data have shortcomings in terms of spatiotemporal alignment and data fusion, resulting in low accuracy in fault diagnosis. Summary of the Invention

[0005] The present invention aims to provide a method and computer device for monitoring the health status of industrial equipment based on multimodal data fusion, so as to improve the spatiotemporal alignment and signal fusion when diagnosing equipment faults through joint diagnosis of multi-source heterogeneous data, thereby improving the accuracy of fault diagnosis.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention provides a method for monitoring the health status of industrial equipment based on multimodal data fusion, comprising the following steps:

[0008] Step 1: Collect low-frequency modal data and high-frequency modal data during the operation of industrial equipment and perform preprocessing; the low-frequency modal data includes equipment temperature parameters, and the high-frequency modal data includes equipment vibration data and sound data;

[0009] Step 2: Input the preprocessed low-frequency modal data into the temporal extraction network to output low-frequency feature vectors; input the preprocessed high-frequency modal data into the two-dimensional convolutional neural network to output high-frequency feature vectors.

[0010] Step 3: Construct a spatiotemporal dual-stream attention mechanism module, including a condition-guided flow and an acoustic-vibration complementary flow; the condition-guided flow is: guided and aligned by temperature temporal features, and then calculates the enhanced vibration temporal features and sound temporal features; the acoustic-vibration complementary flow is: fused the enhanced vibration temporal features and sound temporal features into acoustic-vibration resonance features;

[0011] Step 4: Channel concatenation is performed on the temperature features, enhanced vibration time-series features, and acoustic resonance features. Feature compression is then performed through a global average pooling layer to extract the fusion context vector. The fusion context vector is input into a multilayer perceptron consisting of two fully connected layers and normalized using an activation function. The real-time confidence weights of each feature branch are calculated. Based on the calculated confidence weights, the features of the three branches are dynamically weighted to output the fused health features.

[0012] Step 5: Input the fused health features into the fully connected layer and classifier, and output the current health status label and fault type of the industrial equipment.

[0013] In step one, the preprocessing methods include: resampling and linear interpolation alignment of the low-frequency modal data, and normalization processing to obtain low-frequency sequence data; and using time window sliding segmentation and short-time Fourier transform to convert the high-frequency modal data into a time-frequency spectrum.

[0014] In step two, the temporal extraction network is composed of a cascaded one-dimensional convolutional neural network and a long short-term memory network.

[0015] In step three, after alignment guided by temperature time-series characteristics, the enhanced vibration time-series characteristics are calculated:

[0016] ;

[0017] ;

[0018] in The temperature feature vector, For vibration characteristic vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time mask matrix, The enhanced vibration timing characteristics.

[0019] In step three, after alignment guided by temperature timing features, the enhanced sound timing features are calculated. The calculation method is as follows:

[0020] ;

[0021] ;

[0022] in The temperature feature vector, For sound feature vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time masking matrix, This refers to the enhanced temporal characteristics of the sound.

[0023] In step three, the acoustic-vibration complementary flow is as follows: a mutual attention matrix is ​​constructed, and the enhanced vibration features and acoustic features are fused into acoustic-vibration resonance features. The calculation formula is:

[0024] ;

[0025] ;

[0026] in An attention matrix that complements sound and vibration. For activation function, For the mapping matrix, To enhance the vibration timing characteristics, To enhance the temporal characteristics of the sound, Scaling factor () indicates a splicing operation. This is a characteristic of acoustic resonance.

[0027] In step four, the formula for calculating the integrated health characteristics is as follows:

[0028] ;

[0029] in Temperature characteristics, To enhance the vibration timing characteristics, It is a characteristic of acoustic resonance. and These are the confidence weights for temperature characteristics, vibration enhancement characteristics, and acoustic resonance characteristics, respectively.

[0030] It also includes model optimization methods, employing a multi-task joint consistency constraint loss function for end-to-end collaborative optimization. The formula for the constraint loss function is:

[0031] ;

[0032] in Cross-entropy is used for fault classification loss. For cross-modal semantic contrast loss, For the monotonicity loss of the state degradation trajectory, For the balancing hyperparameters of the cross-modal semantic contrast loss term, This is the balancing hyperparameter for the monotonicity loss term of the state degradation trajectory.

[0033] The formula for calculating the monotonicity loss of the state degradation trajectory is:

[0034] ;

[0035] in () represents the distance metric function. For the fused health characteristics of the device at time t, Let t be the fused health characteristics of the device at the next time step. This is the baseline feature vector under normal operating conditions of the equipment.

[0036] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0037] The technical effects of this invention are as follows:

[0038] (1) This invention achieves efficient spatiotemporal alignment and semantic fusion of multi-source heterogeneous signals. Traditional methods, which use simple splicing, are prone to semantic fragmentation and the curse of dimensionality. This invention constructs a spatiotemporal dual-stream attention mechanism, which uses the macroscopic operating condition trend in the low-frequency temperature features to dynamically activate the abnormal impact segment in the high-frequency vibration and sound features. Combined with dynamic time masking, it achieves spatiotemporal coupling alignment of thermal equilibrium state and mechanical stress wave at the physical mechanism level for the first time, which greatly improves the depth of collaborative feature extraction of heterogeneous signals.

[0039] (2) This invention possesses highly practical software and hardware-level automatic fault tolerance capabilities. Industrial environments are harsh, and sensors often fail due to oil contamination, looseness, or electromagnetic interference. The adaptive collaborative dynamic gating network designed in this invention can extract the fusion context vector in real time and automatically evaluate the signal-to-noise ratio and signal quality of each sensor within milliseconds. Once a modality (such as sound) encounters sudden strong noise or hardware failure, the gating network can automatically approximate its confidence weight to 0 without affecting system operation, achieving extremely strong algorithm-level sensor fault isolation and system fault tolerance robustness.

[0040] (3) This invention significantly improves the accuracy of equipment fault diagnosis in high-noise environments. This invention introduces a mutual attention bottleneck structure in the acoustic-vibration complementary flow, making full use of the spatial complementary characteristics of vibration (structural conduction) and sound (air conduction); at the same time, it introduces cross-modal semantic contrast loss in the training phase, forcibly narrowing the mapping distance of heterogeneous signals under the same working conditions, thereby effectively stripping away the electromagnetic and sharp mechanical background noise in the industrial field, eliminating false feature interference, and significantly reducing the false alarm rate and false negative rate of faults in high-noise and complex variable working conditions.

[0041] (4) This invention possesses sensitive early-stage minor fault detection and long-term degradation early warning capabilities. Traditional threshold-based alarm methods cannot provide early warning for minor damage that has not yet developed into a visible fault. This invention innovatively introduces the state degradation trajectory monotonicity loss Ldeg, which, by applying first-order differential constraints to continuous time-series fusion features, makes the model extremely sensitive to the deviation trend from the feature space to the normal point throughout the entire life cycle. This enables the algorithm to accurately capture the irreversible degradation trajectory of equipment at the stage of minor damage such as minor bearing spalling and early slight wear, significantly extending the early warning cycle for predictive maintenance. Attached Figure Description

[0042] This manual includes the following figures, which illustrate the following:

[0043] Figure 1 The flowchart shows a method for monitoring the health status of industrial equipment based on multimodal data fusion and a computer device according to the present invention.

[0044] Figure 2 This is a schematic diagram of the network structure of the adaptive collaborative dynamic gating fusion module for industrial equipment health status monitoring based on multimodal data fusion, and computer equipment, according to the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, in order to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention, and to facilitate its implementation.

[0046] This invention provides a method for monitoring the health status of industrial equipment based on multimodal data fusion, comprising the following steps:

[0047] Step 1: Collect low-frequency modal data and high-frequency modal data during the operation of industrial equipment and perform preprocessing; the low-frequency modal data includes equipment temperature parameters, and the high-frequency modal data includes equipment vibration data and sound data;

[0048] Step 2: Input the preprocessed low-frequency modal data into the temporal extraction network to output low-frequency feature vectors; input the preprocessed high-frequency modal data into the two-dimensional convolutional neural network to output high-frequency feature vectors.

[0049] Step 3: Construct a spatiotemporal dual-stream attention mechanism module, including a condition-guided flow and an acoustic-vibration complementary flow; the condition-guided flow is: guided and aligned by temperature temporal features, and then calculates the enhanced vibration temporal features and sound temporal features; the acoustic-vibration complementary flow is: fused the enhanced vibration temporal features and sound temporal features into acoustic-vibration resonance features;

[0050] Step 4: Channel concatenation is performed on the temperature features, enhanced vibration time-series features, and acoustic resonance features. Feature compression is then performed through a global average pooling layer to extract the fusion context vector. The fusion context vector is input into a multilayer perceptron consisting of two fully connected layers and normalized using an activation function. The real-time confidence weights of each feature branch are calculated. Based on the calculated confidence weights, the features of the three branches are dynamically weighted to output the fused health features.

[0051] Step 5: Input the fused health features into the fully connected layer and classifier, and output the current health status label and fault type of the industrial equipment.

[0052] In step one, the preprocessing methods include: resampling and linear interpolation alignment of the low-frequency modal data, and normalization processing to obtain low-frequency sequence data; and using time window sliding segmentation and short-time Fourier transform to convert the high-frequency modal data into a time-frequency spectrum.

[0053] In step two, the temporal extraction network is composed of a cascaded one-dimensional convolutional neural network and a long short-term memory network.

[0054] In step three, after alignment guided by temperature time-series characteristics, the enhanced vibration time-series characteristics are calculated:

[0055] ;

[0056] ;

[0057] in The temperature feature vector, For vibration characteristic vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time mask matrix, The enhanced vibration timing characteristics.

[0058] In step three, after alignment guided by temperature timing features, the enhanced sound timing features are calculated. The calculation method is as follows:

[0059] ;

[0060] ;

[0061] in The temperature feature vector, For sound feature vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time masking matrix, This refers to the enhanced temporal characteristics of the sound.

[0062] In step three, the acoustic-vibration complementary flow is as follows: a mutual attention matrix is ​​constructed, and the enhanced vibration features and acoustic features are fused into acoustic-vibration resonance features. The calculation formula is:

[0063] ;

[0064] ;

[0065] in An attention matrix that complements sound and vibration. For activation function, For the mapping matrix, To enhance the vibration timing characteristics, To enhance the temporal characteristics of the sound, Scaling factor () indicates a splicing operation. This is a characteristic of acoustic resonance.

[0066] In step four, the formula for calculating the integrated health characteristics is as follows:

[0067] ;

[0068] in Temperature characteristics, To enhance the vibration timing characteristics, It is a characteristic of acoustic resonance. and These are the confidence weights for temperature characteristics, vibration enhancement characteristics, and acoustic resonance characteristics, respectively.

[0069] It also includes model optimization methods, employing a multi-task joint consistency constraint loss function for end-to-end collaborative optimization. The formula for the constraint loss function is:

[0070] ;

[0071] in Cross-entropy is used for fault classification loss. For cross-modal semantic contrast loss, For the monotonicity loss of the state degradation trajectory, For the balancing hyperparameters of the cross-modal semantic contrast loss term, This is the balancing hyperparameter for the monotonicity loss term of the state degradation trajectory.

[0072] The formula for calculating the monotonicity loss of the state degradation trajectory is:

[0073] ;

[0074] in () represents the distance metric function. For the fused health characteristics of the device at time t, Let t be the fused health characteristics of the device at the next time step. This is the baseline feature vector under normal operating conditions of the equipment.

[0075] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0076] The present invention will now be described in detail.

[0077] Low-frequency and high-frequency modal data of industrial equipment during operation are collected and preprocessed. Low-frequency modal data includes parameters such as equipment temperature, current, and rotational speed; in this invention, temperature is used as an example. High-frequency modal data includes equipment vibration and sound data. The preprocessing method includes: resampling and linear interpolation alignment of the low-frequency modal data, followed by normalization to obtain low-frequency sequence data; and time window sliding segmentation and short-time Fourier transform (STFT) of the high-frequency modal data to convert it into a time-frequency two-dimensional spectrum reflecting spatiotemporal characterization.

[0078] The preprocessed low-frequency modal data is input into a temporal extraction network composed of a one-dimensional convolutional neural network (1D-CNN) and a long short-term memory network (LSTM), which outputs a low-frequency feature vector. In this embodiment of the invention, the low-frequency feature vector is a temperature feature vector. The preprocessed high-frequency modal data (in this embodiment, the high-frequency modal data consists of vibration data and sound data) is input into a parallel two-dimensional convolutional neural network (2D-CNN) to decouple and extract high-dimensional local and global vibration feature vectors. and sound feature vector In the above, B represents the batch size, C represents the number of channels, and T represents the number of time-domain steps after alignment.

[0079] A spatiotemporal dual-stream attention mechanism module is constructed, including a condition-guided flow and an acoustic-vibration complementary flow, to eliminate semantic gaps caused by inconsistencies in the physical dimensions and sampling rates of temperature, vibration, and sound. The condition-guided flow is as follows: it uses temperature features to dynamically activate abnormal impact segments in high-frequency signals, aligns them with temperature temporal features, and calculates the enhanced vibration and sound temporal features.

[0080] After alignment guided by temperature time-series characteristics, the enhanced vibration time-series characteristics are calculated as follows:

[0081] ;

[0082] ;

[0083] in The temperature feature vector, For vibration characteristic vectors, This is the weight matrix. For activation function, Scaling factor It is a dynamic time mask matrix generated by the temperature change rate ΔT; when the temperature change exceeds the threshold, the mask of the corresponding time period is set to 1, the vibration feature attention weight of that time period is amplified, and the rest of the region is suppressed. This enhances the vibration temporal characteristics. When the rate of temperature change exceeds a preset threshold, the mask automatically amplifies the attention weight of the vibration characteristics within the corresponding time period, achieving spatial coupling alignment between the thermal equilibrium state and the mechanical stress wave. Similarly, temperature characteristics are used to guide the alignment interaction of sound characteristics, resulting in aligned enhanced sound characteristics.

[0084] After alignment guided by temperature timing features, the enhanced audio timing features are calculated as follows:

[0085] ;

[0086] ;

[0087] in The temperature feature vector, For sound feature vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time masking matrix, This refers to the enhanced temporal characteristics of the sound.

[0088] The acoustic-vibration complementary flow is achieved by fusing the enhanced vibration and sound temporal features into an acoustic-vibration resonance feature. Specifically, this involves constructing a mutual attention matrix and fusing the enhanced vibration and sound features into the acoustic-vibration resonance feature. The calculation formula is as follows:

[0089] ;

[0090] ;

[0091] in An attention matrix that complements sound and vibration. For activation function, For the mapping matrix, To enhance the vibration timing characteristics, To enhance the temporal characteristics of the sound, Scaling factor () indicates a splicing operation. This is a characteristic of acoustic resonance.

[0092] An Adaptive Co-gating Network is designed to achieve feature-level hardware denoising and automatic fault tolerance for sensors. Temperature features, enhanced vibration time-series features, and acoustic resonance features are concatenated into channels, and feature compression is performed using a global average pooling layer to extract the fused context vector.

[0093]

[0094] The fused context vector is input into a multilayer perceptron consisting of two fully connected layers, and normalized using the Softmax activation function to calculate the real-time confidence weights of each feature branch. :

[0095]

[0096] in () represents a multilayer perceptron consisting of two fully connected layers.

[0097] The features of the three branches are dynamically weighted based on the calculated confidence weights to output a fused health feature; the formula for calculating the fused health feature is as follows:

[0098] ;

[0099] in Temperature characteristics, To enhance the vibration timing characteristics, It is a characteristic of acoustic resonance. and These represent the confidence weights for temperature characteristics, vibration enhancement characteristics, and acoustic resonance characteristics, respectively. When a sensor becomes loose, disconnected, or experiences strong noise pollution, the network automatically approximates the weight coefficient of that fault mode to 0 through the distortion of the context vector XC, achieving fault isolation at the algorithm level.

[0100] The health features are fed into a fully connected layer and a Softmax classifier to output the current health status label (normal, early minor wear, serious failure) and failure type (inner ring failure, outer ring failure, gear tooth breakage) of the industrial equipment.

[0101] This invention also includes a model optimization method. During the model training phase, an end-to-end collaborative optimization is performed using a multi-task joint consistency constraint loss function. The formula for the constraint loss function is as follows:

[0102] ;

[0103] in Cross-entropy is used for fault classification loss. For cross-modal semantic contrast loss, the InfoNCE loss function is used to constrain the spatial mapping distance of temperature, vibration, and sound signals under the same working conditions, thereby eliminating spurious feature noise. To compensate for the monotonicity loss of the state degradation trajectory, by analyzing the continuous time series... Extract the first derivative and constrain its monotonicity as it deviates from the normal operating point. For the balancing hyperparameters of the cross-modal semantic contrast loss term, This is the balancing hyperparameter for the monotonicity loss term of the state degradation trajectory.

[0104] The formula for calculating the monotonicity loss of the state degradation trajectory is as follows:

[0105] ;

[0106] in () represents the distance metric function. For the fused health characteristics of the device at time t, Let t be the fused health characteristics of the device at the next time step. This represents the baseline feature vector under normal operating conditions. By utilizing this joint loss function to forcibly constrain the degradation trajectory of the feature space, we can achieve sensitive detection and early warning of very early, subtle damage to the equipment.

[0107] The beneficial effects of the present invention are described in detail below.

[0108] This invention achieves efficient spatiotemporal alignment and semantic fusion of multi-source heterogeneous signals. Traditional methods, which use simple splicing, are prone to semantic fragmentation and the curse of dimensionality. This invention constructs a spatiotemporal dual-stream attention mechanism, dynamically activating high-frequency vibrations and anomalous impact segments in sound features by utilizing macroscopic operating condition trends in low-frequency temperature features. Combined with dynamic time masks, this invention achieves, for the first time, spatiotemporal coupling alignment of thermal equilibrium state and mechanical stress wave at the physical mechanism level, significantly improving the depth of collaborative feature extraction from heterogeneous signals.

[0109] This invention possesses highly practical software and hardware-level automatic fault tolerance capabilities. Industrial environments are harsh, and sensors often fail due to oil contamination, looseness, or electromagnetic interference. The adaptive cooperative dynamic gating network designed in this invention can extract the fused context vector in real time and automatically evaluate the signal-to-noise ratio and signal quality of each sensor within milliseconds. If a modality (such as sound) encounters sudden strong noise or hardware failure, the gating network can automatically approximate its confidence weight to zero without affecting system operation, achieving extremely strong algorithm-level sensor fault isolation and system fault tolerance robustness.

[0110] This invention significantly improves the accuracy of equipment fault diagnosis in high-noise environments. It introduces a mutual attention bottleneck structure into the acoustic-vibration complementary flow, fully utilizing the spatial complementarity of vibration (structural conduction) and sound (air conduction). Simultaneously, it introduces cross-modal semantic contrast loss during the training phase, forcibly narrowing the mapping distance between heterogeneous signals under the same operating conditions. This effectively removes electromagnetic and sharp mechanical background noise from industrial environments, eliminates spurious feature interference, and significantly reduces the false alarm and false negative rates in high-noise and complex variable operating conditions.

[0111] This invention possesses sensitive early detection of minor faults and long-term degradation warning capabilities. Traditional threshold-based alarm methods cannot provide early warning for minor damage that has not yet developed into a visible fault. This invention innovatively introduces the state degradation trajectory monotonicity loss Ldeg, which, by applying first-order differential constraints to continuous temporal fusion features, makes the model extremely sensitive to the deviation trend from the normal point of the entire life cycle in the feature space. This enables the algorithm to accurately capture the irreversible degradation trajectory of equipment at the stage of minor damage such as minor bearing spalling and early slight wear, significantly extending the early warning cycle for predictive maintenance.

[0112] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A method for monitoring the health status of industrial equipment based on multimodal data fusion, characterized in that, Includes the following steps: Step 1: Collect low-frequency modal data and high-frequency modal data during the operation of industrial equipment and perform preprocessing; the low-frequency modal data includes equipment temperature parameters, and the high-frequency modal data includes equipment vibration data and sound data; Step 2: Input the preprocessed low-frequency modal data into the temporal extraction network to output low-frequency feature vectors; input the preprocessed high-frequency modal data into the two-dimensional convolutional neural network to output high-frequency feature vectors. Step 3: Construct a spatiotemporal dual-stream attention mechanism module, including a condition-guided flow and an acoustic-vibration complementary flow; the condition-guided flow is: guided and aligned by temperature temporal features, and then calculates the enhanced vibration temporal features and sound temporal features; the acoustic-vibration complementary flow is: fused the enhanced vibration temporal features and sound temporal features into acoustic-vibration resonance features; Step 4: Channel concatenation is performed on the temperature features, enhanced vibration time-series features, and acoustic resonance features. Feature compression is then performed through a global average pooling layer to extract the fusion context vector. The fusion context vector is input into a multilayer perceptron consisting of two fully connected layers and normalized using an activation function. The real-time confidence weights of each feature branch are calculated. Based on the calculated confidence weights, the features of the three branches are dynamically weighted to output the fused health features. Step 5: Input the fused health features into the fully connected layer and classifier, and output the current health status label and fault type of the industrial equipment.

2. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: In step one, the preprocessing methods include: resampling and linear interpolation alignment of the low-frequency modal data, and normalization processing to obtain low-frequency sequence data; and using time window sliding segmentation and short-time Fourier transform to convert the high-frequency modal data into a time-frequency spectrum.

3. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: In step two, the temporal extraction network is composed of a cascaded one-dimensional convolutional neural network and a long short-term memory network.

4. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: In step three, after alignment guided by temperature time-series characteristics, the enhanced vibration time-series characteristics are calculated: ； ； in The temperature feature vector, For vibration characteristic vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time mask matrix, The enhanced vibration timing characteristics.

5. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: In step three, after alignment guided by temperature timing features, the enhanced sound timing features are calculated. The calculation method is as follows: ； ； in The temperature feature vector, For sound feature vectors, This is the weight matrix. For activation function, Scaling factor For dynamic time masking matrix, This refers to the enhanced temporal characteristics of the sound.

6. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: In step three, the acoustic-vibration complementary flow is as follows: a mutual attention matrix is ​​constructed, and the enhanced vibration features and acoustic features are fused into acoustic-vibration resonance features. The calculation formula is: ； ； in An attention matrix that complements sound and vibration. For activation function, For the mapping matrix, To enhance the vibration timing characteristics, To enhance the temporal characteristics of the sound, Scaling factor () indicates a splicing operation. This is a characteristic of acoustic resonance.

7. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: In step four, the formula for calculating the integrated health characteristics is as follows: ； in Temperature characteristics, To enhance the vibration timing characteristics, It is a characteristic of acoustic resonance. and These are the confidence weights for temperature characteristics, vibration enhancement characteristics, and acoustic resonance characteristics, respectively.

8. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 1, characterized in that: It also includes model optimization methods, employing a multi-task joint consistency constraint loss function for end-to-end collaborative optimization. The formula for the constraint loss function is: ； in Cross-entropy is used for fault classification loss. For cross-modal semantic contrast loss, For the monotonicity loss of the state degradation trajectory, For the balancing hyperparameters of the cross-modal semantic contrast loss term, This is the balancing hyperparameter for the monotonicity loss term of the state degradation trajectory.

9. The industrial equipment health status monitoring method based on multimodal data fusion as described in claim 8, characterized in that: The formula for calculating the monotonicity loss of the state degradation trajectory is: ； in () represents the distance metric function. For the fused health characteristics of the device at time t, Let t be the fused health characteristics of the device at the next time step. This is the baseline feature vector under normal operating conditions of the equipment.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the method of any one of claims 1 to 9.

Images