A device status monitoring method and system based on multimodal data fusion

Abstract

The present invention discloses a device status monitoring method and system based on multimodal data fusion. The method collects numerical data of device temperature, strain, and vibration acceleration, as well as text data of operation records, and establishes an index table based on maintenance events to achieve temporal alignment of heterogeneous data. It then extracts dynamic features of the device using Fourier transforms and wavelet transforms, and uses an LSTM network to extract text semantic features. A multi-head cross-attention mechanism is designed to achieve deep feature fusion. CatBoost is used to construct a status assessment model, and KL divergence detection is introduced to dynamically update the model. The present invention solves technical difficulties in device status monitoring, such as the difficulty in aligning multi-source heterogeneous data, the difficulty in effectively fusing numerical and text features, and the difficulty of the model adapting to changes in data distribution. This improves the accuracy and reliability of status assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment status monitoring, and in particular to an equipment status monitoring method and system based on multimodal data fusion. Background Art

[0002] Equipment status monitoring is an important means to ensure the safe operation of industrial equipment. Traditional equipment status monitoring mainly relies on the analysis of numerical data such as vibration, temperature and strain collected by sensors. With the deepening of industrial digital transformation, the intelligence level of complex industrial equipment continues to improve. Modern industrial equipment not only generates a large amount of sensor data during operation, but also accumulates a large amount of text record data, such as operating status descriptions, parameter records, fault maintenance records, etc. These text data record important information such as the operating characteristics of the equipment, fault diagnosis information, maintenance experience, and expert knowledge. In industrial equipment status monitoring, how to effectively utilize these multimodal data has become a direction worthy of research. However, the following key technical difficulties still exist in the field of industrial equipment status monitoring:

[0003] First, the asynchrony and heterogeneity of multimodal data in the temporal dimension make it difficult to establish an accurate temporal mapping relationship between numerical sensor data and textual records, restricting the ability to conduct collaborative analysis of multimodal data. Specifically, sensor data is typically collected continuously at a high frequency, while textual records are discrete and irregular. This difference in acquisition characteristics poses a significant challenge to data alignment.

[0004] Secondly, there are significant differences in the representation space between device dynamic temporal features and text semantic features. The lack of effective feature interaction and semantic alignment mechanisms limits the deep integration of multimodal features. Numerical features are mainly reflected in physical properties in the frequency and time-frequency domains, while text features contain rich semantic information. Establishing the correlation between the two modal features is a major challenge.

[0005] Thirdly, the severely imbalanced distribution of equipment failure samples and normal samples in industrial scenarios reduces the ability of the condition assessment model to identify abnormal conditions. In actual operation, the proportion of normal equipment data often exceeds the proportion of failure data. This extreme class imbalance makes it difficult for the model to fully learn the characteristics of failures.

[0006] Finally, dynamic changes in equipment operating conditions lead to persistent drift in data distribution, impacting the model's generalization performance and predictive stability. For example, factors such as changes in equipment load, ambient temperature fluctuations, and component aging can all cause shifts in data distribution, and traditional static models struggle to adapt to these dynamic changes. These technical challenges hinder the practical application of multimodal equipment condition monitoring technology in industrial settings. Summary of the Invention

[0007] The problem to be solved by the present invention is how to provide a method and system for equipment status monitoring based on multimodal data fusion to achieve accurate assessment and early warning of equipment status. In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A device status monitoring method based on multimodal data fusion, characterized by comprising the following steps:

[0009] S1. Collect historical data on equipment operation, including numerical modal data consisting of temperature data, strain data, and vibration acceleration data collected by sensors, and textual modal data consisting of textual records describing operating status, operating parameter records, and maintenance records. An event index table is established based on the maintenance time in the maintenance records. Time alignment is performed on the numerical modal data and the textual modal data according to the event index table. A state label is constructed based on the maintenance time, and a historical training dataset is constructed.

[0010] S2. Preprocessing the historical training data set, including outlier correction and standardization of the numerical modal data, exception processing and vectorization of the text modal data, and establishing a standardized data processing process;

[0011] S3. Extracting multimodal features from the preprocessed historical training dataset and establishing a standardized feature extraction process, including extracting dynamic features from the numerical modality data, extracting semantic features from the text modality data, and fusing the dynamic features and the semantic features to generate a fused feature vector through a multi-head cross-attention mechanism;

[0012] S4. Constructing a state assessment model, including using a time series partitioning method to divide the fused feature vector and its corresponding state label into a training set, a validation set, and a test set in chronological order, training the training set with a classifier to obtain a state assessment model, determining a state determination threshold on the validation set, evaluating model performance on the test set, and determining the influence of each feature on the state assessment through feature contribution analysis;

[0013] S5. Deploy the state assessment model for online monitoring, including collecting real-time multimodal data of equipment operation, generating a real-time fused feature vector according to the standardized data processing process and the standardized feature extraction process, evaluating the equipment state through the state assessment model, and updating the model based on distribution drift detection.

[0014] A further technical solution is that the time alignment of the numerical modal data and the text modal data according to the event index table in S1 specifically includes:

[0015] Performing time alignment on the temperature data and the strain data using a linear interpolation method, and performing time alignment on the vibration acceleration data using a cubic spline interpolation method;

[0016] Based on the maintenance time in the maintenance record, the status label is constructed, and the historical data in a fixed time window before the maintenance time is marked as abnormal status, and the historical data in the remaining time periods is marked as normal status.

[0017] A further technical solution is that: the standardized data processing process established in S2 specifically includes: for the numerical modal data, interpolating and supplementing missing data segments less than a preset time length based on time series characteristics, marking missing data segments greater than the preset time length as invalid monitoring intervals, identifying and correcting outliers through a local weighted regression smoothing method, and normalizing the data to the [0, 1] interval using a maximum and minimum value normalization method;

[0018] For the text modal data, missing timestamps are supplemented by inferring the temporal relationship of context records, duplicate records are identified based on text similarity calculation and the earliest valid record is retained, a predefined pattern is used for format normalization, and a domain dictionary containing equipment component names and professional terms is established, wherein the domain dictionary includes status words describing equipment status, location words describing equipment locations, parameter value words describing equipment parameters and operation words describing operation behaviors, and a word embedding method is used to convert the text modal data into a feature vector.

[0019] A further technical solution is that the extraction of multimodal features in S3 specifically includes:

[0020] For the standardized numerical modal data, the main frequency and frequency band energy are extracted by Fourier transform to form frequency domain features, the energy distribution and scale coefficient are extracted by wavelet transform to form time-frequency features, and the frequency domain features and the time-frequency features are combined to form the dynamic features;

[0021] For the text modal data that has been vectorized, a long short-term memory network is used to encode the operation status description record text, the operation parameter record text, and the maintenance record text respectively, and status words, part words, parameter value words, and operation words in the text are extracted based on the domain dictionary to form a feature sequence to generate the semantic feature;

[0022] The multi-head cross-attention mechanism includes multiple feature interaction units, each of which uses the dynamic feature as a query vector and the semantic feature as a key-value vector to calculate the feature relevance to obtain an interaction weight, and concatenates the interaction weights on the feature dimension and transforms them through a fully connected layer to obtain the fused feature vector.

[0023] A further technical solution is that the construction of the state assessment model in S4 includes:

[0024] The state assessment model is constructed using a CatBoost classifier and trained using cross entropy as a loss function; a state determination threshold is selected on the validation set through ROC curve analysis, a predicted probability value greater than the state determination threshold is determined as an abnormal state, and a predicted probability value less than the state determination threshold is determined as a normal state; and the model performance is evaluated using an F1 score on the test set;

[0025] The SHAP value analysis is used to calculate the feature contribution, and the feature importance ranking is obtained by averaging the feature contribution to identify the key influencing factors that lead to abnormal equipment status.

[0026] A further technical solution is that: deploying the state assessment model for online monitoring in S5 includes: collecting real-time numerical modal data and text modal data, performing outlier correction and standardization processing on the real-time numerical modal data according to the standardized data processing process, and performing exception processing and vectorization processing on the real-time text modal data;

[0027] Processing the real-time numerical modal data and the real-time text modal data according to the standardized feature extraction method, extracting the dynamic features and the semantic features, and generating the real-time fused feature vector through the multi-head cross attention mechanism;

[0028] The real-time fusion feature vector is input into the state assessment model to perform state assessment and output abnormal warning. A further technical solution is that the model update based on distribution drift detection in S5 includes:

[0029] Determine a monitoring feature dimension according to the feature importance ranking, calculate the KL divergence between the real-time fusion feature vector and the training set on the monitoring feature dimension, and trigger a model update when the KL divergence exceeds a data distribution offset threshold;

[0030] A dual-model mechanism is used for updates, in which the original model continues to provide online services, while the model is retrained using an expanded training set containing newly added data, and the model is switched after the newly trained model is verified.

[0031] The present invention also discloses an equipment status monitoring system based on multimodal data fusion, which is characterized by comprising:

[0032] A multimodal data acquisition module is used to collect numerical modal data and text modal data during the operation of the equipment. The numerical modal data includes temperature data, strain data, and vibration acceleration data. The text modal data includes operation status description record text, operation parameter record text, and maintenance record text.

[0033] A multimodal data preprocessing module, used to implement standardized processing of the numerical modal data and the text modal data;

[0034] A multimodal feature extraction module, used to generate a fused feature vector, including a dynamic feature extraction unit, a semantic feature extraction unit, and a feature fusion unit;

[0035] The model deployment module is used to train and deploy status evaluation models based on historical data, including a data alignment unit, a model training unit, and a feature analysis unit;

[0036] The online monitoring module is used to realize real-time evaluation and early warning of equipment status, including a real-time feature generation unit, a status evaluation unit and a model update unit, wherein the model update unit adopts a dual-model mechanism to realize online update of the model.

[0037] A further technical solution is that the online monitoring module further includes:

[0038] Distribution drift detection unit, used to monitor real-time data distribution changes and trigger model updates;

[0039] The status warning unit is used to generate and send warning information based on the status assessment results.

[0040] The beneficial effects of the present invention are as follows:

[0041] The multi-head cross attention mechanism proposed in this invention realizes the deep semantic fusion of numerical modality and text modality through the innovative design of using dynamic features as query vectors and semantic features as key-value vectors. It can automatically learn the correlation weights between different features, significantly improving the feature expression ability and the accuracy of state recognition. It adopts a multimodal data alignment method based on event indexing, constructs an index table with maintenance events as anchors, and adopts differentiated interpolation strategies for different types of data, effectively solving the temporal correlation problem of heterogeneous data and improving the accuracy of data annotation. The designed dual-modal feature extraction method combines Fourier transform and wavelet transform to extract the time series of heterogeneous data. Dynamic features are extracted by using the LSTM network to extract sequence features such as status words and part words in the text, achieving comprehensive capture of device status features and effectively retaining temporal and semantic information; combining the CatBoost classifier with multimodal feature fusion can effectively optimize the processing of mixed features, solve the sample imbalance problem, and accurately evaluate feature importance, thereby improving the performance of the status assessment model; distribution drift detection and dual-model update mechanism based on KL divergence realizes real-time detection of data distribution changes and dynamic update of the model, improving the continuous adaptability of the model while ensuring uninterrupted service, and enhancing the reliability of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the equipment status monitoring method based on multimodal data fusion.

[0043] Figure 2 Construct a flow chart for the condition assessment model.

[0044] Figure 3 Flowchart of the dual-model update mechanism.

[0045] Figure 4 This is the structural diagram of the equipment status monitoring system based on multimodal data fusion. DETAILED DESCRIPTION

[0046] like Figure 1 As shown, the main steps of the method provided by the present invention are as follows:

[0047] (1) Collecting historical data of equipment operation, the historical data includes numerical modal data consisting of temperature data, strain data, and vibration acceleration data collected by sensors, and text modal data consisting of operation status description record text, operation parameter record text, and maintenance record text, and establishing an event index table based on the maintenance time in the maintenance record, performing time alignment on the numerical modal data and the text modal data according to the event index table, constructing a state label based on the maintenance time, and constructing a historical training data set;

[0048] (2) preprocessing the historical training data set, including performing outlier correction and standardization on the numerical modal data, performing exception processing and vectorization on the text modal data, and establishing a standardized data processing process;

[0049] (3) extracting multimodal features from the preprocessed historical training data set and establishing a standardized feature extraction process, including extracting dynamic features from the numerical modal data, extracting semantic features from the text modal data, and fusing the dynamic features and the semantic features to generate a fused feature vector through a multi-head cross attention mechanism;

[0050] (4) constructing a state assessment model, including using a time series partitioning method to divide the fused feature vector and its corresponding state label into a training set, a validation set, and a test set in chronological order, using a classifier to train the training set to obtain a state assessment model, determining a state judgment threshold on the validation set, evaluating the model performance on the test set, and determining the degree of influence of each feature on the state assessment through feature contribution analysis;

[0051] (5) Deploying the state assessment model for online monitoring, including collecting real-time multimodal data of equipment operation, generating real-time fusion feature vectors according to the standardized data processing process and the standardized feature extraction process, evaluating the equipment state through the state assessment model, and updating the model based on distribution drift detection.

[0052] As a preferred embodiment of the present invention, in step (1), the process of data acquisition and time alignment is as follows: (1-1) Numerical modal data acquisition

[0053] Temperature, strain, and vibration acceleration data are collected using sensors placed at key locations on the equipment. Preferably, the sampling frequency of the temperature sensor is 1 Hz, the sampling frequency of the strain sensor is 10 Hz, and the sampling frequency of the vibration acceleration sensor is 1000 Hz. The temperature data is used to reflect the temperature variation trends of various equipment components, the strain data is used to characterize the stress state of the equipment structure, and the vibration acceleration data is used to describe the dynamic response characteristics of the equipment.

[0054] (1-2) Text modality data collection

[0055] Obtain operating status description records, operating parameter records, and maintenance records from the equipment operation management system. The operating status description records contain the operator's observations of the equipment's operating status, the operating parameter records contain the process and environmental parameters during equipment operation, and the maintenance records contain information such as the time, repair location, cause of failure, and treatment measures for equipment overhaul and maintenance. Preferably, the equipment overhaul and maintenance time includes the specific start and end times of the repair.

[0056] (1-3) Time alignment method based on event index

[0057] An event index table is established based on the maintenance time recorded in the maintenance record text, and the maintenance event is used as the key time node. The event index table contains information such as the maintenance event ID, maintenance start time, maintenance end time, and maintenance location. The temperature data and the strain data are time-aligned using the linear interpolation method, and the vibration acceleration data are time-aligned using the cubic spline interpolation method. Preferably, all data are first unified to the same time base, the lowest sampling frequency (1Hz) is selected as the base sampling rate, and the data with high sampling rate is downsampled. When downsampling, for strain data (10Hz), the average value is taken for every 10 data points; for vibration acceleration data (1000Hz), cubic spline interpolation is first performed to maintain signal continuity, and then sampling is performed at the 1Hz time point. In this way, data with different sampling frequencies are aligned to the same sampling rate in the time dimension. Preferably, the historical data of 72 hours before the maintenance time is marked as abnormal, and the historical data of the remaining time periods are marked as normal.

[0058] (1-4) Construction of historical training dataset

[0059] The time-aligned numerical modal data is combined with the text modal data of the corresponding time period to construct a historical training data set containing multimodal data. Each piece of training data includes temperature data, strain data, vibration acceleration data, operating status description record text, operating parameter record text and maintenance record text within the same time window, as well as a status label based on the maintenance time. Preferably, 1 hour is used as the basic time unit for data segmentation, the numerical modal data is segmented, and the text records within the time period are associated with the corresponding data segments. Specifically, the text data is mapped to the corresponding 1-hour time window through the timestamp of the text record. If there are multiple text records of the same type in a certain time window, these records are sorted by timestamp and merged; if a certain type of text record is missing in a certain time window, it is marked with a null value.

[0060] As a preferred embodiment of the present invention, in step (2), data preprocessing includes:

[0061] (2-1) Correction of outliers in numerical modal data

[0062] For numerical modal data, the problem of missing data is first dealt with. Missing data segments that are less than the preset time length are interpolated and supplemented based on the time series characteristics, and missing data segments that are greater than the preset time length are marked as invalid monitoring intervals. Preferably, the preset time length is 1 hour. For the identification and correction of outliers, a local weighted regression smoothing method is used. Preferably, a sliding window size of 24 data points is first selected, and a quadratic polynomial is used to perform local fitting on the data in the window; then the weight is calculated based on the distance from the data point to the fitting point, and the farther the distance, the smaller the weight; finally, the data points that deviate from the fitting curve by more than 3 times the standard deviation are identified as outliers and replaced with the corresponding fitting values.

[0063] (2-2) Standardization of numerical modal data

[0064] The numerical modal data is normalized to the interval [0, 1] using the maximum and minimum value normalization method. Preferably, the temperature data, strain data, and vibration acceleration data are standardized separately, and the normalization parameters are established based on the maximum and minimum values ​​obtained from historical data statistics for subsequent normalization of real-time data. Preferably, for data outside the historical range, the normalization parameters are updated using the following dynamic update mechanism: when the maximum value of the new data exceeds the current maximum value, the maximum value is updated to the new maximum value; when the minimum value of the new data is less than the current minimum value, the minimum value is updated to the new minimum value. After each parameter update, the entire data set is re-normalized using the new normalization parameters.

[0065] (2-3) Exception handling of text modal data

[0066] Missing timestamps are supplemented by inferring temporal relationships in contextual records. Duplicate records are identified based on text similarity calculations, and the earliest valid record is retained. Format normalization is performed using predefined patterns. Preferably, the specific steps for supplementing missing timestamps by inferring temporal relationships in contextual records are: first, calculating the typical time interval between records based on the timestamps of adjacent records; then, for records with missing timestamps, inferring the timestamps of the preceding and succeeding records and the typical time interval based on their position in the record sequence; and finally, verifying the rationality of the inferred timestamps to ensure that they meet temporal constraints. Preferably, text similarity is calculated using an edit distance algorithm, with a similarity threshold set at 0.9. Records exceeding this threshold are considered duplicates. Incomplete or abnormally formatted text records are normalized based on predefined text templates. Preferably, predefined text templates include: the "[time]-[location]-[status description]" template for device status descriptions; the "[time]-[parameter name]-[parameter value]-[unit]" template for operating parameter records; and the "[time]-[maintenance type]-[location]-[fault symptom]-[treatment measure]" template for maintenance records.

[0067] (2-4) Vectorized processing of text modal data

[0068] A domain dictionary containing equipment component names and specialized terminology is established, and word embedding methods are used to convert text modal data into feature vectors. Preferably, the domain dictionary includes the names of core equipment components, common fault types, and maintenance operation terminology. Preferably, the Word2Vec model is used as the word embedding method, with the word vector dimension set to 100. During the initialization phase, word vectors pre-trained with a general Chinese corpus are used for initialization. During the fine-tuning phase, word vectors are trained for 50 iterations using historical equipment maintenance records, with a learning rate set to 0.001 to adapt to the specific semantic representation of the equipment monitoring domain. The generated word vectors can reflect the semantic similarity between words in the equipment domain. For example, the word vectors for "bearing" and "sleeve" have a high cosine similarity. Preferably, the specific steps for text vectorization are: first, calculate the TF-IDF value for each word in the text, where TF represents the frequency of occurrence of the word in the current text, and IDF represents the inverse document frequency of the word in the entire document set; then, multiply each word's word vector by its corresponding TF-IDF value; finally, sum all weighted word vectors and divide by the number of word vectors to obtain the vectorized representation of the text.

[0069] As a preferred embodiment of the present invention, in step (3), feature extraction and fusion include:

[0070] (3-1) Dynamic feature extraction method

[0071] When extracting features from normalized numerical modal data, since normalization does not change the signal's frequency characteristics and relative amplitude relationships, frequency domain and time-frequency analysis can be performed directly. Preferably, the normalized data is first subjected to noise reduction preprocessing before feature extraction. Dynamic features are extracted from the numerical modal data through frequency domain analysis and time-frequency analysis. In frequency domain analysis, the dominant frequency and frequency band energy are extracted using Fourier transform to form frequency domain features. Preferably, based on the operating characteristics of the equipment, the spectrum is divided into a low-frequency band (0-10 Hz) for capturing fundamental frequency features, a mid-frequency band (10-100 Hz) for identifying mechanical fault characteristics, and a high-frequency band (100-500 Hz) for extracting high-frequency fault information. The energy contribution and dominant frequency amplitude of each band are calculated. In a specific embodiment of the present invention, the frequency band energy contribution is 15% in the low-frequency band, 45% in the mid-frequency band, and 40% in the high-frequency band, indicating the presence of clear mid- and high-frequency fault characteristics. In time-frequency analysis, the energy distribution and scaling coefficients are extracted using wavelet transform to form time-frequency features. Preferably, a db4 wavelet basis function is used to perform a 4-layer decomposition to extract energy coefficients and wavelet coefficients at each scale. The frequency domain features and the time-frequency features are combined to form a dynamic feature vector for characterizing the dynamic response characteristics of the device.

[0072] (3-2) Semantic feature extraction method

[0073] Based on the word vector representation obtained by the pre-order processing, the text sequence is further encoded using a long short-term memory network to capture word order relationships and context dependencies. Specifically, the word vector sequence is input into the long short-term memory network in chronological order, and the operating status description record text, operating parameter record text and maintenance record text are processed respectively. Preferably, the hidden layer dimension of the bidirectional LSTM is set to 128 dimensions, and the context relationship is captured through forward and backward state transfer to output the context representation of the text sequence. Preferably, a specific attention layer is constructed for each type of text record, and its structure is as follows: the input layer dimension is the same as the LSTM output dimension (128 dimensions), the middle layer is set with 64 neurons, and the output layer generates attention weights through the softmax function. The attention layer learns the importance of different words to the equipment status through training. For example, fault-related terms such as "abnormal", "wear", and "overheating" obtain higher weights. Preferably, after attention weighting, the 128-dimensional LSTM output is compressed into a 64-dimensional feature vector as the final semantic feature representation of this type of text. (3-3) Multi-head cross attention mechanism

[0074] The multi-head cross attention mechanism includes multiple feature interaction units, each of which uses dynamic features as query vectors and semantic features as key-value vectors to calculate feature relevance to obtain interaction weights, and concatenates each interaction weight on the feature dimension and transforms it through a fully connected layer to obtain a fused feature vector. Preferably, 8 attention heads are set, and the output dimension of each attention head is 64. Specifically, for each attention head, the dot product of the query vector and the key vector is first calculated to obtain the original attention score, and then normalized by the scaling factor and passed through the softmax function to obtain the attention weight, and finally the attention weight is multiplied by the value vector to obtain the weighted feature. After splicing the outputs of all attention heads, the final fused feature vector is obtained by dimensionality reduction through a fully connected layer. The vector contains both the dynamic operating characteristics of the device and the semantic information of the text description. In a specific embodiment of the present invention, the final dimension of the fused feature vector is 512 (8 heads × 64 dimensions), of which the first 256 dimensions mainly reflect the dynamic feature pattern of the device, and the last 256 dimensions mainly reflect the semantic features of the text description. For example, in a bearing failure case, the dynamic feature part is characterized by an increase in the proportion of high-frequency energy, while the semantic feature part emphasizes the weight of keywords such as "bearing", "vibration", and "abnormal sound". Figure 2 , in step (4), the model building process includes:

[0075] (4-1) Status Tag Construction Method

[0076] Based on the maintenance time, the status label is constructed, and the historical data within the fixed time window before the maintenance time is marked as abnormal state (labeled as 1), and the historical data of the remaining time periods are marked as normal state (labeled as 0). Preferably, in order to deal with the problem of sample imbalance, the weight enhancement method is used for abnormal samples during model training. Specifically, the sample weight coefficient w = N normal / N abnormal is set, where N normal and N abnormal are the number of normal samples and abnormal samples, respectively. For example, when the ratio of normal samples to abnormal samples is 9:1, the weight coefficient of the abnormal sample is set to 9, so that the model pays more attention to abnormal samples during training.

[0077] (4-2) Dataset Partitioning Strategy

[0078] Using a time series partitioning method, the historical training dataset containing the fused feature vectors and their corresponding state labels is divided into a training set, a validation set, and a test set in chronological order. Preferably, the partitioning is done in a 6:2:2 ratio, with the first 60% of the data used as the training set, the middle 20% as the validation set, and the last 20% as the test set.

[0079] (4-3) CatBoost classifier training process

[0080] A CatBoost classifier is used to construct a state assessment model, using the fused feature vector obtained in the previous step as input and cross entropy as the loss function for training. Optimally, based on the characteristics of the equipment state assessment task, the main parameters of the model are set as follows: a tree depth of 6, which ensures the model's expressiveness while avoiding overfitting; a learning rate of 0.03, as a smaller learning rate facilitates stable model convergence; 1000 iterations, and an early stopping mechanism. Training is terminated after 50 consecutive rounds of performance improvement on the validation set. This 50-round setting effectively avoids the impact of local fluctuations during the training process.

[0081] (4-4) Method for determining the state judgment threshold

[0082] The state determination threshold is selected by ROC curve analysis on the validation set. The predicted probability value greater than the state determination threshold is determined as an abnormal state, and the predicted probability value less than the state determination threshold is determined as a normal state. By balancing the accuracy and recall rate of the model, the probability value that maximizes the F1 score is selected as the state determination threshold. The model performance is evaluated by the F1 score on the test set to verify the effectiveness of the state determination threshold. Specifically, the ROC curve plots the relationship curve of the true positive rate (recall rate) to the false positive rate by adjusting the determination threshold. Preferably, the optimal threshold is determined by calculating the F1 score (the harmonic mean of the accuracy and recall rate) under different thresholds. In a specific embodiment of the present invention, a maximum F1 score of 0.85 is obtained when the threshold is set to 0.7, at which time the accuracy is 0.83 and the recall rate is 0.87.

[0083] (4-5) Feature Importance Analysis Method

[0084] The SHAP value analysis is used to calculate the feature contribution, and the feature importance ranking is obtained by averaging the feature contribution, which is used to identify the key influencing factors that lead to abnormal equipment status. Preferably, the contribution of each feature to the prediction result of each sample is calculated based on the SHAP value, and the direction and degree of influence of the feature on the model prediction are displayed by a visualization method. In a specific embodiment of the present invention, the top three feature importance rankings are: high-frequency energy proportion (SHAP value 0.35), bearing temperature (SHAP value 0.28), and vibration amplitude (SHAP value 0.25), indicating that these features have an important influence on the equipment status assessment, and the direction and degree of influence of the feature on the model prediction are displayed by a visualization method (such as a SHAP waterfall chart).

[0085] As a preferred embodiment of the present invention, in step (5), online monitoring implementation includes:

[0086] (5-1) Real-time data acquisition and processing

[0087] Real-time numerical modal data and text modal data are collected and processed according to standardized data processing procedures. For real-time numerical modal data, outlier correction and standardization are performed using the same methods as historical data, including interpolation of missing data and identification and correction of outliers. For real-time text modal data, exception processing and vectorization processing are performed, including timestamp supplementation, duplicate record identification and text normalization. Preferably, a data cache time of 5 minutes is set to ensure data integrity and continuity. During the data cache period, the system can complete data acquisition, transmission and preprocessing while ensuring real-time requirements.

[0088] (5-2) Real-time feature generation process

[0089] Real-time numerical modal data and real-time text modal data are processed according to standardized feature extraction methods to extract dynamic features and semantic features, and a real-time fused feature vector is generated through a multi-head cross-attention mechanism. Preferably, a sliding window method is used for feature extraction, with a window length of 1 hour (consistent with the training time) and a step size of 5 minutes, that is, the window is slid every 5 minutes and the features are updated. For the extracted features, quality checks are performed by checking statistical indicators such as the range and variance of the feature values. If an anomaly is found, an alarm is triggered and the most recent valid feature is used.

[0090] (5-3) Status Assessment and Early Warning Methods

[0091] The real-time fused feature vector is input into the state assessment model for state assessment and output of anomaly warnings. Preferably, the system issues an anomaly warning when the anomaly probability value output by the model exceeds the state determination threshold. The warning information includes the anomaly probability value, the primary feature causing the anomaly (the feature with the highest contribution, identified by the SHAP value), and its current value, providing a reference for equipment maintenance personnel in making decisions.

[0092] (5-4) Distribution drift detection method

[0093] The monitoring feature dimensions are determined based on feature importance ranking. The KL divergence between the real-time fused feature vector and the training set on the monitoring feature dimensions is calculated. A model update is triggered when the KL divergence exceeds the data distribution shift threshold. Preferably, the top five features by feature importance are selected as monitoring dimensions, and a sliding window with a 48-hour window size is used to calculate the feature distribution. The window size is set based on the periodicity of equipment operation (typically 24 hours) and data volume requirements. The 48-hour window size encompasses two complete operating cycles, ensuring the reliability of feature distribution calculation. Distribution drift is determined to have occurred when the KL divergence for five consecutive windows (i.e., 240 hours) exceeds a preset data distribution shift threshold. In one specific embodiment of the present invention, analysis of monitoring data from a production facility showed that when the data distribution shift threshold was set to 0.3, the system was able to effectively identify significant changes in the equipment's operating mode. This threshold was selected based on empirical data analysis: a KL divergence less than 0.3 typically indicates fluctuations in normal equipment operation, while a KL divergence greater than 0.3 often indicates a significant change in equipment operating conditions. For example, after a key component of the equipment is replaced, the KL divergence of the feature distribution exceeds 0.3 for five consecutive windows, triggering the model update mechanism to adapt to the new operating status of the equipment in a timely manner.

[0094] (5-5) Dual-model update mechanism

[0095] Reference Figure 3 A dual-model mechanism is used for updates, where the original model continues to provide online services while the model is retrained using an expanded training set containing newly added data. After the newly trained model is verified, a model switch is performed. Preferably, the expanded training set includes the most recent month of the original training data and the newly added drift data. The new model's performance indicators on the validation set must exceed those of the original model and must have passed testing on the most recent online data before a model switch is performed. Preferably, a 24-hour smooth transition strategy is used for model switching. The 24-hour transition period is chosen based on the complete equipment operation cycle, allowing for a smooth model switch within a full production cycle. During the switch, the prediction results of the new and old models are linearly weighted and fused over time: as time progresses, the weight of the new model increases linearly from 0 to 1, while the weight of the old model decreases linearly from 1 to 0. This gradual switching approach avoids sudden changes in prediction results caused by model switching. Through this online monitoring implementation, an adaptive equipment status monitoring system is constructed that can promptly detect and respond to changes in data distribution and maintain the model's predictive performance.

[0096] like Figure 4 As shown, the present invention also provides an equipment status monitoring system based on multimodal data fusion, including a multimodal data acquisition module, a multimodal data preprocessing module, a multimodal feature extraction module, a model deployment module and an online monitoring module, and each module is connected through a data bus.

[0097] The multimodal data acquisition module is used to collect numerical modal data and textual modal data during equipment operation. Numerical modal data includes temperature data, strain data, and vibration acceleration data collected by sensors; textual modal data includes operating status description records, operating parameter records, and maintenance records.

[0098] The multimodal data preprocessing module is used to achieve standardized processing of numerical modal data and textual modal data. It includes a numerical data processing unit and a textual data processing unit, which preprocess the numerical modal data and textual modal data respectively to ensure data quality.

[0099] The multimodal feature extraction module is used to generate a fused feature vector and includes a dynamic feature extraction unit, a semantic feature extraction unit, and a feature fusion unit. The dynamic feature extraction unit is used to extract frequency domain and time-frequency features from numerical modal data; the semantic feature extraction unit is used to extract semantic features from text modal data; and the feature fusion unit achieves feature fusion through a multi-head cross-attention mechanism.

[0100] The model deployment module is used to train and deploy a state assessment model based on historical data. It includes a data alignment unit, a model training unit, and a feature analysis unit. The data alignment unit implements temporal alignment of multimodal data, the model training unit completes training of the state assessment model, and the feature analysis unit is responsible for feature importance analysis.

[0101] The online monitoring module, designed to provide real-time assessment and early warning of device status, includes a real-time feature generation unit, a status assessment unit, and a model update unit. The real-time feature generation unit processes real-time data and generates features, while the status assessment unit performs status assessment and early warning. The model update unit utilizes a dual-model mechanism to enable online model updates. The module also includes a distribution drift detection unit, which monitors changes in real-time data distribution and triggers model updates, and a status early warning unit, which generates and sends early warning information based on the status assessment results.

[0102] Preferably, a distributed architecture is adopted for deployment between modules, and data transmission is achieved through message queues to ensure the real-time performance and reliability of the system.

Claims

1. A device status monitoring method based on multimodal data fusion, characterized in that: The method comprises the following steps: S1. Collect historical data on equipment operation, including numerical modal data consisting of temperature data, strain data, and vibration acceleration data collected by sensors, and textual modal data consisting of textual records describing operating status, operating parameter records, and maintenance records. An event index table is established based on the maintenance time in the maintenance records. Time alignment is performed on the numerical modal data and the textual modal data according to the event index table. A state label is constructed based on the maintenance time, and a historical training dataset is constructed. S2. Preprocessing the historical training data set, including outlier correction and standardization of the numerical modal data, exception processing and vectorization of the text modal data, and establishing a standardized data processing process; S3. Extracting multimodal features from the preprocessed historical training dataset and establishing a standardized feature extraction process, including extracting dynamic features from the numerical modality data, extracting semantic features from the text modality data, and fusing the dynamic features and the semantic features to generate a fused feature vector through a multi-head cross-attention mechanism; S4. Constructing a state assessment model, including using a time series partitioning method to divide the fused feature vector and its corresponding state label into a training set, a validation set, and a test set in chronological order, training the training set with a classifier to obtain a state assessment model, determining a state determination threshold on the validation set, evaluating model performance on the test set, and determining the influence of each feature on the state assessment through feature contribution analysis; S5. Deploy the state assessment model for online monitoring, including collecting real-time multimodal data of equipment operation, generating a real-time fused feature vector according to the standardized data processing process and the standardized feature extraction process, evaluating the equipment state through the state assessment model, and updating the model based on distribution drift detection.

2. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The step of performing time alignment on the numerical modal data and the text modal data according to the event index table in S1 specifically includes: Performing time alignment on the temperature data and the strain data using a linear interpolation method, and performing time alignment on the vibration acceleration data using a cubic spline interpolation method; Based on the maintenance time in the maintenance record, the status label is constructed, and the historical data in a fixed time window before the maintenance time is marked as abnormal status, and the historical data in the remaining time periods is marked as normal status.

3. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The standardized data processing process established in S2 specifically includes: For the numerical modal data, missing data segments shorter than a preset time length are interpolated and supplemented based on time series characteristics, missing data segments longer than the preset time length are marked as invalid monitoring intervals, outliers are identified and corrected using a local weighted regression smoothing method, and the data are normalized to the [0, 1] interval using a maximum and minimum value normalization method; For the text modal data, missing timestamps are supplemented by inferring the temporal relationship of context records, duplicate records are identified based on text similarity calculation and the earliest valid record is retained, a predefined pattern is used for format normalization, and a domain dictionary containing equipment component names and professional terms is established, wherein the domain dictionary includes status words describing equipment status, location words describing equipment locations, parameter value words describing equipment parameters and operation words describing operation behaviors, and a word embedding method is used to convert the text modal data into a feature vector.

4. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The multimodal feature extraction in S3 specifically includes: For the standardized numerical modal data, the main frequency and frequency band energy are extracted by Fourier transform to form frequency domain features, the energy distribution and scale coefficient are extracted by wavelet transform to form time-frequency features, and the frequency domain features and the time-frequency features are combined to form the dynamic features; For the text modal data that has been vectorized, a long short-term memory network is used to encode the operation status description record text, the operation parameter record text, and the maintenance record text respectively, and status words, part words, parameter value words, and operation words in the text are extracted based on a domain dictionary to form a feature sequence to generate the semantic feature; The multi-head cross-attention mechanism includes multiple feature interaction units, each of which uses the dynamic feature as a query vector and the semantic feature as a key-value vector to calculate the feature relevance to obtain an interaction weight, and concatenates the interaction weights on the feature dimension and transforms them through a fully connected layer to obtain the fused feature vector.

5. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The state assessment model constructed in S4 includes: CatBoost classifier is used to construct the state assessment model, and cross entropy is used as the loss function for training; Selecting a state determination threshold by ROC curve analysis on the validation set, determining a predicted probability value greater than the state determination threshold as an abnormal state, and determining a predicted probability value less than the state determination threshold as a normal state, and evaluating model performance by F1 score on the test set; The SHAP value analysis is used to calculate the feature contribution, and the feature importance ranking is obtained by averaging the feature contribution to identify the key influencing factors that lead to abnormal equipment status.

6. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: Deploying the status assessment model for online monitoring in S5 includes: Collecting the real-time numerical modal data and the text modal data, performing outlier correction and standardization processing on the real-time numerical modal data according to the standardized data processing flow, and performing outlier processing and vectorization processing on the real-time text modal data; Processing the real-time numerical modal data and the real-time text modal data according to the standardized feature extraction process, extracting the dynamic features and the semantic features, and generating the real-time fused feature vector through the multi-head cross attention mechanism; The real-time fused feature vector is input into the state assessment model to perform state assessment and output an abnormality warning.

7. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The model update based on distribution drift detection in S5 includes: Determine the monitoring feature dimension according to the feature importance ranking, calculate the KL divergence of the real-time fusion feature vector and the training set on the monitoring feature dimension, and trigger a model update when the KL divergence exceeds a data distribution offset threshold; A dual-model mechanism is used for updates, in which the original model continues to provide online services, while the model is retrained using an expanded training set containing newly added data, and the model is switched after the newly trained model is verified.

8. An equipment status monitoring system based on multimodal data fusion, characterized in that: include: A multimodal data acquisition module is used to collect numerical modal data and text modal data during the operation of the equipment. The numerical modal data includes temperature data, strain data, and vibration acceleration data. The text modal data includes operation status description record text, operation parameter record text, and maintenance record text. A multimodal data preprocessing module, used to implement standardized processing of the numerical modal data and the text modal data; A multimodal feature extraction module, used to generate a fused feature vector, including a dynamic feature extraction unit, a semantic feature extraction unit, and a feature fusion unit; The model deployment module is used to train and deploy status evaluation models based on historical data, including a data alignment unit, a model training unit, and a feature analysis unit; The online monitoring module is used to realize real-time evaluation and early warning of equipment status, including a real-time feature generation unit, a status evaluation unit and a model update unit, wherein the model update unit adopts a dual-model mechanism to realize online update of the model.

9. The device status monitoring system based on multimodal data fusion according to claim 8, characterized in that: The online monitoring module also includes: Distribution drift detection unit, used to monitor real-time data distribution changes and trigger model updates; The status warning unit is used to generate and send warning information based on the status assessment results.

Images