Equipment state monitoring method and system based on multi-modal data fusion

Abstract

The invention discloses an equipment state monitoring method and system based on multi-modal data fusion. According to the method, numerical data of equipment temperature, strain and vibration acceleration and operation record text data are acquired, an index table is established based on a maintenance event to realize heterogeneous data time sequence alignment, equipment dynamic features are extracted by combining Fourier transform and wavelet transform, and text semantic features are extracted by using an LSTM network. And designing a multi-head cross attention mechanism to realize deep feature fusion, constructing a state evaluation model by adopting CatBoost, and introducing KL divergence detection to dynamically update the model. The technical problems that in equipment state monitoring, multi-source heterogeneous data is difficult to align, numerical features and text features are difficult to effectively fuse, and a model is difficult to adapt to data distribution changes are solved, and the accuracy and reliability of state evaluation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment status monitoring, and in particular to an equipment status monitoring method and system based on multimodal data fusion. Background Art

[0002] Equipment status monitoring is an important means to ensure the safe operation of industrial equipment. Traditional equipment status monitoring mainly relies on the analysis of numerical data such as vibration, temperature and strain collected by sensors. With the deepening of industrial digital transformation, the intelligence level of complex industrial equipment continues to improve. Modern industrial equipment not only generates a large amount of sensor data during operation, but also accumulates a large amount of text record data, such as operating status descriptions, parameter records, fault maintenance records, etc. These text data record important information such as the operating characteristics of the equipment, fault diagnosis information, maintenance experience, and expert knowledge. In industrial equipment status monitoring, how to effectively use these multimodal data has become a direction worthy of research. However, the following key technical difficulties still exist in the field of industrial equipment status monitoring:

[0003] First, the asynchrony and heterogeneity of multimodal data in the time dimension make it difficult to establish an accurate time-series mapping relationship between numerical sensor data and text record data, which restricts the collaborative analysis capabilities of multimodal data. Specifically, sensor data is usually collected continuously at a high frequency, while text records are discrete and irregular. This difference in collection characteristics poses a huge challenge to data alignment;

[0004] Secondly, there are significant differences in the representation space between the dynamic time series features of the device and the semantic features of the text. The lack of effective feature interaction and semantic alignment mechanisms limits the deep fusion of multimodal features. Numerical features are mainly reflected in the physical characteristics of the frequency domain and time-frequency domain, while text features contain rich semantic information. How to establish the correlation between the two modal features is an important difficulty;

[0005] Thirdly, the severely unbalanced distribution characteristics of equipment failure samples and normal samples in industrial scenarios reduce the ability of the state assessment model to identify abnormal conditions. In actual operation, the proportion of normal operation data of equipment often exceeds the proportion of failure data. This extreme class imbalance problem makes it difficult for the model to fully learn the fault characteristics;

[0006] Finally, the dynamic changes in equipment operating conditions lead to continuous drift in data distribution, affecting the generalization performance and prediction stability of the model. For example, factors such as equipment load changes, ambient temperature fluctuations, and component aging can cause changes in data distribution, and traditional static models are difficult to adapt to such dynamic changes. These technical difficulties restrict the actual application of multimodal equipment status monitoring technology in industrial sites. Summary of the invention

[0007] The problem to be solved by the present invention is how to provide a method and system for equipment status monitoring based on multimodal data fusion to achieve accurate evaluation and early warning of equipment status. In order to solve the above technical problem, the technical solution adopted by the present invention is:

[0008] A device status monitoring method based on multimodal data fusion, characterized in that it comprises the following steps:

[0009] S1. Collect historical data of equipment operation, the historical data includes numerical modal data consisting of temperature data, strain data, and vibration acceleration data collected by sensors, and text modal data consisting of operation status description record text, operation parameter record text, and maintenance record text, and establish an event index table based on the maintenance time in the maintenance record, perform time alignment on the numerical modal data and the text modal data according to the event index table, construct a state label based on the maintenance time, and construct a historical training data set;

[0010] S2. Preprocessing the historical training data set, including performing outlier correction and standardization on the numerical modal data, performing outlier processing and vectorization on the text modal data, and establishing a standardized data processing process;

[0011] S3. extracting multimodal features from the preprocessed historical training data set and establishing a standardized feature extraction process, including extracting dynamic features from the numerical modal data, extracting semantic features from the text modal data, and fusing the dynamic features and the semantic features to generate a fused feature vector through a multi-head cross attention mechanism;

[0012] S4. constructing a state assessment model, including using a time series partitioning method to divide the fused feature vector and its corresponding state label into a training set, a validation set and a test set in chronological order, using a classifier to train the training set to obtain a state assessment model, determining a state judgment threshold on the validation set, evaluating model performance on the test set, and determining the influence of each feature on the state assessment through feature contribution analysis;

[0013] S5. Deploy the state assessment model for online monitoring, including collecting real-time multimodal data of equipment operation, generating a real-time fused feature vector according to the standardized data processing process and the standardized feature extraction process, evaluating the equipment state through the state assessment model, and updating the model based on distribution drift detection.

[0014] A further technical solution is that: the time alignment of the numerical modal data and the text modal data according to the event index table in S1 specifically includes:

[0015] The temperature data and the strain data are time-aligned using a linear interpolation method, and the vibration acceleration data are time-aligned using a cubic spline interpolation method;

[0016] Based on the maintenance time in the maintenance record, the status label is constructed, and the historical data in a fixed time window before the maintenance time is marked as an abnormal state, and the historical data in the remaining time periods are marked as a normal state.

[0017] A further technical solution is that: the standardized data processing flow established in S2 specifically includes: for the numerical modal data, the missing data segments shorter than the preset time length are interpolated and supplemented based on the time series characteristics, the missing data segments longer than the preset time length are marked as invalid monitoring intervals, and the outliers are identified and corrected by the local weighted regression smoothing method, and the data are normalized to the [0, 1] interval by the maximum and minimum value normalization method;

[0018] For the text modal data, missing timestamps are supplemented by inferring the time relationship of context records, duplicate records are identified based on text similarity calculation and the earliest valid record is retained, a predefined pattern is used for format normalization, and a domain dictionary containing equipment component names and professional terms is established, wherein the domain dictionary includes status words describing equipment status, location words describing equipment locations, parameter value words describing equipment parameters and operation words describing operation behaviors, and a word embedding method is used to convert the text modal data into a feature vector.

[0019] A further technical solution is that the multimodal feature extraction in S3 specifically includes:

[0020] For the numerical modal data that has been standardized, the main frequency and the frequency band energy are extracted by Fourier transform to form frequency domain features, the energy distribution and the scale coefficient are extracted by wavelet transform to form time-frequency features, and the frequency domain features and the time-frequency features are combined to form the dynamic features;

[0021] For the text modal data that has been vectorized, the operation status description record text, the operation parameter record text and the maintenance record text are respectively encoded using a long short-term memory network, and the state words, part words, parameter value words and operation words in the text are extracted based on the domain dictionary to form a feature sequence to generate the semantic feature;

[0022] The multi-head cross-attention mechanism includes multiple feature interaction units, each of which uses the dynamic feature as a query vector and the semantic feature as a key-value vector to calculate the feature relevance to obtain an interaction weight, and concatenates the interaction weights on the feature dimension and transforms them through a fully connected layer to obtain the fused feature vector.

[0023] A further technical solution is that: constructing the state assessment model in S4 includes:

[0024] The state assessment model is constructed by using CatBoost classifier, and cross entropy is used as the loss function for training; a state determination threshold is selected by ROC curve analysis on the validation set, a predicted probability value greater than the state determination threshold is determined as an abnormal state, and a predicted probability value less than the state determination threshold is determined as a normal state, and the model performance is evaluated by F1 score on the test set;

[0025] The SHAP value analysis is used to calculate the feature contribution, and the feature importance ranking is obtained by averaging the feature contribution to identify the key influencing factors that lead to abnormal status of the equipment.

[0026] A further technical solution is that: the deployment of the state assessment model for online monitoring in S5 includes: collecting the real-time numerical modal data and the text modal data, performing outlier correction and standardization processing on the real-time numerical modal data according to the standardized data processing flow, and performing exception processing and vectorization processing on the real-time text modal data;

[0027] Processing the real-time numerical modality data and the real-time text modality data according to the standardized feature extraction method, extracting the dynamic features and the semantic features, and generating the real-time fused feature vector through the multi-head cross attention mechanism;

[0028] The real-time fusion feature vector is input into the state assessment model to perform state assessment and output abnormal warning. A further technical solution is that the model update based on distribution drift detection in S5 includes:

[0029] Determine the monitoring feature dimension according to the feature importance ranking, calculate the KL divergence of the real-time fusion feature vector and the training set on the monitoring feature dimension, and trigger the model update when the KL divergence exceeds the data distribution deviation threshold;

[0030] A dual-model mechanism is used for updating, in which the original model continues to provide online services, while the model is retrained using an expanded training set containing new data, and the model is switched after the newly trained model is verified.

[0031] The present invention also discloses an equipment status monitoring system based on multimodal data fusion, which is characterized by comprising:

[0032] A multimodal data acquisition module, used to collect numerical modal data and text modal data during the operation of the equipment, wherein the numerical modal data includes temperature data, strain data and vibration acceleration data, and the text modal data includes operation status description record text, operation parameter record text and maintenance record text;

[0033] A multimodal data preprocessing module, used to implement standardized processing of the numerical modal data and the textual modal data;

[0034] A multimodal feature extraction module, used to generate a fused feature vector, including a dynamic feature extraction unit, a semantic feature extraction unit and a feature fusion unit;

[0035] A model deployment module, which is used to train and deploy state evaluation models based on historical data, including a data alignment unit, a model training unit, and a feature analysis unit;

[0036] The online monitoring module is used to realize real-time evaluation and early warning of equipment status, including a real-time feature generation unit, a status evaluation unit and a model updating unit, wherein the model updating unit adopts a dual-model mechanism to realize online updating of the model.

[0037] A further technical solution is that the online monitoring module further includes:

[0038] Distribution drift detection unit, used to monitor real-time data distribution changes and trigger model updates;

[0039] The status warning unit is used to generate and send warning information based on the status assessment results.

[0040] The beneficial effects of the present invention are as follows:

[0041] The multi-head cross-attention mechanism proposed in the present invention realizes the deep semantic fusion of numerical mode and text mode through the innovative design of taking dynamic features as query vectors and semantic features as key-value vectors, and can automatically learn the correlation weights between different features, which significantly improves the feature expression ability and the accuracy of state recognition. It adopts a multimodal data alignment method based on event indexing, builds an index table with maintenance events as anchor points, and adopts differentiated interpolation strategies for different types of data, which effectively solves the temporal correlation problem of heterogeneous data and improves the accuracy of data annotation. The designed bimodal feature extraction method combines Fourier transform and wavelet transform Dynamic features are extracted by using the LSTM network to extract sequence features such as status words and part words in the text, which realizes the comprehensive capture of equipment status features and effectively retains the timing and semantic information; the CatBoost classifier is combined with multimodal feature fusion to effectively optimize the processing of mixed features, solve the sample imbalance problem, and accurately evaluate the importance of features, thereby improving the performance of the status assessment model; the distribution drift detection and dual-model update mechanism based on KL divergence realizes real-time detection of data distribution changes and dynamic update of the model, which improves the continuous adaptability of the model while ensuring uninterrupted service and enhances the reliability of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the equipment status monitoring method based on multimodal data fusion.

[0043] Figure 2 Construct a flow chart for the condition assessment model.

[0044] Figure 3 Flowchart of the dual model update mechanism.

[0045] Figure 4 This is the structure diagram of the equipment status monitoring system based on multimodal data fusion. DETAILED DESCRIPTION

[0046] like Figure 1 As shown, the main steps of the method provided by the present invention are as follows:

[0047] (1) Collecting historical data of equipment operation, the historical data includes numerical modal data consisting of temperature data, strain data, and vibration acceleration data collected by sensors, and text modal data consisting of operation status description record text, operation parameter record text, and maintenance record text, and establishing an event index table based on the maintenance time in the maintenance record, performing time alignment on the numerical modal data and the text modal data according to the event index table, constructing a state label based on the maintenance time, and constructing a historical training data set;

[0048] (2) preprocessing the historical training data set, including performing outlier correction and standardization processing on the numerical modality data, performing outlier processing and vectorization processing on the text modality data, and establishing a standardized data processing flow;

[0049] (3) extracting multimodal features from the preprocessed historical training data set and establishing a standardized feature extraction process, including extracting dynamic features from the numerical modality data, extracting semantic features from the text modality data, and fusing the dynamic features and the semantic features to generate a fused feature vector through a multi-head cross attention mechanism;

[0050] (4) constructing a state assessment model, including using a time series partitioning method to divide the fused feature vector and its corresponding state label into a training set, a validation set, and a test set in chronological order, using a classifier to train the training set to obtain a state assessment model, determining a state judgment threshold on the validation set, evaluating model performance on the test set, and determining the influence of each feature on the state assessment through feature contribution analysis;

[0051] (5) deploying the state assessment model for online monitoring, including collecting real-time multimodal data of equipment operation, generating a real-time fused feature vector according to the standardized data processing process and the standardized feature extraction process, evaluating the equipment state through the state assessment model, and updating the model based on distribution drift detection.

[0052] As a preferred embodiment of the present invention, in step (1), the process of data acquisition and time alignment is as follows: (1-1) Numerical modal data acquisition

[0053] Temperature data, strain data and vibration acceleration data are collected through sensors arranged at key parts of the equipment. Preferably, the sampling frequency of the temperature sensor is 1 Hz, the sampling frequency of the strain sensor is 10 Hz, and the sampling frequency of the vibration acceleration sensor is 1000 Hz. The temperature data is used to reflect the temperature change trend of each component of the equipment, the strain data is used to characterize the stress state of the equipment structure, and the vibration acceleration data is used to describe the dynamic response characteristics of the equipment.

[0054] (1-2) Text modality data collection

[0055] The operation status description record text, operation parameter record text and maintenance record text are obtained from the equipment operation management system. Among them, the operation status description record text contains the operator's observation record of the equipment operation status, the operation parameter record text contains the process parameters and environmental parameters during the equipment operation process, and the maintenance record text contains information such as the time of equipment maintenance, maintenance location, cause of failure and treatment measures. Preferably, the time of equipment maintenance includes the specific maintenance start time and end time.

[0056] (1-3) Time alignment method based on event index

[0057] An event index table is established according to the maintenance time recorded in the maintenance record text, and the maintenance event is used as a key time node. The event index table contains information such as maintenance event ID, maintenance start time, maintenance end time, and maintenance location. The temperature data and the strain data are time-aligned using a linear interpolation method, and the vibration acceleration data is time-aligned using a cubic spline interpolation method. Preferably, all data are first unified to the same time base, the lowest sampling frequency (1Hz) is selected as the base sampling rate, and the data with a high sampling rate is downsampled. When downsampling, for strain data (10Hz), the average value is taken for every 10 data points; for vibration acceleration data (1000Hz), cubic spline interpolation is first performed to maintain the continuity of the signal, and then sampling is performed at a time point of 1Hz. In this way, data with different sampling frequencies are aligned to the same sampling rate in the time dimension. Preferably, the historical data 72 hours before the maintenance time is marked as abnormal, and the historical data of the remaining time periods are marked as normal.

[0058] (1-4) Construction of historical training data set

[0059] The time-aligned numerical modal data is combined with the text modal data of the corresponding time period to construct a historical training data set containing multimodal data. Each piece of training data includes temperature data, strain data, vibration acceleration data, operating status description record text, operating parameter record text and maintenance record text in the same time window, as well as a status label based on the maintenance time annotation. Preferably, 1 hour is used as the basic time unit for data segmentation, the numerical modal data is segmented, and the text records in the time period are associated with the corresponding data segments. Specifically, the text data is mapped to the corresponding 1-hour time window through the timestamp of the text record. If there are multiple text records of the same type in a certain time window, these records are sorted and merged by timestamp; if a certain type of text record is missing in a certain time window, it is marked with a null value.

[0060] As a preferred embodiment of the present invention, in step (2), data preprocessing includes:

[0061] (2-1) Correction of outliers in numerical modal data

[0062] For numerical modal data, the problem of missing data is dealt with first. Missing data segments that are shorter than the preset time length are interpolated and supplemented based on the timing characteristics, and missing data segments that are longer than the preset time length are marked as invalid monitoring intervals. Preferably, the preset time length is 1 hour. For the identification and correction of outliers, a local weighted regression smoothing method is used. Preferably, a sliding window size of 24 data points is first selected, and a quadratic polynomial is used to perform local fitting on the data in the window; then the weight is calculated based on the distance from the data point to the fitting point, and the farther the distance, the smaller the weight; finally, data points that deviate from the fitting curve by more than 3 times the standard deviation are identified as outliers, and replaced with the corresponding fitting values.

[0063] (2-2) Standardization of numerical modal data

[0064] The maximum and minimum value normalization method is used to normalize the numerical modal data to the interval [0, 1]. Preferably, the temperature data, strain data and vibration acceleration data are standardized respectively, and the standardized parameters are established based on the maximum and minimum values ​​obtained from the historical data statistics, which are used for the subsequent standardized processing of real-time data. Preferably, for data beyond the historical range, the standardized parameters are updated using the following dynamic update mechanism: when the maximum value of the new data exceeds the current maximum value, the maximum value is updated to the new maximum value; when the minimum value of the new data is less than the current minimum value, the minimum value is updated to the new minimum value. After each parameter update, the entire data set is re-normalized using the new standardized parameters.

[0065] (2-3) Exception handling of text modal data

[0066] The missing timestamps are supplemented by inferring the temporal relationship of the context records, duplicate records are identified based on text similarity calculation and the earliest valid record is retained, and the format is normalized using a predefined pattern. Preferably, the specific steps of supplementing the missing timestamps by inferring the temporal relationship of the context records are: first, the typical recording time interval is calculated based on the timestamps of adjacent records; then, for the records with missing timestamps, based on their position in the record sequence, the timestamps of the previous and next records and the typical time interval are inferred; finally, the rationality of the inferred timestamp is verified to ensure that it meets the timing constraints. Preferably, the text similarity is calculated using the edit distance algorithm, and the similarity threshold is set to 0.9. Records exceeding the threshold are regarded as duplicate records. For incomplete or abnormally formatted text records, normalization is performed based on predefined text templates. Preferably, the predefined text templates include: equipment status description template "[time]-[location]-[status description]", operation parameter record template "[time]-[parameter name]-[parameter value]-[unit]", and maintenance record template "[time]-[maintenance type]-[location]-[fault phenomenon]-[handling measures]".

[0067] (2-4) Vectorization of Text Modality Data

[0068] A domain dictionary containing equipment component names and professional terms is established, and the text modal data is converted into a feature vector using a word embedding method. Preferably, the domain dictionary includes the names of the core components of the equipment, common fault types, and maintenance operation terms. Preferably, the Word2Vec model is used as the word embedding method, and the word vector dimension is set to 100 dimensions. In the initialization stage, the word vector pre-trained with the general Chinese corpus is used for initialization; in the fine-tuning stage, the equipment history maintenance record text is used for training for 50 iterations, and the learning rate is set to 0.001 to adapt to the specific semantic representation of the equipment monitoring field. The generated word vector can reflect the semantic similarity relationship between words in the equipment field, for example, the word vector cosine similarity of "bearing" and "sleeve" is high. Preferably, the specific steps of text vectorization are: first, the TF-IDF value of each word in the text is calculated, where TF represents the frequency of occurrence of the word in the current text, and IDF represents the inverse document frequency of the word in the entire document set; then the word vector of each word is multiplied by its corresponding TF-IDF value; finally, all weighted word vectors are summed and divided by the number of word vectors to obtain the vectorized representation of the text.

[0069] As a preferred embodiment of the present invention, in step (3), feature extraction and fusion include:

[0070] (3-1) Dynamic feature extraction method

[0071] When extracting features from normalized numerical modal data, since normalization does not change the frequency characteristics and relative amplitude relationship of the signal, frequency domain and time-frequency analysis can be performed directly. Preferably, for the normalized data, noise reduction preprocessing is first performed, and then feature extraction is performed. For numerical modal data, dynamic features are extracted through frequency domain analysis and time-frequency analysis. In frequency domain analysis, the main frequency and frequency band energy are extracted through Fourier transform to form frequency domain features. Preferably, based on the operating characteristics of the equipment, the spectrum is divided into a low frequency band (0-10Hz) for capturing fundamental frequency features, a mid-frequency band (10-100Hz) for identifying mechanical fault features, and a high frequency band (100-500Hz) for extracting high-frequency fault information, and the energy proportion and main frequency amplitude of each frequency band are calculated. In a specific embodiment of the present invention, the frequency band energy proportions are: 15% in the low frequency band, 45% in the mid-frequency band, and 40% in the high frequency band, indicating that there are obvious mid- and high-frequency fault features. In time-frequency analysis, energy distribution and scale coefficients are extracted through wavelet transform to form time-frequency features. Preferably, the db4 wavelet basis function is used to perform 4-layer decomposition to extract energy coefficients and wavelet coefficients at each scale. The frequency domain features and the time-frequency features are combined to form a dynamic feature vector for characterizing the dynamic response characteristics of the device.

[0072] (3-2) Semantic feature extraction method

[0073] Based on the word vector representation obtained by the previous processing, the text sequence is further encoded using a long short-term memory network to capture the word order relationship and context dependency. Specifically, the word vector sequence is input into the long short-term memory network in chronological order, and the operation status description record text, operation parameter record text and maintenance record text are processed respectively. Preferably, the hidden layer dimension of the bidirectional LSTM is set to 128 dimensions, and the context relationship is captured through forward and backward state transfer to output the context representation of the text sequence. Preferably, a specific attention layer is constructed for each type of text record, and its structure is as follows: the input layer dimension is the same as the LSTM output dimension (128 dimensions), 64 neurons are set in the middle layer, and the output layer generates attention weights through the softmax function. The attention layer learns the importance of different words to the equipment status through training. For example, fault-related terms such as "abnormal", "wear", and "overheating" obtain higher weights. Preferably, after attention weighting, the 128-dimensional LSTM output is compressed into a 64-dimensional feature vector as the final semantic feature representation of this type of text. (3-3) Multi-head cross attention mechanism

[0074] The multi-head cross attention mechanism includes multiple feature interaction units, each of which uses dynamic features as query vectors and semantic features as key-value vectors to calculate feature relevance to obtain interaction weights, and concatenates each interaction weight on the feature dimension and transforms it through a fully connected layer to obtain a fused feature vector. Preferably, 8 attention heads are set, and the output dimension of each attention head is 64. Specifically, for each attention head, the dot product of the query vector and the key vector is first calculated to obtain the original attention score, and then the attention weight is obtained by normalizing it through a scaling factor and passing it through a softmax function, and finally the attention weight is multiplied by the value vector to obtain a weighted feature. After concatenating the outputs of all attention heads, the final fused feature vector is obtained by dimensionality reduction through a fully connected layer, which contains both the dynamic operating characteristics of the device and the semantic information of the text description. In a specific embodiment of the present invention, the final dimension of the fused feature vector is 512 (8 heads × 64 dimensions), of which the first 256 dimensions mainly reflect the dynamic feature mode of the device, and the last 256 dimensions mainly reflect the semantic features of the text description. For example, in a bearing failure case, the dynamic feature part is highlighted by the increase in the proportion of high-frequency energy, while the semantic feature part emphasizes the weight of keywords such as "bearing", "vibration" and "abnormal sound". Figure 2 , in step (4), the model building process includes:

[0075] (4-1) Status label construction method

[0076] Construct status labels based on maintenance time, mark historical data within a fixed time window before the maintenance time as abnormal (labeled as 1), and mark historical data in other time periods as normal (labeled as 0). Preferably, in order to deal with the problem of sample imbalance, the abnormal samples are weighted during model training. Specifically, the sample weight coefficient w=Nnormal / Nabnormal is set, where Nnormal and Nabnormal are the number of normal samples and abnormal samples, respectively. For example, when the ratio of normal samples to abnormal samples is 9:1, the weight coefficient of abnormal samples is set to 9, so that the model pays more attention to abnormal samples during training.

[0077] (4-2) Dataset Partitioning Strategy

[0078] The time series partitioning method is used to divide the historical training data set containing the fused feature vector and its corresponding state label into a training set, a validation set and a test set in chronological order. Preferably, the partitioning is performed in a ratio of 6:2:2, that is, the first 60% of the data is used as the training set, the middle 20% of the data is used as the validation set, and the last 20% of the data is used as the test set.

[0079] (4-3) CatBoost classifier training process

[0080] The state assessment model is constructed using CatBoost classifier, and the fused feature vector obtained in the previous step is used as input, and the cross entropy is used as the loss function for training. Preferably, based on the characteristics of the equipment state assessment task, the main parameters of the model are set as follows: the depth of the tree is 6, which ensures the expressiveness of the model while avoiding overfitting, the learning rate is 0.03, and a smaller learning rate helps the model to converge stably, the number of iterations is 1000, and the early stopping mechanism is used. When the performance does not improve for 50 consecutive rounds on the validation set, the training is stopped. The setting of 50 rounds can effectively avoid the influence of local fluctuations in the training process.

[0081] (4-4) Method for determining state judgment threshold

[0082] The state determination threshold is selected by ROC curve analysis on the validation set. The predicted probability value greater than the state determination threshold is determined as an abnormal state, and the predicted probability value less than the state determination threshold is determined as a normal state. By balancing the accuracy and recall of the model, the probability value that maximizes the F1 score is selected as the state determination threshold. The model performance is evaluated by the F1 score on the test set to verify the effectiveness of the state determination threshold. Specifically, the ROC curve plots the relationship curve of the true positive rate (recall rate) to the false positive rate by adjusting the determination threshold. Preferably, the optimal threshold is determined by calculating the F1 score (the harmonic mean of the accuracy and recall rate) under different thresholds. In a specific embodiment of the present invention, when the threshold is set to 0.7, the maximum F1 score of 0.85 is obtained, at which time the accuracy is 0.83 and the recall rate is 0.87.

[0083] (4-5) Feature importance analysis method

[0084] The SHAP value analysis is used to calculate the feature contribution, and the feature importance ranking is obtained by averaging the feature contribution, which is used to identify the key influencing factors that cause abnormal equipment status. Preferably, the contribution of each feature to the prediction result of each sample is calculated based on the SHAP value, and the direction and degree of influence of the feature on the model prediction are displayed by a visualization method. In a specific embodiment of the present invention, the top three feature importance rankings are: high-frequency band energy proportion (SHAP value 0.35), bearing temperature (SHAP value 0.28), and vibration amplitude (SHAP value 0.25), indicating that these features have an important influence on the equipment status assessment, and the direction and degree of influence of the feature on the model prediction are displayed by a visualization method (such as a SHAP waterfall chart).

[0085] As a preferred embodiment of the present invention, in step (5), online monitoring implementation includes:

[0086] (5-1) Real-time data collection and processing

[0087] Collect real-time numerical modal data and text modal data, and process them according to standardized data processing procedures. For real-time numerical modal data, use the same method as historical data to perform outlier correction and standardization, including interpolation of missing data and identification and correction of outliers. For real-time text modal data, perform exception processing and vectorization processing, including timestamp supplementation, duplicate record identification and text normalization. Preferably, set a data cache time of 5 minutes to ensure data integrity and continuity. During the data cache period, the system can complete data acquisition, transmission and preprocessing while ensuring real-time requirements.

[0088] (5-2) Real-time feature generation process

[0089] According to the standardized feature extraction method, real-time numerical modal data and real-time text modal data are processed to extract dynamic features and semantic features, and a real-time fused feature vector is generated through a multi-head cross-attention mechanism. Preferably, a sliding window method is used for feature extraction, with a window length of 1 hour (consistent with training) and a step length of 5 minutes, that is, the window is slid every 5 minutes and the features are updated. For the extracted features, quality checks are performed by checking statistical indicators such as the range and variance of the feature values. If an abnormality is found, an alarm is triggered and the most recent valid feature is used.

[0090] (5-3) Status assessment and early warning methods

[0091] The real-time fused feature vector is input into the state assessment model for state assessment and output of abnormal warning. Preferably, when the abnormal probability value output by the model exceeds the state judgment threshold, the system issues an abnormal warning. The warning information includes the abnormal probability value, the main feature causing the abnormality (the feature with the greatest contribution based on SHAP value identification) and its current value, providing a decision-making reference for equipment maintenance personnel.

[0092] (5-4) Distribution drift detection method

[0093] According to the feature importance ranking, the monitoring feature dimension is determined, and the KL divergence of the real-time fusion feature vector and the training set on the monitoring feature dimension is calculated. When the KL divergence exceeds the data distribution offset threshold, the model update is triggered. Preferably, the top 5 features in terms of feature importance are selected as the monitoring dimension, and a sliding window is set to calculate the feature distribution, and the window size is 48 hours. The setting of the window size takes into account the periodicity of the equipment operation (usually 24 hours) and the data volume requirements. The selection of 48 hours can include two complete operation cycles to ensure the reliability of the feature distribution calculation. When the KL divergence of 5 consecutive windows (i.e., 240 hours) exceeds the preset data distribution offset threshold, it is determined that distribution drift occurs. In a specific embodiment of the present invention, the monitoring data analysis of a certain production equipment shows that when the data distribution offset threshold is set to 0.3, the system can effectively identify significant changes in the equipment operation mode. The selection of this threshold is based on empirical data analysis: a KL divergence less than 0.3 usually indicates fluctuations in the normal operation of the equipment, and a KL divergence greater than 0.3 often means that the equipment operating conditions have changed significantly. For example, after a key component of the equipment is replaced, the KL divergence of the feature distribution exceeds 0.3 for five consecutive windows, triggering the model update mechanism to adapt to the new operating status of the equipment in a timely manner.

[0094] (5-5) Dual Model Update Mechanism

[0095] Reference Figure 3 , a dual-model mechanism is used for updating, in which the original model continues to provide online services, while the model is retrained using an expanded training set containing new data, and the model is switched after the new training model is verified. Preferably, the expanded training set includes the latest month's data of the original training data and the newly added drift data. The performance index of the new model on the verification set must exceed that of the original model, and the model is switched only after it is tested on the latest online data. Preferably, the model switching adopts a 24-hour smooth transition strategy. The selection of 24 hours as the transition period is based on the complete cycle of equipment operation. The smooth switching of the model can be completed within a complete production cycle. During the switching period, the prediction results of the new and old models are linearly weighted and fused according to time: as time goes on, the weight of the new model increases linearly from 0 to 1, and the weight of the old model decreases linearly from 1 to 0 accordingly. This progressive switching method can avoid the sudden change of prediction results caused by model switching. Through the above online monitoring implementation scheme, an equipment status monitoring system with adaptive capabilities is constructed, which can timely discover and respond to changes in data distribution and maintain the prediction performance of the model.

[0096] like Figure 4 As shown, the present invention also provides an equipment status monitoring system based on multimodal data fusion, including a multimodal data acquisition module, a multimodal data preprocessing module, a multimodal feature extraction module, a model deployment module and an online monitoring module, and each module is connected through a data bus.

[0097] The multimodal data acquisition module is used to collect numerical modal data and text modal data during the operation of the equipment. Among them, the numerical modal data includes temperature data, strain data and vibration acceleration data collected by sensors; the text modal data includes operation status description record text, operation parameter record text and maintenance record text.

[0098] The multimodal data preprocessing module is used to realize the standardized processing of numerical modal data and text modal data. It includes a numerical data processing unit and a text data processing unit, which preprocess the numerical modal data and text modal data respectively to ensure data quality.

[0099] The multimodal feature extraction module is used to generate a fused feature vector, and includes a dynamic feature extraction unit, a semantic feature extraction unit, and a feature fusion unit. The dynamic feature extraction unit is used to extract frequency domain and time-frequency features from numerical modal data; the semantic feature extraction unit is used to extract semantic features from text modal data; and the feature fusion unit realizes feature fusion through a multi-head cross attention mechanism.

[0100] The model deployment module is used to train and deploy a state assessment model based on historical data, and includes a data alignment unit, a model training unit, and a feature analysis unit. The data alignment unit implements time alignment of multimodal data, the model training unit completes the training of the state assessment model, and the feature analysis unit is responsible for feature importance analysis.

[0101] The online monitoring module is used to realize real-time evaluation and early warning of equipment status, and includes a real-time feature generation unit, a status evaluation unit and a model update unit. The real-time feature generation unit processes real-time data and generates features, the status evaluation unit performs status evaluation and early warning, and the model update unit uses a dual-model mechanism to realize online update of the model. The module also includes a distribution drift detection unit and a status early warning unit, wherein the distribution drift detection unit is used to monitor the distribution changes of real-time data and trigger model updates, and the status early warning unit is used to generate and send early warning information based on the status evaluation results.

[0102] Preferably, a distributed architecture is adopted between the modules, and data transmission is achieved through message queues to ensure the real-time and reliability of the system.

Claims

1. A device status monitoring method based on multimodal data fusion, characterized in that: The method comprises the following steps: S1. Collect historical data of equipment operation, the historical data includes numerical modal data consisting of temperature data, strain data, and vibration acceleration data collected by sensors, and text modal data consisting of operation status description record text, operation parameter record text, and maintenance record text, and establish an event index table based on the maintenance time in the maintenance record, perform time alignment on the numerical modal data and the text modal data according to the event index table, construct a state label based on the maintenance time, and construct a historical training data set; S2. Preprocessing the historical training data set, including performing outlier correction and standardization on the numerical modal data, performing outlier processing and vectorization on the text modal data, and establishing a standardized data processing process; S3. extracting multimodal features from the preprocessed historical training data set and establishing a standardized feature extraction process, including extracting dynamic features from the numerical modal data, extracting semantic features from the text modal data, and fusing the dynamic features and the semantic features to generate a fused feature vector through a multi-head cross attention mechanism; S4. constructing a state assessment model, including using a time series partitioning method to divide the fused feature vector and its corresponding state label into a training set, a validation set and a test set in chronological order, using a classifier to train the training set to obtain a state assessment model, determining a state judgment threshold on the validation set, evaluating model performance on the test set, and determining the influence of each feature on the state assessment through feature contribution analysis; S5. Deploy the state assessment model for online monitoring, including collecting real-time multimodal data of equipment operation, generating a real-time fused feature vector according to the standardized data processing process and the standardized feature extraction process, evaluating the equipment state through the state assessment model, and updating the model based on distribution drift detection.

2. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The step of performing time alignment on the numerical modal data and the text modal data according to the event index table in S1 specifically includes: The temperature data and the strain data are time-aligned using a linear interpolation method, and the vibration acceleration data are time-aligned using a cubic spline interpolation method; Based on the maintenance time in the maintenance record, the status label is constructed, and the historical data in a fixed time window before the maintenance time is marked as an abnormal state, and the historical data in the remaining time periods are marked as a normal state.

3. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The standardized data processing process established in S2 specifically includes: For the numerical modal data, the missing data segments shorter than the preset time length are interpolated and supplemented based on the time series characteristics, the missing data segments longer than the preset time length are marked as invalid monitoring intervals, and the outliers are identified and corrected by the local weighted regression smoothing method, and the maximum and minimum value normalization method is used to normalize the data to the [0, 1] interval; For the text modal data, missing timestamps are supplemented by inferring the time relationship of context records, duplicate records are identified based on text similarity calculation and the earliest valid record is retained, a predefined pattern is used for format normalization, and a domain dictionary containing equipment component names and professional terms is established, wherein the domain dictionary includes status words describing equipment status, location words describing equipment locations, parameter value words describing equipment parameters and operation words describing operation behaviors, and a word embedding method is used to convert the text modal data into a feature vector.

4. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The multimodal features extracted in S3 specifically include: For the numerical modal data that has been standardized, the main frequency and the frequency band energy are extracted by Fourier transform to form frequency domain features, the energy distribution and the scale coefficient are extracted by wavelet transform to form time-frequency features, and the frequency domain features and the time-frequency features are combined to form the dynamic features; For the text modal data that has been vectorized, the operation status description record text, the operation parameter record text and the maintenance record text are respectively encoded using a long short-term memory network, and the state words, part words, parameter value words and operation words in the text are extracted based on the domain dictionary to form a feature sequence to generate the semantic feature; The multi-head cross-attention mechanism includes multiple feature interaction units, each of which uses the dynamic feature as a query vector and the semantic feature as a key-value vector to calculate the feature relevance to obtain an interaction weight, and concatenates the interaction weights on the feature dimension and transforms them through a fully connected layer to obtain the fused feature vector.

5. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The state assessment model constructed in S4 includes: The state assessment model is constructed by using CatBoost classifier, and cross entropy is used as the loss function for training; a state determination threshold is selected by ROC curve analysis on the validation set, a predicted probability value greater than the state determination threshold is determined as an abnormal state, and a predicted probability value less than the state determination threshold is determined as a normal state, and the model performance is evaluated by F1 score on the test set; The SHAP value analysis is used to calculate the feature contribution, and the feature importance ranking is obtained by averaging the feature contribution to identify the key influencing factors that lead to abnormal status of the equipment.

6. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: Deploying the state assessment model for online monitoring in S5 includes: Collecting the real-time numerical modal data and the text modal data, performing outlier correction and standardization processing on the real-time numerical modal data according to the standardized data processing flow, and performing outlier processing and vectorization processing on the real-time text modal data; Processing the real-time numerical modality data and the real-time text modality data according to the standardized feature extraction method, extracting the dynamic features and the semantic features, and generating the real-time fused feature vector through the multi-head cross attention mechanism; The real-time fused feature vector is input into the state assessment model to perform state assessment and output an abnormal warning.

7. The device status monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The model updating based on distribution drift detection in S5 includes: Determine the monitoring feature dimension according to the feature importance ranking, calculate the KL divergence of the real-time fusion feature vector and the training set on the monitoring feature dimension, and trigger the model update when the KL divergence exceeds the data distribution deviation threshold; A dual-model mechanism is used for updating, in which the original model continues to provide online services, while the model is retrained using an expanded training set containing new data, and the model is switched after the newly trained model is verified.

8. An equipment status monitoring system based on multimodal data fusion, characterized in that: include: A multimodal data acquisition module, used to collect numerical modal data and text modal data during the operation of the equipment, wherein the numerical modal data includes temperature data, strain data and vibration acceleration data, and the text modal data includes operation status description record text, operation parameter record text and maintenance record text; A multimodal data preprocessing module, used to implement standardized processing of the numerical modal data and the textual modal data; A multimodal feature extraction module, used to generate a fused feature vector, including a dynamic feature extraction unit, a semantic feature extraction unit and a feature fusion unit; A model deployment module, which is used to train and deploy state evaluation models based on historical data, including a data alignment unit, a model training unit, and a feature analysis unit; The online monitoring module is used to realize real-time evaluation and early warning of equipment status, including a real-time feature generation unit, a status evaluation unit and a model updating unit, wherein the model updating unit adopts a dual-model mechanism to realize online updating of the model.

9. The device status monitoring system based on multimodal data fusion according to claim 8, characterized in that: The online monitoring module also includes: Distribution drift detection unit, used to monitor real-time data distribution changes and trigger model updates; The status warning unit is used to generate and send warning information based on the status assessment results.

Images