Industrial equipment fault prediction method based on multi-modal data

Abstract

The invention discloses an industrial equipment fault prediction method based on multi-modal data, and belongs to the technical field of specific calculation models, and the method comprises the steps: carrying out the preprocessing according to the collected multi-modal data of the operation of industrial equipment, so as to unify the format of the multi-modal data, and obtaining the structural data; extracting features of the structured data one by one according to data categories, and obtaining a multi-modal fusion feature through a dynamic fusion mechanism; according to the multi-modal fusion features, a fault prediction classification score is obtained through a deep neural network model to perform fault prediction; and when the drift parameter of the multi-modal data is greater than a preset threshold value, performing incremental training on the deep neural network model through the multi-modal data collected in real time to update parameters of the deep neural network model. Through multi-modal data unified processing, dynamic feature fusion, deep neural network modeling and an online learning mechanism, the problems of insufficient multi-modal data fusion, prediction uncertainty quantization deficiency, poor model adaptability and the like are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of specific calculation models, and specifically relates to an industrial equipment fault prediction method based on multimodal data. Background Art

[0002] With the rapid development of industrial intelligence and automation, equipment health management has become a critical component in ensuring production safety, reducing maintenance costs, and improving operational efficiency in sectors such as manufacturing, energy, and transportation. One of the core tasks of equipment health management is failure prediction. Its goal is to identify potential failure risks through real-time monitoring of equipment operating status, providing a scientific basis for maintenance planning and resource optimization.

[0003] In industrial scenarios, equipment operation data is often heterogeneous and multi-source, including time series data collected by sensors (such as vibration, temperature, and pressure), text data (such as maintenance logs and operating manuals), and image / video data (such as equipment appearance status). However, existing technologies mainly rely on a single data source (such as vibration signals or temperature data) for modeling, which makes it difficult to fully capture the complex characteristics of equipment degradation.

[0004] Therefore, deep learning technology, especially large models such as Transformers, has made significant progress in fields such as natural language processing and computer vision. It can be used to process multimodal data and complex time series features, significantly improving fault prediction performance. However, it has significant limitations in processing multimodal data fusion, modeling long time series dependencies, and quantifying uncertainty.

[0005] Industrial equipment operating data often consists of heterogeneous data from multiple sources (such as vibration signals, maintenance logs, and equipment images). Existing technologies struggle to efficiently and uniformly encode and fuse this data, resulting in incomplete feature representation and impacting the accuracy of fault identification. Furthermore, the operating environment and failure modes of industrial equipment can change dynamically over time (e.g., load fluctuations and sensor drift). Existing models lack adaptive update mechanisms, resulting in reduced long-term prediction performance.

[0006] Therefore, developing an industrial equipment fault prediction method that can integrate multimodal data and has online learning capabilities has become a key technical requirement in the field of industrial equipment health management. Summary of the Invention

[0007] The present invention provides an industrial equipment fault prediction method based on multimodal data. Through multimodal feature extraction and fusion, combined with a large model learning framework, fault prediction is achieved. It also supports online learning and continuous update optimization, and can adapt to equipment aging, environmental changes and new failure modes.

[0008] The technical solution adopted in the present invention is:

[0009] A method for predicting industrial equipment failure based on multimodal data, comprising:

[0010] According to the multimodal data of industrial equipment operation obtained by preprocessing, the multimodal data is unified in format to obtain structured data, wherein the multimodal data categories include time series data, text data, image and video data;

[0011] According to the data category, features are extracted from the structured data one by one, and a multimodal fusion feature is obtained through a dynamic fusion mechanism;

[0012] According to the multimodal fusion features, a fault prediction classification score is obtained through a deep neural network model to perform fault prediction;

[0013] When the drift parameter of the multimodal data is greater than a preset threshold, the deep neural network model is incrementally trained using the multimodal data collected in real time to update the deep neural network model parameters.

[0014] The industrial equipment fault prediction method based on multimodal data provided in the present invention also has the following additional technical features:

[0015] The time series data, text data, image and video data are specifically:

[0016] The time series data includes at least any one of vibration, temperature, current, voltage, and pressure data collected in real time;

[0017] The text data includes at least one of a device maintenance log, an operation manual, and a fault record text;

[0018] The image and video data at least include images or videos recording the operation of the device.

[0019] The multimodal data format is unified through preprocessing, specifically:

[0020] According to the multimodal data categories, processing one by one to obtain data vectors or tensors, and aligning the data vectors or tensors by timestamps;

[0021] Among them, according to the time series data, a fixed-length or variable-length tensor is obtained through a sliding window method or a dynamic filling and truncation method;

[0022] Based on the text data, a context-dependent embedding vector is obtained through a pre-trained language model;

[0023] According to the image and video data, a multi-dimensional feature vector is obtained through a convolutional neural network.

[0024] The preprocessing further includes:

[0025] According to the time series data, noise elimination, missing value filling and outlier processing are performed;

[0026] Among them, noise elimination is performed by moving average filtering or wavelet denoising or threshold judgment method;

[0027] Fill missing values ​​through time interpolation or neighboring data feature interpolation;

[0028] Identify outliers through statistical methods or machine learning algorithms and mark or remove them accordingly.

[0029] According to the data category, features are extracted from the structured data one by one, specifically:

[0030] According to the time series data, the time series features are extracted through the deep network, and the frequency domain features are extracted through the time-frequency analysis method;

[0031] According to the image video data, the features of the video frame are extracted through the deep visual encoder;

[0032] Among them, extracting temporal features through deep networks includes extracting short-term local features through one-dimensional convolutional neural networks, extracting long-term dependencies through long short-term memory networks, and extracting periodic dependencies through time convolutional networks.

[0033] Through the dynamic fusion mechanism, a multimodal fusion feature is obtained, specifically:

[0034] Through the attention mechanism, according to the mutual information between the multiple features extracted, the feature fusion weights are set one by one, and the multiple features extracted are fused to obtain a multimodal fusion feature; or,

[0035] A feature vector is obtained by combining the extracted multiple features through a splicing operation, and a multimodal fusion feature is obtained through a learnable transformation layer model, wherein the learnable transformation layer model is provided with multiple fusion strategies.

[0036] The deep neural network model is specifically:

[0037] The deep neural network model uses a self-attention mechanism to obtain a query vector, a key vector, and a value vector based on the multimodal fusion features at different times. A weighted weight is set for the value vector based on the similarity between the query vector and the key vector, and the multimodal fusion features at multiple times are obtained by weighted summation.

[0038] Through multi-head attention, the fault prediction classification score is obtained based on multi-time multimodal fusion features to perform fault prediction;

[0039] The result of the fault prediction includes at least one of normal, potential fault, and high-risk fault.

[0040] The drift parameter of the multimodal data is greater than a preset threshold, specifically:

[0041] Obtaining a drift parameter for each multimodal data category based on the multimodal data at adjacent moments;

[0042] Obtaining a drift parameter of the multimodal data by weighted averaging or selecting a maximum value based on the plurality of drift parameters;

[0043] If the drift parameter of the multimodal data is greater than a preset threshold, it is determined that data drift occurs;

[0044] The method for obtaining the drift parameter includes at least one of KS test, KL divergence, mean comparison and variance comparison.

[0045] The deep neural network model is incrementally trained using multimodal data collected in real time, specifically:

[0046] Models deployed on edge servers or industrial gateways are trained based on multimodal data collected in real time.

[0047] The model deployed in the cloud is trained based on real-time collected multimodal data and historical multimodal data.

[0048] The models deployed on edge servers or industrial gateways and those deployed on the cloud are as follows:

[0049] The number of attention heads and / or network layers of the model deployed on the edge server or industrial gateway is smaller than that of the model deployed on the cloud;

[0050] The training results of the model deployed in the cloud guide the training direction of the model deployed on the edge server or industrial gateway.

[0051] Due to the adoption of the above technical solution, the beneficial effects achieved by the present invention are as follows:

[0052] 1. In this invention, structured data is generated based on the collected multimodal data of industrial equipment operation through preprocessing to unify the multimodal data format. By unifying the format of multimodal data (time series, text, image / video), the difficulty of integrating multi-source heterogeneous data in the prior art is solved. The generation of structured data provides a consistent input format for subsequent feature extraction and modeling, improving the model's compatibility with multimodal features and processing efficiency.

[0053] Based on the data category, features are extracted from the structured data one by one, and a multimodal fusion feature is obtained through a dynamic fusion mechanism. Features are extracted separately for different data categories (time series, text, image / video) to avoid the limitations of single-modal modeling. The dynamic fusion mechanism dynamically adjusts feature weights based on the mutual information of each modality, improving the model's sensitivity to key fault characteristics and reducing redundant information interference. The fused multimodal feature representation can comprehensively capture the multi-dimensional information of the equipment degradation process, significantly improving the accuracy of fault diagnosis.

[0054] Based on the multimodal fusion features, a deep neural network model is used to obtain a fault prediction classification score for fault prediction. The deep neural network model based on the Transformer architecture can efficiently capture long-term temporal dependencies and overcome the gradient vanishing problem of traditional models in long-sequence modeling. The output fault classification score (such as "normal," "potential fault," and "high-risk fault") provides operations and maintenance personnel with a quantitative risk assessment and supports scientific decision-making.

[0055] When the drift parameter of the multimodal data is greater than a preset threshold, the deep neural network model is incrementally trained using the multimodal data collected in real time to update the deep neural network model parameters. Data drift detection monitors changes in the equipment operating environment (such as sensor drift and equipment aging) in real time, dynamically triggering model updates to avoid performance degradation of traditional fixed models due to environmental changes. In the presence of data drift, continuous iteration of the model driven by real-time data improves the model's adaptability to new failure modes and extends the long-term effectiveness of the equipment health management system.

[0056] In summary, the present invention solves the core problems of the existing technology, such as insufficient multimodal data fusion, weak long-term time series modeling capabilities, lack of prediction uncertainty quantification, and poor model adaptability, through unified multimodal data processing, dynamic feature fusion, deep neural network modeling, and online learning mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0058] Figure 1 The figure is a flow chart of the method for predicting industrial equipment failure based on multimodal data according to one embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to more clearly illustrate the overall concept of the present invention, a detailed description is given below in an exemplary manner in conjunction with the accompanying drawings.

[0060] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0061] like Figure 1 As shown, a method for predicting industrial equipment failure based on multimodal data includes:

[0062] S100: According to the collected multimodal data of the operation of the industrial equipment, preprocessing is performed to unify the format of the multimodal data to obtain structured data, wherein the multimodal data categories include time series data, text data, image and video data.

[0063] The core purpose of this step is to unify the multimodal data formats of industrial equipment operations and generate structured data to address the difficulty of integrating multi-source heterogeneous data. Preprocessing converts time series, text, image / video data, and other data into consistent representations (such as tensors or embedded vectors). This provides compatible input for subsequent feature extraction and modeling, improving the model's efficiency and accuracy in processing multimodal features.

[0064] Multimodal data types include time series data (vibration, temperature, etc.), text data (maintenance logs, etc.), and image / video data (device appearance status). It is understood that this step divides the data into categories based on the data source and characteristics and determines the preprocessing strategy accordingly.

[0065] For time series data preprocessing, convert the time series data into fixed-length or variable-length tensors using a sliding window approach or dynamic padding and truncation. In this step, select the sliding window or padding and truncation strategy based on the equipment's operational requirements (such as the time window range for fault detection) to ensure that the data length matches the model input.

[0066] For text data preprocessing, a pretrained language model is used to convert unstructured text into contextually relevant embedding vectors. This step determines whether to perform domain fine-tuning based on the semantic complexity of the text to improve the model's understanding of industrial terminology.

[0067] For image / video data preprocessing, we extract multidimensional feature vectors of image / video frames using convolutional neural networks or visual transformers. We adjust the feature extraction strategy based on image resolution and frame rate to ensure consistency of spatiotemporal information.

[0068] It is important to note that the timestamps of data from different modalities are aligned and the preprocessing results are stored in a unified format (such as a tensor). By verifying the consistency of timestamps, the synchronization of data in the time series dimension is ensured, avoiding feature misalignment caused by asynchronous acquisition.

[0069] This step converts multimodal data into a unified format through standardization, eliminating model input inconsistencies caused by differences in data length and format, improving data compatibility, and solving the problem of integrating multi-source heterogeneous data in industrial scenarios.

[0070] S200: extracting features from the structured data one by one according to the data category, and obtaining a multimodal fusion feature through a dynamic fusion mechanism.

[0071] The core purpose of this step is to generate a unified multimodal fusion feature through multimodal feature extraction and dynamic fusion, thereby improving the model's comprehensive perception and discrimination of equipment fault characteristics. Key features are extracted from different modal data, such as time series, text, and images / video. By dynamically adjusting the weights of each modality, these features are integrated into a unified representation, providing high-quality input for subsequent fault prediction.

[0072] In this step, features are extracted by data category. For time series data, 1D-CNN (one-dimensional convolutional neural network), LSTM (long short-term memory network), or TCN (temporal convolutional network) are used to extract local, long-term, or cyclical dependency features. For text data, a pre-trained language model is used to generate context-sensitive embedding vectors to preserve semantic relevance. For image / video data, convolutional neural networks or visual transformers are used to extract multidimensional feature vectors to capture visual anomalies. The appropriate feature extraction method is selected based on the data type to ensure that the key features of each data type are fully explored.

[0073] Dynamically fuse multimodal features, calculate the mutual information between each modal feature, and dynamically adjust the fusion weights (e.g., cross-modal attention mechanism). Alternatively, after splicing the features of different modalities, automatically optimize the fusion strategy through a fully connected layer or a learnable transformation layer (e.g., multi-head attention). This step selects the fusion method based on the task requirements. If the correlation between modalities needs to be emphasized, the attention mechanism is used; if the calculation needs to be simplified, the splicing plus learnable transformation is used to achieve fusion.

[0074] Generate multimodal fusion features and output the dynamic fusion results as a unified multimodal feature representation (such as a tensor or high-dimensional vector). It is important to verify whether the fused features can effectively cover the multi-dimensional information of equipment degradation (such as vibration anomalies, maintenance patterns in text records, and appearance damage in images) to ensure the integrity of subsequent model inputs.

[0075] This step extracts features from different modal data types (time series, text, and images / video) to avoid the limitations of single-modality modeling and enhance feature perception. Multimodal feature fusion integrates temporal, semantic, and visual information (such as temperature trends, maintenance record keywords, and equipment surface cracks) to comprehensively capture equipment degradation information and significantly improve the accuracy of fault identification.

[0076] S300: Based on the multimodal fusion features, a deep neural network model is used to obtain a fault prediction classification score to perform fault prediction.

[0077] The core purpose of this step is to model multimodal fusion features using a deep neural network model, output a fault prediction classification score, and achieve accurate prediction of equipment failures. By introducing large model architectures such as the Transformer, combined with self-attention mechanisms and multi-task learning, this method addresses the shortcomings of traditional models in modeling long-term dependencies, perceiving multimodal features, and achieving reliable predictions, thereby improving the accuracy and robustness of fault identification.

[0078] The multimodal fusion features (e.g., unified tensors or embedding vectors) output from the previous step are used as input to the deep neural network model. The compatibility of the input data dimensions with the model input layer is verified to ensure the integrity and consistency of the feature representation.

[0079] The deep neural network uses a Transformer-based encoder architecture, combined with a multi-head attention mechanism and a feedforward network, to perform deep learning of multimodal fusion features. The model's ability to fit fault patterns is evaluated by the convergence of a validation set loss function (such as cross-entropy loss). A fault classification score is output on top of the Transformer to categorize the device status (e.g., "normal," "potential fault," or "high-risk fault").

[0080] The Transformer architecture is a deep learning model based on the self-attention mechanism, which implements sequence modeling through multi-head attention and feedforward networks. It replaces traditional RNN / LSTM, solves the problem of modeling long sequence dependencies, and supports parallel processing of multimodal features.

[0081] The multi-head attention mechanism maps input features to multiple attention heads, calculates query-key-value similarity separately, and then performs a weighted fusion of the results. This enhances the model's ability to perceive complex relationships between different modalities (such as time series, text, and images) and dynamically adjusts the contribution of each modal feature.

[0082] This step uses deep neural network models (such as the Transformer) to model multimodal fusion features, combining self-attention mechanisms with a multi-task learning framework to output a fault classification score. Compared to traditional methods, this step significantly improves the accuracy, robustness, and uncertainty quantification of fault prediction, providing a high-reliability solution for industrial equipment health management.

[0083] S400: When the drift parameter of the multimodal data is greater than a preset threshold, incremental training is performed on the deep neural network model using the multimodal data collected in real time to update the deep neural network model parameters.

[0084] The core purpose of this step is to dynamically update the parameters of the deep neural network model through an incremental training mechanism to address the problem of model performance degradation caused by changes in the distribution of multimodal data over time. When the drift parameters of the multimodal data (such as distribution differences and changes in statistical characteristics) exceed the preset threshold, the system triggers the incremental training process and uses the new data collected in real time to adjust the model parameters to ensure that the model continues to adapt to changes in the equipment operating environment and the emergence of new failure modes.

[0085] To detect data drift, the distribution of real-time multimodal data (time series, text, images / video) is compared with historical training data, and drift parameters (such as the KS test statistic, KL divergence, and mean-variance difference) are calculated. The drift parameters of each modality are combined using a weighted average or maximum method to generate an overall drift index. If the overall drift parameter is greater than a preset threshold (such as a KL divergence threshold of 0.1), data drift is determined to have occurred.

[0086] Triggering incremental training starts the incremental training process in the model. New samples are extracted from the real-time data stream and combined with historical data to construct an incremental training set. When data drift is triggered, the system automatically activates the incremental training module without manual intervention.

[0087] By monitoring data drift in real time and triggering incremental training, the model can adapt to the effects of device aging, environmental changes, and new failure modes. Compared to traditional fixed models, this step significantly extends the model's effective lifecycle and reduces the degradation of predictive performance caused by changes in data distribution.

[0088] By dynamically updating model parameters, prediction biases caused by data drift (such as misjudgments caused by sensor drift) can be corrected. Traditional models require retraining on the entire data set after data drift, while this step uses incremental updates to quickly correct the model and shorten response time.

[0089] This step addresses the issue of model performance degradation in industrial scenarios caused by dynamic changes in the equipment operating environment through data drift detection and incremental training. By identifying data drift through methods such as the KS test and KL divergence, combined with model parameter updates, the model's long-term adaptability and predictive stability are significantly improved.

[0090] As a preferred embodiment of the present invention, the time series data, text data, image and video data are specifically:

[0091] The time series data includes at least any one of vibration, temperature, current, voltage, and pressure data collected in real time;

[0092] The text data includes at least one of a device maintenance log, an operation manual, and a fault record text;

[0093] The image and video data at least include images or videos recording the operation of the device.

[0094] The core purpose of this implementation is to clarify the specific content and collection methods of time series data, text data, and image / video data, providing a clear data source for subsequent multimodal feature extraction and fusion. By limiting the data type and collection scope, this ensures the efficient integration of multi-source heterogeneous data during model training and prediction, addressing the inadequate modeling of single data sources in existing technologies and improving the comprehensiveness and accuracy of fault prediction.

[0095] In time series data, vibration data is collected in real time via accelerometers during equipment operation, indicating abnormal wear or loosening of mechanical components. Temperature data uses thermocouples or infrared sensors to monitor surface or internal temperature changes in equipment, used to identify overheating or cooling system failures. Current / voltage data is collected via power monitoring modules to capture abnormal fluctuations in electrical systems. Pressure data uses pressure sensors to monitor the operating status of hydraulic or pneumatic systems, used to diagnose sealing or fluid dynamics issues.

[0096] Sensors continuously collect data at a fixed sampling frequency (e.g., 1kHz) to ensure the integrity of timing information. It's important to note that key sensors should be selected based on the equipment type (e.g., vibration and temperature data should be prioritized for rotating equipment). Furthermore, the data collection frequency should meet the requirements for fault feature extraction (e.g., vibration signals must cover the fault-related frequency band).

[0097] In text data, equipment maintenance logs record the time of equipment failure, repair measures, and a text description of the cause of the failure. Operation manuals provide structured text of equipment standard operating procedures and troubleshooting guides. Fault log text includes descriptive information about historical failure cases (e.g., "bearing wear causes abnormal vibration").

[0098] Filter relevant text data based on equipment type (e.g., maintenance logs related to bearing failures are preferentially extracted for rotating equipment).

[0099] In image / video data, equipment appearance images are captured by industrial cameras to identify surface conditions (such as cracks, corrosion, and foreign matter). Motion trajectory videos are recorded by high-speed cameras to track the dynamics of moving parts (such as gears and belts) and capture abnormal vibrations or deviations.

[0100] Use industrial-grade cameras (e.g., megapixel resolution) to ensure detailed information can be discerned. Select the image / video acquisition frequency based on the equipment's operating conditions (e.g., 30 frames per second or higher for high-speed moving parts). Verify the spatiotemporal consistency of the image / video data (e.g., timestamp error with sensor data is less than 100ms).

[0101] Compared to traditional methods that rely on a single data source (e.g., vibration signals alone), which can easily overlook the multi-dimensional manifestations of equipment failures (e.g., historical maintenance experience in text or appearance damage in images), this implementation addresses the challenge of integrating multi-source heterogeneous data and improves data compatibility and interpretability by clarifying the specific types and collection methods of time series data (e.g., vibration signals), text data (e.g., maintenance logs), and image / video data (e.g., images of equipment appearance).

[0102] The combination of fault descriptions in text data and visual anomalies (such as crack propagation) in image / video data enables the model to identify latent failure modes (such as “material fatigue caused by low vibration but high temperature”), supporting dynamic failure mode recognition.

[0103] As a preferred embodiment of the present invention, the pretreatment includes:

[0104] According to the time series data, noise elimination, missing value filling and outlier processing are performed;

[0105] Among them, noise elimination is performed by moving average filtering or wavelet denoising or threshold judgment method;

[0106] Fill missing values ​​through time interpolation or neighboring data feature interpolation;

[0107] Identify outliers through statistical methods or machine learning algorithms and mark or remove them accordingly.

[0108] Time series data collected at industrial sites inevitably contains noise, missing values, and abnormal peaks, such as abnormal spikes caused by sensor failures and network transmission delays. The core purpose of this implementation is to improve the quality of time series data through preprocessing. This method addresses the common noise, missing value, and outlier issues found in industrial equipment operation data, providing high-quality, consistent data input for subsequent feature extraction and model training.

[0109] For noise removal, a moving average filter averages the sampling points within a sliding window of time series data, smoothing high-frequency noise (such as spike noise in vibration signals) while preserving low-frequency fault characteristics (such as the periodic pulse signals of bearing faults). Wavelet denoising uses the wavelet transform to decompose the signal into different frequency components. Thresholding processing (such as soft thresholding or hard thresholding) removes the noise components and retains the valid signal. Threshold determination is based on statistical distribution (such as the 3σ principle) or dynamic thresholds (such as the mean + standard deviation within a sliding window) to identify and truncate noise points that exceed a reasonable range.

[0110] Select a method based on the noise type (e.g., moving average filtering is suitable for periodic noise, while wavelet denoising is suitable for non-stationary noise). Furthermore, evaluate the filtering effect using signal-to-noise ratio or mean square error to ensure that key fault characteristics (e.g., shock pulses in bearing faults) are preserved.

[0111] To fill missing values, time interpolation uses linear, spline, or cubic spline interpolation to estimate missing values ​​based on data from adjacent time steps. This method can quickly restore data continuity but may overlook nonlinear trends (such as degradation signals caused by slowly rising temperatures). Neighboring data feature interpolation combines statistical features (such as mean and median) of adjacent time windows or historical data under similar operating conditions to fill missing values. This method is suitable for complex missing scenarios (such as long-term sensor failures) and restores data rationality through contextual information.

[0112] For short-term missing values ​​(e.g., 1-3 sampling points), time interpolation is preferred. For long-term missing values ​​or complex operating conditions, neighboring data feature interpolation is used. The filling effect is verified by residual analysis (e.g., root mean square error) between the filled data and the original data to ensure that the filled value does not affect the expression of the fault characteristics.

[0113] Statistical methods for outlier processing include Z-score detection (the standard deviation of a data point from the mean), IQR (interquartile range) detection, or dynamic thresholding of sliding windows to identify outliers. This helps adapt to changes in equipment operating conditions (such as signal baseline drift caused by load fluctuations) and improves anomaly detection accuracy. Machine learning algorithms, such as isolation forests, local anomaly factors, or autoencoders, detect outliers that deviate from the normal distribution. These algorithms can efficiently detect rare anomalies (such as transient sensor failures) and support high-dimensional data (such as multi-sensor joint analysis). Identified outliers are marked or removed, and the abnormal events are recorded for subsequent analysis.

[0114] For occasional anomalies (such as short-term sensor failures), a labeling strategy is adopted and corrected in conjunction with contextual information. For systematic anomalies (such as sensor drift), they are directly eliminated and the calibration mechanism of the data acquisition module is triggered. Anomaly density analysis (such as the number of anomalies per unit time) determines whether the anomaly detection threshold needs to be adjusted to adapt to changes in the device operating environment.

[0115] This implementation addresses the issue of low-quality time series data in industrial scenarios by eliminating noise, filling missing values, and addressing outliers. The combined application of moving average filtering, wavelet denoising, time interpolation, and machine learning algorithms significantly improves data quality, laying a solid foundation for subsequent feature extraction and model training. This solution is not only compatible with a variety of sensor data (such as vibration, temperature, and pressure), but also supports adaptive adjustments under dynamic conditions, providing reliable data support for equipment failure prediction in complex industrial scenarios.

[0116] Furthermore, preprocessing is performed to unify the multimodal data format, specifically:

[0117] According to the multimodal data categories, processing one by one to obtain data vectors or tensors, and aligning the data vectors or tensors by timestamps;

[0118] Among them, according to the time series data, a fixed-length or variable-length tensor is obtained through a sliding window method or a dynamic filling and truncation method;

[0119] Based on the text data, a context-dependent embedding vector is obtained through a pre-trained language model;

[0120] According to the image and video data, a multi-dimensional feature vector is obtained through a convolutional neural network.

[0121] The core purpose of this embodiment is to unify multimodal data (time series, text, image / video) into a structured format (such as tensor or embedding vector) through preprocessing methods to provide compatible input for subsequent feature extraction and model training.

[0122] For the processing and tensorization of time series data, a sliding window method is used to split continuous time series into fixed-length subsequences (e.g., a window size of 100 sampling points), generating fixed-length tensors (shaped as [number of time steps, number of features]). This method preserves local time series features (e.g., short-term abnormal fluctuations) and adapts to model input requirements. A dynamic padding and truncation method is used to fill (e.g., fill with zeros) or truncate (e.g., retain the last N time steps) time series of inconsistent lengths to generate variable-length tensors. This method is suitable for processing irregular-length data and ensures dimensional consistency of the input tensor.

[0123] Select a method based on the device's operational requirements (e.g., sliding window methods are suitable for capturing local patterns, while dynamic padding and truncation methods are suitable for handling long-term degradation trends). Additionally, verify that the tensor dimensions match the model input requirements (e.g., the Transformer model requires fixed-length input).

[0124] For the vectorization of text data, pre-trained language models, using models like BERT and GPT, convert text (such as maintenance logs) into contextually relevant embedding vectors. This enhances the expressive power of text features and improves the model's ability to discern textual information such as maintenance logs. Furthermore, pre-trained models are fine-tuned for industrial scenarios to adapt them to specialized terminology related to equipment failures (e.g., "bearing wear" and "seal failure").

[0125] Choose whether to perform domain fine-tuning based on the semantic complexity of the text (e.g., general terms do not require fine-tuning, while specialized terms do). Evaluate the effectiveness of the pre-trained model by verifying the similarity of the embedding vectors (e.g., cosine similarity).

[0126] For feature extraction of image / video data, a convolutional neural network model is used to extract multidimensional feature vectors (such as color histograms and texture features) from image / video frames. This allows for the identification of visual anomalies such as cracks and corrosion from device appearance images (e.g., edge features extracted through convolution kernels). Frame-level alignment of video data is performed based on timestamps to ensure that the time steps of the image sequence and time series data are consistent.

[0127] Adjust the feature extraction strategy based on the image resolution and frame rate (e.g., use a lightweight CNN for low-resolution images). In addition, verify the consistency of the timestamps of image / video features and sensor data (e.g., the error is less than 100ms).

[0128] It should be noted that the timestamp alignment of multimodal data ensures that the different modal data are aligned in the time dimension through interpolation (such as linear interpolation) or discarding mismatched time steps, avoiding feature misalignment caused by asynchronous acquisition (such as mismatch between vibration signals and video frame times). The processed data vectors or tensors are stored in a unified format to facilitate subsequent modeling.

[0129] Verify the alignment results by comparing the dimensions of the tensors of different modalities (e.g., whether the sampling frequency of the time series matches that of the image frames). If the timestamp difference is too large, the mismatched data is discarded to avoid model input errors.

[0130] This embodiment solves the problem of difficulty integrating multi-source heterogeneous data in industrial scenarios through the combined application of sliding window methods, pre-trained language models, and convolutional neural networks. Through timestamp alignment and unified format conversion, it provides high-quality input for subsequent multimodal feature fusion and deep learning modeling, significantly improving the comprehensiveness and robustness of fault prediction. It is compatible with a variety of equipment types (such as rotating equipment and hydraulic systems), supports dynamic fault pattern recognition, and is applicable to industrial equipment health management scenarios.

[0131] As a preferred embodiment of the present invention, features are extracted from the structured data one by one according to the data category, specifically:

[0132] According to the time series data, the time series features are extracted through the deep network, and the frequency domain features are extracted through the time-frequency analysis method;

[0133] According to the image video data, the features of the video frame are extracted through the deep visual encoder;

[0134] Among them, extracting temporal features through deep networks includes extracting short-term local features through one-dimensional convolutional neural networks, extracting long-term dependencies through long short-term memory networks, and extracting periodic dependencies through time convolutional networks.

[0135] The core purpose of this implementation is to extract key features from multimodal structured data (time series data and image / video data) to address the limitations of single-modality modeling in traditional methods. By extracting temporal features (short-term local features, long-term dependencies, and periodic dependencies) and frequency domain features from time series using a deep network, and combining this with a deep visual encoder to extract multidimensional feature vectors from images / videos, the model's ability to perceive device degradation processes is significantly improved.

[0136] For feature extraction of time series data, one-dimensional convolutional neural networks (1D-CNNs) use a sliding window of convolution kernels to extract short-term local features of vibration signals (such as shock pulses and local anomalies). Long short-term memory networks (LSTMs) model long-term dependencies in time series (such as the gradual changes in bearing wear). Temporal convolutional networks (TCNs) use dilated convolutions to extract periodic dependencies (such as the periodic vibration of gear meshing).

[0137] In frequency domain feature extraction, time-frequency analysis methods (such as fast Fourier transform FFT or wavelet transform) convert time series signals into frequency domain representation and extract specific frequency band energy (such as the fault frequency characteristics of bearing defects).

[0138] Select a method based on the fault feature type (e.g., 1D-CNN for local anomalies, LSTM for long-term degradation trends, TCN for periodic faults). Verify whether frequency domain features can complement time series features (e.g., identifying fault frequency components through spectral energy distribution).

[0139] For feature extraction from image / video data, deep visual encoders, such as convolutional neural networks (CNNs), use pre-trained models (e.g., ResNet50) to extract local image features (e.g., crack morphology, surface corrosion). Visual Transformers (ViTs) use a self-attention mechanism to extract global image features (e.g., the overall appearance of the device). After splitting the video into consecutive frames, features are extracted frame by frame and concatenated into a time-series feature vector.

[0140] Align the feature vectors (e.g., [height, width, number of channels]) output by CNN or ViT with the timestamp to form a unified representation.

[0141] Select a model based on image resolution and operating condition complexity (e.g., lightweight CNN is preferred for low-resolution images). Verify that image features can capture visual anomalies during equipment degradation (e.g., progressive changes in crack growth).

[0142] This implementation uses a combined 1D-CNN, LSTM, and TCN model to extract short-term local features, long-term dependencies, and periodic features from time series data, improving the comprehensiveness of feature extraction. Time-frequency analysis methods (such as FFT) extract frequency domain features (such as the fault frequency of bearing defects), supplementing the deficiencies of the time series model and enhancing the model's ability to perceive complex fault modes. Image / video features (such as crack textures) and time series features (such as vibration signals) are represented through a unified tensor, providing compatible input for subsequent multimodal fusion and supporting multimodal feature fusion.

[0143] In summary, this implementation method jointly models the short-term, long-term and periodic features of time series through 1D-CNN, LSTM and TCN, extracts frequency domain features in combination with time-frequency analysis methods, and uses CNN / ViT to extract multi-dimensional feature vectors of images / videos, which significantly improves the comprehensiveness and complementarity of multimodal features.

[0144] Furthermore, through the dynamic fusion mechanism, a multimodal fusion feature is obtained, specifically:

[0145] Through the attention mechanism, according to the mutual information between the multiple features extracted, the feature fusion weights are set one by one, and the multiple features extracted are fused to obtain a multimodal fusion feature; or,

[0146] A feature vector is obtained by combining the extracted multiple features through a splicing operation, and a multimodal fusion feature is obtained through a learnable transformation layer model, wherein the learnable transformation layer model is provided with multiple fusion strategies.

[0147] The core purpose of this step is to generate unified multimodal fusion features through a dynamic fusion mechanism (attention mechanism or learnable transformation layer model) to solve the alignment problem of multi-source heterogeneous data in the semantic space and enhance the model's comprehensive perception of the device degradation process.

[0148] Example 1: Dynamic Fusion of Attention Mechanisms. Mutual information is calculated for multiple extracted features (such as time-frequency features of time series, convolutional features of images, and text embedding vectors). Mutual information can be achieved using statistical methods (such as KL divergence) or deep learning models (such as mutual information maximization networks).

[0149] Mutual information measures the information shared between two random variables; larger values ​​indicate stronger correlation between the variables. It is used to quantify the correlation between different modal features (such as the co-degradation of vibration and temperature signals) and guide dynamic weight allocation.

[0150] Based on the mutual information value, dynamic weights are assigned to each modal feature (e.g., modalities with higher mutual information values ​​are given higher weights). The weights can be normalized using the Softmax function to ensure that the sum of the weights is 1. The weighted feature vectors are then element-wise added or concatenated and weighted summed to generate a multimodal fusion feature.

[0151] If you need to emphasize the correlation between modalities (such as the co-degradation of vibration and temperature signals), the attention mechanism is the preferred choice. By calculating the similarity between the query, key, and value, the attention mechanism dynamically adjusts feature weights, enabling it to adaptively capture the global dependencies between multimodal features (such as the co-variation between vibration and temperature data).

[0152] Example 2: Learnable Transformation Layer Model Fusion: Multiple extracted features (such as time series tensors, text embedding vectors, and image feature vectors) are concatenated into a unified format (such as [B, T, C1+C2+C3]) according to the channel or time dimension.

[0153] The learnable transformation layer modeling includes a self-attention mechanism, a variant of the attention mechanism, which calculates the correlation between different positions within a feature. By calculating the global dependencies of the concatenated features, it dynamically adjusts the weights of each modal feature. The fully connected layer compresses the feature dimensions through linear transformations and nonlinear activation functions (such as ReLU) to extract cross-modal interaction information. Residual connections add the original concatenated features to the transformed features, preserving low-order feature information.

[0154] Choose a fusion strategy based on task requirements (e.g., self-attention only, full connection + residual connection). If you need to simplify computational complexity, prioritize fully connected layers; if you need to model long-range dependencies, choose the self-attention mechanism.

[0155] This embodiment solves the difficulty of fusing heterogeneous multi-source data in industrial scenarios by combining an attention mechanism with a learnable transformation layer model. By dynamically adjusting weights through mutual information calculation and combining self-attention with a fully connected layer strategy, the comprehensiveness and robustness of multimodal fusion features are significantly improved.

[0156] As a preferred embodiment of the present invention, the deep neural network model is specifically:

[0157] The deep neural network model uses a self-attention mechanism to obtain a query vector, a key vector, and a value vector based on the multimodal fusion features at different times. A weighted weight is set for the value vector based on the similarity between the query vector and the key vector, and the multimodal fusion features at multiple times are obtained by weighted summation.

[0158] Through multi-head attention, the fault prediction classification score is obtained based on multi-time multimodal fusion features to perform fault prediction;

[0159] The result of the fault prediction includes at least one of normal, potential fault, and high-risk fault.

[0160] The core purpose of this implementation is to utilize deep neural network models, specifically self-attention and multi-head attention mechanisms, to process multi-modal fusion features at multiple moments in time to achieve accurate fault prediction classification scores. Through a dynamic weighted summation approach, the model's focus on features at different time points and modalities is enhanced, thereby improving the accuracy of fault prediction.

[0161] The self-attention mechanism transforms the multimodal fusion features at each time point into a query vector (Query), a key vector (Key), and a value vector (Value) through feature conversion. This is usually achieved through linear transformation, that is, mapping the input features into the Q, K, and V spaces respectively through a fully connected layer.

[0162] The similarity between the query vector and all key vectors is calculated (e.g., using dot products or scaled dot products) to generate an attention score. The attention scores are normalized using the Softmax function to obtain the weighted weights of each value vector. Based on the calculated weights, the value vectors are weighted summed to obtain the weighted multi-time multimodal fusion features.

[0163] Select an appropriate similarity metric (such as dot product or scaled dot product) based on the task requirements to ensure that the importance of different time points can be effectively distinguished. In addition, verify whether the weighted features better capture key information about equipment degradation (such as abnormal fluctuations in vibration signals).

[0164] In this embodiment, the self-attention process described above is repeated multiple times (i.e., multiple heads) through a multi-head attention mechanism, each time using a different set of parameters to learn different feature representations. This helps the model understand data from different perspectives and improves its expressiveness. The different feature representations obtained by the multiple heads are concatenated and resized back to their original size through an additional linear transformation. Based on the final feature representation, the fault prediction classification score is calculated using a fully connected layer and an activation function (such as Softmax).

[0165] Based on the model performance evaluation results, select an appropriate number of heads (such as 8 or 16 heads) to maximize the model's expressiveness without excessively increasing computational complexity. Use metrics such as accuracy and recall on the validation set to evaluate the model's fault prediction performance.

[0166] This implementation uses a self-attention mechanism to dynamically adjust the weights of features at different time points, enabling the model to focus more on time periods containing important fault information. Furthermore, the multi-head attention mechanism allows the model to understand data from multiple perspectives, capturing more diverse fault patterns and thus improving the accuracy of fault prediction. Both self-attention and multi-head attention mechanisms have excellent generalization capabilities and can adapt to the fault prediction needs of different types of equipment and operating conditions.

[0167] As a preferred embodiment of the present invention, the drift parameter of the multimodal data is greater than a preset threshold, specifically:

[0168] Obtaining a drift parameter for each multimodal data category based on the multimodal data at adjacent moments;

[0169] Obtaining a drift parameter of the multimodal data by weighted averaging or selecting a maximum value based on the plurality of drift parameters;

[0170] If the drift parameter of the multimodal data is greater than a preset threshold, it is determined that data drift occurs;

[0171] The method for obtaining the drift parameter includes at least one of KS test, KL divergence, mean comparison and variance comparison.

[0172] The core purpose of this implementation is to detect data drift in multimodal data, i.e., changes in data distribution over time. This is done by calculating drift parameters for multimodal data categories at adjacent moments and using a weighted average or maximum value selection method to comprehensively assess overall data drift. If the drift parameter exceeds a preset threshold, data drift is considered to have occurred. This helps to promptly detect data changes, allowing adjustments to the model or other measures to ensure system accuracy and reliability.

[0173] In this embodiment, the drift parameter of each multimodal data category is calculated, and one or more methods are selected from KS test, KL divergence, mean comparison and variance comparison to calculate the drift parameter.

[0174] The KS test compares the cumulative distribution functions (CDFs) of two sample sets and is applicable to continuous distributions. The KL divergence measures the difference between two probability distributions and is asymmetric, making it applicable to discrete distributions. Mean comparison directly compares the means of two sample sets, which is simple and intuitive but not comprehensive. Variance comparison compares the variances of two sample sets and can capture changes in the degree of data dispersion.

[0175] For each multimodal data category, use the selected method to calculate the difference in data distribution between adjacent moments and obtain the corresponding drift parameter. Select an appropriate drift detection method based on the data type (e.g., continuous or discrete) and the application scenario. Ensure that the selected method can effectively distinguish between normal fluctuations and actual data drift.

[0176] Comprehensively evaluate the drift parameters of multimodal data. For weighted averaging, assign weights to each drift parameter based on the importance of each modal data (e.g., vibration signals may be more important than temperature data), and then calculate the weighted average. Alternatively, select the maximum value of all drift parameters as the final drift parameter, emphasizing the most significant data changes.

[0177] If you are concerned about overall data stability, you might prefer the weighted average method. If you are more concerned about data drift in extreme cases, you might choose the maximum method. Verify that the comprehensive evaluation results accurately reflect the overall data drift.

[0178] To determine data drift, set a reasonable drift parameter threshold based on historical data and business requirements. Compare the calculated drift parameter with the preset threshold. If the drift parameter is greater than the threshold, data drift is determined to have occurred.

[0179] Preset thresholds should take into account the inherent fluctuation range of the data and the business tolerance, ensuring that they can sensitively detect true data drift while not triggering frequent alerts due to normal fluctuations. Thresholds should be updated regularly to adapt to long-term changes in data distribution trends.

[0180] This implementation accurately calculates drift parameters and sets reasonable thresholds to promptly detect changes in data distribution. When data drift is detected, model parameters can be adjusted or the model can be retrained to ensure its performance on the new data.

[0181] As a preferred embodiment of this implementation, the deep neural network model is incrementally trained using multimodal data collected in real time, specifically:

[0182] Models deployed on edge servers or industrial gateways are trained based on multimodal data collected in real time.

[0183] The model deployed in the cloud is trained based on real-time collected multimodal data and historical multimodal data.

[0184] The core purpose of this embodiment is to incrementally train a deep neural network model using real-time multimodal data to adapt to the time-varying data distribution in industrial scenarios. This process involves collaboratively deploying the model on edge devices (such as edge servers or industrial gateways) and in the cloud, updating it based on real-time data and historical and real-time data, respectively, to improve the model's timeliness, robustness, and generalization capabilities.

[0185] Incremental training of edge models involves deploying lightweight versions of deep neural network models to edge servers or industrial gateways. Sensors, cameras, and other devices collect multimodal data (such as vibration signals, temperature, images, videos, and text logs) in real time, and perform pre-processing operations such as standardization and normalization.

[0186] Use the current batch of real-time data to fine-tune or update the parameters of the local model (for example, using online learning algorithms such as SGD and Adam optimizer). Regularly upload the updated parameters of the edge model to the cloud for global model aggregation.

[0187] For incremental training of cloud-based models, the cloud receives real-time data from multiple edge nodes and combines it with long-term historical data (such as fault samples from the past few months). This multi-source, heterogeneous data undergoes unified format conversion, outlier removal, and missing value filling. Based on this blend of real-time and historical data, the cloud-based master model undergoes periodic retraining or continuous fine-tuning. The updated model parameters or new model versions are distributed to edge devices, replacing the old model or performing incremental updates.

[0188] In this embodiment, the edge model processes data locally and performs incremental training, reducing data transmission latency and improving prediction response speed. This incremental training continuously incorporates pattern changes in new data, enabling the model to adapt to data drift caused by equipment aging, changing operating conditions, and other factors. Only small-scale models are trained on the edge, while the cloud handles the majority of computing tasks, forming a collaborative "edge-light, cloud-heavy" architecture.

[0189] Edge models can be personalized and trained based on local device characteristics, while cloud models integrate global data while taking into account universality.

[0190] Specifically, the models deployed on edge servers or industrial gateways and those deployed on the cloud are as follows:

[0191] The number of attention heads and / or network layers of the model deployed on the edge server or industrial gateway is smaller than that of the model deployed on the cloud;

[0192] The training results of the model deployed in the cloud guide the training direction of the model deployed on the edge server or industrial gateway.

[0193] The core purpose of this embodiment is to achieve resource optimization configuration and training collaborative guidance mechanism by deploying deep neural network models with different structures on edge servers or industrial gateways and in the cloud.

[0194] Edge models choose lightweight structures, such as reducing the number of attention heads (e.g., from 8 to 2), reducing the number of Transformer layers (e.g., from 6 to 2), or adopting lightweight architectures such as MobileNet and TinyBERT.

[0195] The cloud model maintains the complete model structure, uses more attention heads (e.g., 8 heads) and deeper networks (e.g., 6 layers or more), supports higher-dimensional feature extraction and more complex pattern recognition. Lightweight models are deployed on edge devices, while complex models are deployed on cloud servers.

[0196] The cloud model trains a large cloud model based on historical data and real-time data collection, achieving high-quality feature representation and prediction capabilities. It extracts the cloud model's intermediate layer outputs (such as the hidden state of the Transformer encoder), extracts the attention weight distribution or classification probability output, and establishes a "teacher-student" relationship to guide the edge model's learning process.

[0197] Edge model training uses knowledge distillation, using the cloud model output as soft labels to guide the edge model to learn more accurate probability distributions; the weight allocation strategy of key attention heads in the cloud model is migrated to the edge model; and some parameters of the cloud model are used as initialization values ​​of the edge model to accelerate the convergence process.

[0198] Periodically send cloud model updates to the edge to replace old models or perform incremental model updates.

[0199] This embodiment achieves high-performance inference with limited resources through a combination of streamlined architecture and knowledge distillation. Guided by the cloud-based model, the edge model learns similar attention mechanisms and feature extraction pathways, enhancing the interpretability and decision consistency of the overall system. The cloud-based model continuously absorbs new data, dynamically adjusts model parameters, and transfers knowledge back to the edge model, enabling it to adapt promptly to new operating conditions.

[0200] This embodiment builds an efficient distributed intelligent training system through differentiated model structure design and cloud-to-edge knowledge guidance mechanisms. Lightweight models are deployed on the edge to meet resource constraints, while complex models are deployed on the cloud to ensure modeling accuracy. Co-evolution between models is achieved through methods such as knowledge distillation and attention weight migration. This not only addresses the issue of insufficient computing power on edge devices but also significantly improves the robustness, consistency, and adaptability of the model, making it suitable for a variety of industrial intelligent application scenarios that are sensitive to real-time performance, stability, and energy consumption.

[0201] Anything not described in the present invention can be achieved by adopting or drawing on existing technologies.

[0202] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0203] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for predicting industrial equipment failure based on multimodal data, characterized in that: include: According to the multimodal data of industrial equipment operation obtained by preprocessing, the multimodal data is unified in format to obtain structured data, wherein the multimodal data categories include time series data, text data, image and video data; According to the data category, features are extracted from the structured data one by one, and a multimodal fusion feature is obtained through a dynamic fusion mechanism; According to the multimodal fusion features, a fault prediction classification score is obtained through a deep neural network model to perform fault prediction; When the drift parameter of the multimodal data is greater than a preset threshold, the deep neural network model is incrementally trained using the multimodal data collected in real time to update the deep neural network model parameters.

2. The method for predicting industrial equipment failure based on multimodal data according to claim 1, characterized in that: The time series data, text data, image and video data are specifically: The time series data includes at least any one of vibration, temperature, current, voltage, and pressure data collected in real time; The text data includes at least one of a device maintenance log, an operation manual, and a fault record text; The image and video data at least include images or videos recording the operation of the device.

3. The method for predicting industrial equipment failure based on multimodal data according to claim 1, characterized in that: The multimodal data format is unified through preprocessing, specifically: According to the multimodal data categories, processing one by one to obtain data vectors or tensors, and aligning the data vectors or tensors by timestamps; Among them, according to the time series data, a fixed-length or variable-length tensor is obtained through a sliding window method or a dynamic filling and truncation method; Based on the text data, a context-dependent embedding vector is obtained through a pre-trained language model; According to the image and video data, a multi-dimensional feature vector is obtained through a convolutional neural network.

4. The method for predicting industrial equipment failure based on multimodal data according to claim 3, characterized in that: The preprocessing further includes: According to the time series data, noise elimination, missing value filling and outlier processing are performed; Among them, noise elimination is performed by moving average filtering or wavelet denoising or threshold judgment method; Fill missing values ​​through time interpolation or neighboring data feature interpolation; Identify outliers through statistical methods or machine learning algorithms and mark or remove them accordingly.

5. The method for predicting industrial equipment failure based on multimodal data according to claim 1, characterized in that: According to the data category, features are extracted from the structured data one by one, specifically: According to the time series data, the time series features are extracted through the deep network, and the frequency domain features are extracted through the time-frequency analysis method; According to the image video data, the features of the video frame are extracted through the deep visual encoder; Among them, extracting temporal features through deep networks includes extracting short-term local features through one-dimensional convolutional neural networks, extracting long-term dependencies through long short-term memory networks, and extracting periodic dependencies through time convolutional networks.

6. The method for predicting industrial equipment failure based on multimodal data according to claim 5, characterized in that: Through the dynamic fusion mechanism, a multimodal fusion feature is obtained, specifically: Through the attention mechanism, according to the mutual information between the multiple features extracted, the feature fusion weights are set one by one, and the multiple features extracted are fused to obtain a multimodal fusion feature; or, A feature vector is obtained by combining the extracted multiple features through a splicing operation, and a multimodal fusion feature is obtained through a learnable transformation layer model, wherein the learnable transformation layer model is provided with multiple fusion strategies.

7. The method for predicting industrial equipment failure based on multimodal data according to claim 1, characterized in that: The deep neural network model is specifically: The deep neural network model uses a self-attention mechanism to obtain a query vector, a key vector, and a value vector based on the multimodal fusion features at different times. A weighted weight is set for the value vector based on the similarity between the query vector and the key vector, and the multimodal fusion features at multiple times are obtained by weighted summation. Through multi-head attention, the fault prediction classification score is obtained based on multi-time multimodal fusion features to perform fault prediction; The result of the fault prediction includes at least one of normal, potential fault, and high-risk fault.

8. The method for predicting industrial equipment failure based on multimodal data according to claim 1, characterized in that: The drift parameter of the multimodal data is greater than a preset threshold, specifically: Obtaining a drift parameter for each multimodal data category based on the multimodal data at adjacent moments; Obtaining a drift parameter of the multimodal data by weighted averaging or selecting a maximum value based on the plurality of drift parameters; If the drift parameter of the multimodal data is greater than a preset threshold, it is determined that data drift occurs; The method for obtaining the drift parameter includes at least one of KS test, KL divergence, mean comparison and variance comparison.

9. The method for predicting industrial equipment failure based on multimodal data according to claim 8, characterized in that: The deep neural network model is incrementally trained using multimodal data collected in real time, specifically: Models deployed on edge servers or industrial gateways are trained based on multimodal data collected in real time. The model deployed in the cloud is trained based on real-time collected multimodal data and historical multimodal data.

10. The method for industrial equipment failure prediction based on multimodal data according to claim 9, characterized in that: The models deployed on edge servers or industrial gateways and those deployed on the cloud are as follows: The number of attention heads and / or network layers of the model deployed on the edge server or industrial gateway is smaller than that of the model deployed on the cloud; The training results of the model deployed in the cloud guide the training direction of the model deployed on the edge server or industrial gateway.

Images