Equipment fault prediction method and system based on multi-modal data, equipment and medium

Abstract

The invention relates to the technical field of data processing, and particularly provides an equipment fault prediction method and system based on multi-modal data, equipment and a medium, and the method comprises the steps: building a sensor adapter registration center, configuring and registering corresponding data adapters and feature extractors for various types of sensors, so as to achieve the dynamic extension of the types; after receiving the original data of the heterogeneous sensor, calling a corresponding data adapter from a registration center according to the type to generate preprocessed data; calling a corresponding feature extractor, extracting features of different time scales from the preprocessed data, and constructing a multi-dimensional feature vector library; performing cross-modal fusion on the features in the feature library by adopting an attention mechanism to generate a fusion feature vector; and finally, inputting the fused feature vector into a deep learning model to carry out fault prediction and uncertainty quantization, and outputting a prediction result. According to the invention, the fault identification precision and the system practicability are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a device fault prediction method, system, device and medium based on multi-modal data. BACKGROUND

[0002] In the field of industrial equipment state monitoring and fault prediction, using multi-modal heterogeneous sensors (such as vibration, temperature, current, etc.) for data acquisition has become a trend. However, the existing methods have significant limitations: first, the system is usually preset for fixed sensor types, lacking scalability, and when new sensors are introduced, the core code often needs to be modified or even the system needs to be restarted, making it difficult to adapt to the dynamic changes of the sensor network in modern intelligent factories. Second, the processing method of heterogeneous data is rigid and cannot achieve adaptive cleaning and standardization at the perception level. More importantly, traditional feature fusion methods (such as simple concatenation or weighted average) cannot deeply mine the complex spatio-temporal correlation and interaction dependence between different sensor modalities, resulting in insufficient early warning capability for early and complex faults. In addition, the prediction results often lack quantification of their own uncertainty, making it difficult for maintenance personnel to assess risk confidence, thereby affecting the accuracy of decision-making and the practicality of the system. Therefore, the industry urgently needs a fault prediction method that can dynamically adapt to sensor types, intelligently fuse multi-modal data, and quantify the reliability of the prediction. SUMMARY

[0003] In view of the above deficiencies of the prior art, the present application provides a device fault prediction method, system, device and medium based on multi-modal data to solve the above technical problems.

[0004] In a first aspect, the present application provides a device fault prediction method based on multi-modal data, comprising: Establishing a sensor adapter registration center, configuring and registering corresponding data adapters and feature extractors for multiple types of sensors to support dynamic expansion of sensor types; Receiving raw data from multiple heterogeneous sensors, calling the corresponding data adapter based on the type of the sensor, and converting the raw data into preprocessed data; Calling the corresponding feature extractor based on the type of the sensor, extracting multi-scale features associated with fault mode time characteristics from the preprocessed data, and constructing a multi-dimensional feature vector library; Using an attention mechanism to perform cross-modal fusion on the multi-scale features in the multi-dimensional feature vector library to generate a fusion feature vector; Inputting the fusion feature vector into a deep learning model for fault prediction and uncertainty quantification, and outputting a prediction result.

[0005] In an optional embodiment, a sensor adapter registry center is established, and a corresponding data adapter and feature extractor are configured and registered for each type of sensor, including: For each type of sensor, a corresponding data adapter and a corresponding feature extractor are developed or configured in advance, and the sensor type is associated and bound with its corresponding data adapter and feature extractor; The association and binding relationship is stored in a retrievable data structure in the sensor adapter registry center, forming a sensor processing capability library that supports dynamic query; The data adapter is configured to clean, standardize and format the raw data of the corresponding type of sensor; The feature extractor is configured to extract features related to the sensor type for fault prediction from the data processed by the data adapter.

[0006] In an optional embodiment, the data adapter is configured to perform one or more of the following processes: The raw byte stream or built-in protocol data from the sensor is parsed into a standardized data format within the system; The parsed data is subjected to a cyclic redundancy check or outlier detection algorithm to identify and eliminate invalid or erroneous data; The raw count value is converted to a value with actual physical meaning according to the sensor sensitivity coefficient.

[0007] In an optional embodiment, the feature extractor is configured to extract one or more of the following features from the preprocessed data: Time domain features: including mean, root mean square, peak value, peak to peak value, kurtosis or waveform factor; Frequency domain features: including spectral centroid, frequency variance or specific frequency band energy proportion obtained by fast Fourier transform; Time-frequency domain features: including wavelet coefficient energy at each layer obtained by wavelet transform.

[0008] In an optional embodiment, based on the type of sensor, the corresponding feature extractor is called to extract multi-scale features associated with fault mode time characteristics from the preprocessed data, and a multi-dimensional feature vector library is constructed, including: Based on the preset fault mode time characteristics, the feature extractor is invoked to perform multi-time window analysis on the preprocessed data to extract: short-time scale features, including time-domain statistical features and frequency-domain spectral features for detecting instantaneous anomalies; medium-time scale features, including statistical trend features for monitoring slow changing trends; and long-time scale features, including degradation indicators and periodic features for analyzing long-term degradation patterns. All features extracted from multiple heterogeneous sensors at the same timestamp are organized into a unified multidimensional feature vector. The multidimensional feature vectors are stored in a multidimensional feature vector library according to the time series, and incremental updates of the multidimensional feature vector library are supported.

[0009] In an optional implementation, an attention mechanism is employed to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library to generate a fused feature vector, including: The feature sequence of each sensor mode is input into a multi-head self-attention layer, the importance weights of features at different time points within the mode are calculated, and a weighted intra-modal context feature representation is generated. The weighted modal context feature representations are input into a cross-modal attention layer. By calculating the key-value query relationship between different modal feature sequences, the cross-modal attention weights reflecting the interaction and dependency between modalities are obtained. Based on the cross-modal attention weights, features from different sensor modalities are weighted and aggregated, and the weighted features are concatenated or summed to generate a fused feature vector.

[0010] In an optional implementation, the fused feature vector is input into a deep learning model for fault prediction and uncertainty quantification, and the prediction result is output, including: The fused feature vector is input into a pre-trained deep learning model, which outputs a prediction result including the fault type, probability of occurrence, and expected occurrence time. When making model predictions, Monte Carlo Dropout is enabled to perform multiple forward propagation calculations on the same fused feature vector. In each calculation, some network neurons are randomly discarded, thus obtaining a set of differential prediction results. Based on the distribution of prediction results generated by the multiple random inferences, its statistical variance is calculated to quantify the uncertainty of this prediction. The uncertainty is mapped to a confidence score. When the score is lower than a preset threshold, the system generates a prompt message and suggests manual review.

[0011] Secondly, the present invention provides a device fault prediction system based on multimodal data, comprising: The registration management module is used to establish a sensor adapter registration center, configure corresponding data adapters and feature extractors for various types of sensors and register them to support the dynamic expansion of sensor types; An adaptation processing module is used to receive raw data from multiple heterogeneous sensors, and call the corresponding data adapter based on the sensor type to convert the raw data into preprocessed data. The feature extraction module is used to call the corresponding feature extractor based on the sensor type, extract multi-scale features associated with the time characteristics of the fault mode from the preprocessed data, and construct a multi-dimensional feature vector library; The feature fusion module is used to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library using an attention mechanism to generate a fused feature vector. The fault prediction module is used to input the fused feature vector into the deep learning model for fault prediction and uncertainty quantification, and output the prediction result.

[0012] Thirdly, a device is provided, comprising: Memory used to store equipment failure prediction programs based on multimodal data; A processor is configured to implement the steps of the device fault prediction method based on multimodal data as provided in the first aspect when executing the device fault prediction program based on multimodal data.

[0013] Fourthly, a computer-readable medium is provided, on which a device fault prediction program based on multimodal data is stored, wherein when the device fault prediction program based on multimodal data is executed by a processor, the device fault prediction program based on multimodal data implements the steps of the device fault prediction method based on multimodal data provided in the first aspect.

[0014] The beneficial effects of this invention are as follows: The device fault prediction method, system, device, and medium based on multimodal data for doctor-patient interaction management provided by this invention achieve dynamic registration and plug-and-play functionality for sensor types by establishing a sensor adapter registration center, fundamentally solving the problems of poor system scalability and weak adaptability. Through multi-scale feature extraction and attention-based cross-modal fusion, the inherent correlations of heterogeneous data can be deeply explored, significantly improving the accuracy of identifying complex faults and early subtle anomalies. In addition, the introduction of an uncertainty quantification mechanism enables confidence assessment of the prediction results, greatly enhancing the reliability of decision support and the practicality of the system, providing a safer and more reliable predictive maintenance solution for intelligent manufacturing. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0021] The equipment fault prediction method based on multimodal data provided in this embodiment of the invention is executed by a computer device, and correspondingly, the equipment fault prediction system based on multimodal data runs in the computer device.

[0022] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a device fault prediction system based on multimodal data. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0023] like Figure 1 As shown, the method includes: S1. Establish a sensor adapter registration center to configure and register corresponding data adapters and feature extractors for various types of sensors to support dynamic expansion of sensor types; S2. Upon receiving raw data from multiple heterogeneous sensors, the corresponding data adapter is invoked based on the sensor type to convert the raw data into preprocessed data; S3. Based on the sensor type, call the corresponding feature extractor to extract multi-scale features associated with the time characteristics of the fault mode from the preprocessed data, and construct a multi-dimensional feature vector library; S4. Employ an attention mechanism to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library to generate a fused feature vector; S5. Input the fused feature vector into the deep learning model for fault prediction and uncertainty quantification, and output the prediction result.

[0024] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0025] S101. Sensor Processing Capability Registration First, the system pre-develops or configures corresponding processing modules for each type of sensor. For example, for the "XYZ-100 vibration acceleration sensor," the development team will write a corresponding data adapter and a corresponding feature extractor.

[0026] The data adapter is configured to perform the following processing flow: Protocol parsing: The raw byte stream (such as [0x01,0x03,0x00,0x0A,...]) transmitted from the sensor via the ModbusTCP protocol is parsed into a unified JSON format within the system, containing fields such as "timestamp", "sensor_id" and "raw_value".

[0027] Data cleaning: Apply an outlier detection algorithm based on the 3σ criterion to the parsed "raw_value" to identify and remove transient spikes caused by transmission interference.

[0028] Dimensional conversion: Based on the sensor's sensitivity coefficient of 100mV / g, the original voltage count value (e.g., 2500) is converted into an acceleration value (25m / s²) with practical physical meaning. 2 ).

[0029] The feature extractor is configured to extract the following features from the preprocessed data: Time-domain characteristics: Calculate the root mean square (RMS) and kurtosis of the vibration signal to reflect the total vibration energy and impact components.

[0030] Frequency domain characteristics: Perform Fast Fourier Transform (FFT) on the signal to calculate the energy proportion of the spectrum in the 1000Hz to 2000Hz frequency band in order to monitor specific fault frequencies of the bearing.

[0031] Subsequently, the system associates and binds the sensor type "VIB_XYZ100" with the data adapter and feature extractor. This binding relationship is persistently stored in the registry center's database in the form of a key-value pair data structure, forming a sensor processing capability library that supports dynamic queries. An example of its logical storage format is as follows: The data adapter "Adapter_VIB_XYZ100.jar" and the feature extractor "Extractor_VIB_XYZ100.jar" are configured and associated for the sensor type "VIB_XYZ100"; similarly, the data adapter "Adapter_TEMP_PT100.jar" and the feature extractor "Extractor_TEMP_PT100.jar" are configured and associated for the sensor type "TEMP_PT100". These predefined mapping relationships together constitute a sensor processing capability library that supports dynamic queries.

[0032] S102. Runtime Dynamic Calls and Data Processing When a new XYZ-100 vibration acceleration sensor is connected to the system, the system workflow is as follows: The system identifies the sensor type as "VIB_XYZ100" by reading sensor metadata or sending detection commands.

[0033] Subsequently, the system sent a query request to the sensor adapter registry.

[0034] The registry center retrieves the corresponding data adapter (Adapter_VIB_XYZ100.jar) and feature extractor (Extractor_VIB_XYZ100.jar) based on the type "VIB_XYZ100" and dynamically loads them into the runtime environment.

[0035] Subsequently, the raw data stream uploaded by the sensor is automatically parsed, cleaned, and transformed by the data adapter to generate preprocessed data.

[0036] The preprocessed data is then passed to the feature extractor, which extracts features in both the time and frequency domains.

[0037] Finally, the extracted features are organized into feature vectors for use by upper-layer applications (such as fault prediction models).

[0038] If a new, unregistered sensor is connected, maintenance personnel can configure its model, protocol documentation, and processing logic through the management interface and package it into a new adapter and extractor, registering it in the central library. This enables dynamic expansion of the system without requiring system downtime or restart.

[0039] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0040] S201. Sensor Type Identification and Adapter Activation The system continuously monitors the data access layer. Taking the monitoring of a CNC machine tool as an example, when the system receives a raw data message, the sensor type identifier first parses the message. The identifier can determine the sensor type by reading the device identifier in the message header (such as the slave address 0x01 in the Modbus protocol) or by parsing its unique data frame format.

[0041] For example, the system identifies that slave address 0x01 corresponds to a registered "VIB_XYZ100" type vibration sensor. It also identifies a specific header in another data stream as corresponding to a "TEMP_PT100" type temperature sensor. Upon successful type identification, the system immediately sends a query request to the sensor adapter registry. Based on the identified type string (e.g., "VIB_XYZ100"), the registry retrieves the bound data adapter program Adapter_VIB_XYZ100.jar and instantiates and loads it into the system runtime environment using Java class loading or similar dynamic linking technology. For the "TEMP_PT100" type, Adapter_TEMP_PT100.jar is also dynamically loaded.

[0042] S202. Specific processing flow of data adapter The system routes raw, unprocessed byte stream data to the corresponding data adapter instance. Each adapter encapsulates specific data processing logic for its sensor type.

[0043] For the vibration sensor adapter Adapter_VIB_XYZ100.jar: Protocol parsing: It parses the raw Modbus TCP byte stream (e.g., [0x01, 0x03, 0x04, 0x00, 0x0A, 0x...]) to extract the payload. According to the datasheet for this sensor model, the data is a 2-byte signed integer representing the raw voltage value.

[0044] Data verification and cleaning: The adapter applies Cyclic Redundancy Check (CRC) to the parsed continuous data points to ensure frame integrity. Subsequently, outlier detection based on the IsolationForest algorithm is performed to automatically identify and remove physically impossible peaks caused by transient electromagnetic interference.

[0045] Dimensional conversion: Finally, based on the preset sensor sensitivity coefficient (e.g., 100mV / g), the original voltage value is converted to units with SI (m / s²). 2 The vibration acceleration value of ). At this point, the original byte stream has been transformed into a clean, timestamped, physically meaningful data record, for example: {timestamp:1621234567890,value:9.8,unit:'m / s'} 2 '}.

[0046] For the temperature sensor adapter Adapter_TEMP_PT100.jar: The processing flow is similar but the logic is different. It may convert the PT100 resistance value into a temperature value in degrees Celsius (°C) by calling a calculation program based on the Callendar-VanDusen formula, and then perform a moving average filter to smooth the reading.

[0047] S203. Output preprocessed data All invoked data adapters work in parallel, processing their respective sensor data streams. They output preprocessed data with a uniform format, timestamp alignment, and cleaned and normalized values. This preprocessed data is published to a unified data bus or message queue (such as a Kafka topic), providing clean and usable input for subsequent feature extractors.

[0048] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0049] S301. Multi-time window analysis and feature extraction Based on the preset fault mode time characteristics, the system calls the corresponding feature extractor loaded from the registry center to perform parallel analysis of the preprocessed data stream across multiple time windows.

[0050] The short timescale feature extraction system (e.g., 1-second window, 100Hz sampling) calls the feature extractor of the vibration sensor to perform real-time analysis on the vibration acceleration data of the most recent second (assuming 100 data points).

[0051] Time-domain statistical characteristics: Extract the root mean square (RMS) value of the data in this window to reflect the overall energy; calculate kurtosis, which is highly sensitive to shock anomalies.

[0052] Frequency domain spectral characteristics: Perform a Fast Fourier Transform (FFT) on this signal to calculate the spectral centroid and extract the sideband energy near the characteristic frequency of the bearing outer ring fault (such as 125Hz) to detect instantaneous jamming or pitting of the bearing.

[0053] The system uses the feature extractor of the temperature sensor to perform trend analysis on the temperature readings of the most recent 10 minutes (e.g., 600 data points at 1-second intervals) for medium-time scale feature extraction (e.g., 10-minute window, moving calculation).

[0054] Statistical trend characteristics: Calculate the moving average of the time window to smooth short-term fluctuations; calculate the slope of temperature change during the time period through linear regression to monitor the slow overheating trend of the equipment.

[0055] Long-term feature extraction (e.g., 30-day window, daily aggregation) systems call the feature extractor of the same vibration sensor, but perform macroscopic analysis on a historical RMS value sequence of up to 30 days.

[0056] Degradation Index: The ratio of the current day's RMS value to the RMS value of the device in its initial health state is used as the Health Index (HI). This value starts from 1.0 and slowly increases, indicating long-term performance degradation.

[0057] Periodic characteristics: Autocorrelation analysis was performed on the 30-day daily average vibration energy sequence to identify possible periodic fatigue patterns related to production shifts.

[0058] S302. Feature Vector Organization and Storage All features are extracted with precise timestamps. The feature vector assembler within the system collects the latest features extracted from all heterogeneous sensors (vibration, temperature, current, etc.) at fixed time intervals (e.g., whole seconds T0).

[0059] For example, at timestamp T0, the assembler will collect: From vibration sensors: RMS_T0, Kurtosis_T0, Spectral_Centroid_T0; From the temperature sensor: Moving_Average_T0, Slope_T0; Health_Index_T0 is from long-term analysis of vibration sensors; Subsequently, the assembler organizes these features into a unified multidimensional feature vector, which takes the form of: [RMS_T0,Kurtosis_T0,Spectral_Centroid_T0,Moving_Average_T0,Slope_T0,Health_Index_T0, ...].

[0060] S303. Incremental Update of Feature Vector Library Finally, this multidimensional feature vector is appended to an external multidimensional feature vector library (such as a time-series database like InfluxDB or TimescaleDB). This database is designed to support high-throughput time-series data writing, enabling incremental updates. New feature vectors are continuously appended without overwriting historical data, forming a continuously growing time-series feature dataset that provides a complete data foundation for subsequent online model learning and historical data backtesting analysis.

[0061] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0062] S401. Intramodal attention computation (mining unimodal temporal dependencies) The system first inputs the feature sequence of each sensor mode into an independent multi-head self-attention layer. Taking vibration mode and temperature mode as examples: Vibration mode processing: The vibration feature sequence of the past 60 time steps (each time step includes features such as RMS and kurtosis) is input into the vibration multi-head self-attention layer. This layer calculates the importance weights of different time points within the sequence. For example, it may find that the weight of time t-5 (when an abnormal peak in kurtosis occurs) is much higher than that of other stationary times. By performing a weighted summation on the entire sequence, a fixed-length vibration context feature representation C that represents the key contextual information of the vibration sequence is generated. vib .

[0063] Temperature modal processing: Similarly, the temperature feature sequence is input into its corresponding multi-head self-attention layer. This layer may focus more on the temperature rise trend in the most recent time steps (t-1, t-2) and generate a weighted temperature context feature representation C. temp .

[0064] Input: For a given mode m, its feature sequence is represented as:

[0065] Where T is the number of time steps (e.g., 60). It is the dimension of the modality feature.

[0066] Self-attention calculation: Linear transformation: Transforming the input sequence through a learnable weight matrix The mapping is a vector of queries, keys, and values.

[0067]

[0068]

[0069]

[0070] Attention score and weights: Calculate the dot product of each query and all keys, scale the result, and apply the Softmax function to obtain the attention weight matrix. .

[0071]

[0072] Here, A m [i,j] represents the degree of attention the model pays to the j-th time step at the i-th time step.

[0073] Weighted summation: The value vector Vm is weighted and summed using attention weights to obtain a new context-aware feature representation for each time step.

[0074]

[0075] Generating intramodal contextual feature representations: To obtain a fixed-length vector to represent the entire sequence, Z is typically used. m Perform pooling (e.g., take the output of the last time step, or perform average pooling). Employ a pooling operation Pool( (e.g., averaging over the time dimension):

[0076] C here m This refers to the context feature vector (such as C) that represents key information about mode m. vib ).

[0077] Multi-head self-attention: The process described above is single-head attention. In practice, multiple heads (e.g., h heads) are used, each with independent... Generate h different The outputs of these heads are concatenated and then passed through a linear layer. The fusion process yields the context representation C of mode m. m .

[0078] S402. Cross-modal attention computation (mining intermodal interaction relationships) Subsequently, the system represents all these weighted modal context features (C vib C temp C current ...) are combined into a new sequence and input into a cross-modal attention layer.

[0079] In this layer, taking the aforementioned vibration anomaly as an example, the system is represented by vibration modes C. vib As a query, C is represented by temperature modality. temp Current mode representation C current These are used as keys and values. By calculating the similarity between the query and all keys (i.e., attention scores), the model can learn that when vibrations are abnormal (C... vib As a query, which parts of the temperature mode should the system "focus on" (from C)? temp (Extract Value). This process calculates cross-modal attention weights that reflect intermodal interactions and dependencies.

[0080] The goal of cross-modal attention computation is to use the contextual information of one modality as a guide (Query) to query and aggregate relevant information from other modalities.

[0081] Input: A set of context feature representations obtained after all modalities undergo intra-modal attention. , where M is the total number of modes.

[0082] Select query mode: Select a representation C of mode p (e.g., vibration mode). p As a query.

[0083]

[0084] Process other modalities as keys and values: Use the representation of all modalities (usually including the query modal itself) as the source of keys and values.

[0085]

[0086]

[0087] in , , It is a learnable weight matrix. The Concat operation stacks M vectors into an M×d matrix. k The matrix.

[0088] Calculate query With all keys The similarity.

[0089]

[0090] This weight vector Each element This indicates the importance of the j-th mode in the context of query mode p (i.e., vibration mode accounts for 0.6, temperature mode accounts for 0.3).

[0091] S403. Adaptive Weighted Fusion Finally, the system dynamically weights and aggregates feature representations from all sensor modes based on the calculated cross-modal attention weights. For example, the final weight distribution might be: vibration mode 0.6, temperature mode 0.3, and current mode 0.1. This indicates that in the current context, vibration is the most critical indicator, while temperature provides important corroborating information.

[0092] The system concatenates these weighted modal representations to generate a comprehensive fusion feature vector F that highlights key information for current fault warning. fused This vector is then fed into a downstream deep learning prediction model.

[0093] Using the calculated cross-modal attention weights The original modal context representation C m (Or a value vector that has undergone another linear transformation) is weighted and summed to generate the final fused feature vector F. fused .

[0094]

[0095] in It is a weight vector The j-th element in It is an optional linear transformation weight matrix used to adjust the dimension or feature space of each modality representation before fusion. The final result is... You can then input the downstream prediction model.

[0096] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0097] S501. Deep Learning Model Prediction F will be fused with feature vector F fused The model is input into a pre-trained deep learning model for prediction. This model employs an encoder-decoder architecture and consists of the following components: The encoder consists of a two-layer Long Short-Term Memory (LSTM) network, which is used to further capture temporal dependencies in the fused features.

[0098] The decoder consists of three parallel fully connected (dense) layers, each responsible for the output of a different task. Fault type classification layer: Outputs a probability distribution, such as [bearing inner ring fault: 0.75, normal: 0.20, bearing outer ring fault: 0.05]; Occurrence Probability Regression Layer: Outputs a scalar representing the overall probability of a failure occurring within a specified future time period, such as 0.82; Expected Time Regression Layer: Outputs a scalar representing the estimated time from the predicted failure occurrence, such as 48.5 hours.

[0099] S502. Uncertainty Quantification Based on Monte Carlo Dropout To quantify the uncertainty of the prediction, the Dropout layer is kept active during model prediction. For the same fused feature vector F... fused Perform T forward propagation calculations (e.g., T=50). In each forward propagation, a certain percentage (e.g., 20%) of neurons in the network are randomly discarded (set to zero), thus obtaining a set of differential prediction results.

[0100] For the fault type, the number of times "bearing inner ring fault" occurred in 50 predictions was recorded as 38, and the mean of its probability estimate was 0.76.

[0101] For the time of occurrence, record the results of 50 predictions to form a numerical set [45.2, 52.1, 47.8,..., 49.3] hours.

[0102] S503. Confidence Calculation and Decision Support The uncertainty quantification module calculates the statistical variance of these 50 prediction results. Taking the occurrence time as an example, its variance is calculated as σ. 2 = 15.2.

[0103] The system presets a variance threshold θ = 10.0. Since the current variance of 15.2 > the threshold of 10.0, the system determines that the uncertainty of this prediction is high. The confidence estimator maps this uncertainty to a confidence score, for example, using the formula: Confidence = max(0, 1 - σ) 2 The confidence level was calculated to be 0.696 (or 69.6%).

[0104] Because the confidence score is lower than the preset confidence threshold (e.g., 0.75), when generating the prediction report, the output device will not only give the core conclusion that "the probability of bearing inner ring failure is expected to occur within the next 48.5 hours is 82%", but will also add a significant prompt message: "The confidence level of this prediction is low (69.6%), and manual review is recommended", thus reminding maintenance personnel to make a final decision based on the actual situation of the equipment.

[0105] This implementation method greatly improves the reliability of prediction results and the practicality of the system by quantifying the uncertainty of prediction and introducing a confidence assessment mechanism, providing strong support for accurate decision-making in industrial settings.

[0106] In some embodiments, the equipment fault prediction system based on multimodal data may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the equipment fault prediction system based on multimodal data may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for predicting equipment failures based on multimodal data.

[0107] In this embodiment, the equipment fault prediction system based on multimodal data can be divided into multiple functional modules according to its functions, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0108] The registration management module is used to establish a sensor adapter registration center, configure corresponding data adapters and feature extractors for various types of sensors and register them to support the dynamic expansion of sensor types; An adaptation processing module is used to receive raw data from multiple heterogeneous sensors, and call the corresponding data adapter based on the sensor type to convert the raw data into preprocessed data. The feature extraction module is used to call the corresponding feature extractor based on the sensor type, extract multi-scale features associated with the time characteristics of the fault mode from the preprocessed data, and construct a multi-dimensional feature vector library; The feature fusion module is used to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library using an attention mechanism to generate a fused feature vector. The fault prediction module is used to input the fused feature vector into the deep learning model for fault prediction and uncertainty quantification, and output the prediction result.

[0109] Figure 3The device fault prediction method based on multimodal data provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0110] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0111] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0112] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0113] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0114] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0115] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0116] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0117] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0118] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0120] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A method for predicting equipment failures based on multimodal data, characterized in that, include: Establish a sensor adapter registration center to configure and register corresponding data adapters and feature extractors for various types of sensors, in order to support the dynamic expansion of sensor types; Upon receiving raw data from multiple heterogeneous sensors, the corresponding data adapter is invoked based on the sensor type to convert the raw data into preprocessed data. Based on the sensor type, the corresponding feature extractor is invoked to extract multi-scale features associated with the time characteristics of the failure mode from the preprocessed data, and a multi-dimensional feature vector library is constructed. An attention mechanism is used to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library to generate a fused feature vector. The fused feature vector is input into a deep learning model for fault prediction and uncertainty quantification, and the prediction result is output.

2. The method according to claim 1, characterized in that, Establish a sensor adapter registration center to configure and register corresponding data adapters and feature extractors for various types of sensors, including: For each type of sensor, a corresponding data adapter and a corresponding feature extractor are pre-developed or configured, and the sensor type is associated and bound with its corresponding data adapter and feature extractor. The association and binding relationship is persistently stored in the sensor adapter registry center in the form of a searchable data structure, forming a sensor processing capability library that supports dynamic querying; The data adapter is configured to perform data cleaning, standardization, and format unification on raw data from corresponding types of sensors. The feature extractor is configured to extract features related to the sensor type for fault prediction from the data processed by the data adapter.

3. The method according to claim 2, characterized in that, The data adapter is configured to perform one or more of the following processes: Parse the raw byte stream or built-in protocol data from the sensor into a unified, standardized data format that is consistent with the system. Cyclic redundancy check or outlier detection algorithms are applied to the parsed data to identify and remove invalid or erroneous data; The original count values ​​are converted into values ​​with actual physical meaning based on the sensor sensitivity coefficient.

4. The method according to claim 2, characterized in that, The feature extractor is configured to extract one or more of the following features from the preprocessed data: Time-domain characteristics include mean, root mean square value, peak value, peak-to-peak value, kurtosis, or waveform factor. Frequency domain characteristics include the spectral centroid, frequency variance, or energy proportion of a specific frequency band obtained through Fast Fourier Transform. Time-frequency domain characteristics: including the energy of wavelet coefficients at each level obtained through wavelet transform.

5. The method according to claim 1, characterized in that, Based on the sensor type, the corresponding feature extractor is invoked to extract multi-scale features associated with the time characteristics of the failure mode from the preprocessed data, and a multi-dimensional feature vector library is constructed, including: Based on the preset fault mode time characteristics, the feature extractor is invoked to perform multi-time window analysis on the preprocessed data to extract multi-scale features, wherein the multi-time scale features are: short-time scale features, including time-domain statistical features and frequency-domain spectral features for detecting instantaneous anomalies; medium-time scale features, including statistical trend features for monitoring slow changing trends; and long-time scale features, including degradation indicators and periodic features for analyzing long-term degradation patterns. All features extracted from multiple heterogeneous sensors at the same timestamp are organized into a unified multidimensional feature vector. The multidimensional feature vectors are stored in a multidimensional feature vector library according to the time series, and incremental updates of the multidimensional feature vector library are supported.

6. The method according to claim 1, characterized in that, An attention mechanism is used to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library to generate a fused feature vector, including: The feature sequence of each sensor mode is input into a multi-head self-attention layer, the importance weights of features at different time points within the mode are calculated, and a weighted intra-modal context feature representation is generated. The weighted modal context feature representations are input into a cross-modal attention layer. By calculating the key-value query relationship between different modal feature sequences, the cross-modal attention weights reflecting the interaction and dependency between modalities are obtained. Based on the cross-modal attention weights, features from different sensor modalities are weighted and aggregated, and the weighted features are concatenated or summed to generate a fused feature vector.

7. The method according to claim 1, characterized in that, The fused feature vector is input into a deep learning model for fault prediction and uncertainty quantification, and the prediction results are output, including: The fused feature vector is input into a pre-trained deep learning model, which outputs a prediction result including the fault type, probability of occurrence, and expected occurrence time. When making model predictions, MonteCarlo Dropout is enabled to perform multiple forward propagation calculations on the same fused feature vector. In each calculation, some network neurons are randomly discarded to obtain a set of prediction results with differences. Based on the distribution of prediction results generated by the multiple random inferences, its statistical variance is calculated to quantify the uncertainty of this prediction. The uncertainty is mapped to a confidence score. When the score is lower than a preset threshold, the system generates a prompt for manual review.

8. A device fault prediction system based on multimodal data, characterized in that, include: The registration management module is used to establish a sensor adapter registration center, configure corresponding data adapters and feature extractors for various types of sensors and register them to support the dynamic expansion of sensor types; An adaptation processing module is used to receive raw data from multiple heterogeneous sensors, and call the corresponding data adapter based on the sensor type to convert the raw data into preprocessed data. The feature extraction module is used to call the corresponding feature extractor based on the sensor type, extract multi-scale features associated with the time characteristics of the fault mode from the preprocessed data, and construct a multi-dimensional feature vector library. The feature fusion module is used to perform cross-modal fusion of multi-scale features in the multi-dimensional feature vector library using an attention mechanism to generate a fused feature vector. The fault prediction module is used to input the fused feature vector into the deep learning model for fault prediction and uncertainty quantification, and output the prediction result.

9. A device fault prediction device based on multimodal data, characterized in that, include: Memory used to store equipment failure prediction programs based on multimodal data; A processor is configured to implement the steps of the device fault prediction method based on multimodal data as described in any one of claims 1-7 when executing the device fault prediction program based on multimodal data.

10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a device fault prediction program based on multimodal data, which, when executed by a processor, implements the steps of the device fault prediction method based on multimodal data as described in any one of claims 1-7.

Images