A method and apparatus for fault equipment analysis based on multimodal data

Abstract

This invention discloses a fault equipment analysis method and apparatus based on multimodal data, belonging to the technical field of equipment condition monitoring. The method includes: real-time acquisition of multimodal data of the equipment to be monitored; extraction of low-dimensional feature vectors from the multimodal data using a preset encoding architecture, and optimization transformation of the low-dimensional feature vectors to obtain modal feature vectors; determination of state deviation values ​​based on the modal feature vectors, and determination of whether the monitored equipment is abnormal based on the magnitude of the state deviation values; if the monitored equipment is determined to be abnormal, fault analysis processing is performed based on the modal feature vectors to obtain the analysis results. This invention can monitor the actual operating status of equipment in real time using multimodal data collected by various sensors, and when the equipment is abnormal, it performs equipment analysis based on the data of abnormal time nodes in the multimodal data to achieve secondary verification of the abnormal state, which can improve the accuracy of monitoring and simplify the monitoring steps to improve processing efficiency.

Description

Technical Field

[0001] This invention relates to the technical field of equipment condition monitoring, and in particular to a method and apparatus for fault equipment analysis based on multimodal data. Background Technology

[0002] With the continuous development of the power industry, the power system infrastructure is also increasing. The operating environment of these infrastructures and equipment is complex. Once any piece of equipment fails, it will affect the stable operation of the entire power system. Therefore, it is necessary to monitor its operating status in real time and effectively to avoid safety problems.

[0003] To monitor equipment for anomalies in real time, one common approach is to deploy a variety of sensor devices, including partial discharge sensors, temperature and humidity sensors, accelerometers, and detectors, to monitor the equipment status through each device. Then, machine learning techniques, such as support vector machines (SVM) and random forests (RF), are introduced to determine whether the equipment is abnormal based on any of the monitoring data.

[0004] However, the above method has the following technical problems: Although the above method can determine whether the equipment is abnormal based on any data, the equipment is easily affected by the various connected devices and fluctuates during operation. Therefore, after the equipment is determined to be abnormal, technicians often need to spend a lot of time and energy to conduct a secondary test on the equipment in order to verify the actual status of the abnormal equipment and carry out subsequent maintenance. Therefore, the above method is not only difficult to determine the actual operating status of the equipment, but also has cumbersome steps, low processing efficiency, and is difficult to meet the current application needs. Summary of the Invention

[0005] This invention provides a fault equipment analysis method and apparatus based on multimodal data, which can solve the technical problems of existing technologies that make it difficult to determine the actual operating status of equipment and have low processing efficiency.

[0006] A first aspect of this invention provides a method for fault equipment analysis based on multimodal data, the method comprising: Real-time acquisition of multimodal data from the device under monitoring, wherein the multimodal data is obtained by collecting different data from different monitoring devices while the device under monitoring is running in real time, and the data quality is optimized; A preset encoding architecture is invoked to extract low-dimensional feature vectors from the multimodal data, and the low-dimensional feature vectors are optimized and transformed to obtain modal feature vectors. The state deviation value is determined based on the modal feature vector, and the magnitude of the state deviation value is used to determine whether the monitored device is abnormal. The state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state. If the monitored device is determined to be abnormal, fault analysis processing is performed based on the modal feature vector to obtain the analysis results. The fault analysis processing is the process of determining the fault type and severity based on the abnormal time nodes of the modal feature vector.

[0007] This invention can monitor the actual operating status of equipment in real time through multimodal data collected by various sensors. When the equipment is abnormal, it can analyze the equipment based on the data of abnormal time nodes in the multimodal data to achieve secondary verification of the abnormal status. This eliminates the need for technicians to spend a lot of time and energy to conduct secondary testing on the equipment, thereby improving the accuracy of monitoring and simplifying the monitoring steps to improve processing efficiency and meet existing application needs.

[0008] In conjunction with the first aspect, in one implementation, the step of performing fault analysis processing based on the modal feature vector to obtain the analysis result includes: Anomaly time points are obtained from the modal feature vector based on a sliding window, where the anomaly time point is the time point when the state deviation value meets the threshold. Obtain the abnormal duration data corresponding to the abnormal time node, wherein the abnormal duration data is the abnormal data within a preset duration before and after the abnormal time node. The preset anomaly analysis model is invoked to determine the fault type and severity of the monitored equipment based on the anomaly duration data.

[0009] In conjunction with the first aspect, in one implementation, the step of invoking a preset anomaly analysis model to determine the fault type and severity of the monitored device based on the anomaly duration data includes: The fault type is obtained by classifying the abnormal duration data using the loss function of a preset anomaly analysis model. Based on the key feature parameters corresponding to the fault type, abnormal parameter values ​​are extracted from the abnormal duration data, and the severity of the fault is determined by comparing the values ​​of the abnormal parameters with several preset fault thresholds.

[0010] In conjunction with the first aspect, in one implementation, the preset anomaly analysis model is a model trained using historical data prior to the anomaly time point; before training the preset anomaly analysis model, the method further includes: Historical modal data of the device to be monitored is acquired, and the historical modal data is integrated and verified to obtain historical training data; If the amount of historical training data meets the requirements, then the historical training data is used for model training. If the amount of historical training data is insufficient, the sample size of the historical training data is expanded by data augmentation and supplementary data collection methods, and the expanded historical training data is used for model training.

[0011] In conjunction with the first aspect, in one implementation, determining the state deviation value based on the modal feature vector and determining whether the monitored device is abnormal based on the magnitude of the state deviation value includes: The Euclidean distance and Mahalanobis distance between the modal feature vector and the preset state baseline are calculated respectively, wherein the preset state baseline is a vector generated based on the normal operation data of the device under health monitoring. The state deviation value is calculated using the Euclidean distance and the Mahalanobis distance; If the state deviation value is greater than the preset deviation threshold, the monitored device is determined to be abnormal. If the state deviation value is less than or equal to the preset deviation threshold, the device under monitoring is determined to be normal.

[0012] In conjunction with the first aspect, in one implementation, the step of invoking a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, and optimizing and transforming the low-dimensional feature vectors to obtain modal feature vectors, includes: A preset encoding architecture is invoked to extract low-dimensional feature vectors from the multimodal data, wherein the preset encoding architecture is obtained by setting the number of layers in the encoder and decoder and training them. The low-dimensional feature vectors are optimized and transformed to obtain modal feature vectors that conform to a custom scaling ratio. The optimization and transformation process includes feature alignment, multiple scaling and evaluation, and accuracy verification.

[0013] In conjunction with the first aspect, in one implementation, the feature alignment operation is an operation that maps each of the low-dimensional feature vectors to the same dimensional space; The multi-group scaling and evaluation operation involves selecting feature vectors that meet custom conditions from a number of low-dimensional feature vectors. The custom conditions are a preset evaluation discrimination, a preset correlation between evaluation and device status, and a preset evaluation stability threshold for low-dimensional feature vectors. The accuracy verification operation is to calculate whether the accuracy retention rate of the low-dimensional feature vectors selected by the multiple scaling and evaluation operations meets a preset threshold.

[0014] In conjunction with the first aspect, in one implementation, the method further includes: If the device under monitoring is determined to be abnormal, the camera at the site of the device under monitoring is called to obtain real-time images of the device's operation. Based on the fault type identified in the analysis results, image features are extracted from the real-time operating images, and the structure of the device to be monitored is analyzed and processed based on these image features.

[0015] In conjunction with the first aspect, in one implementation, the real-time acquisition of multimodal data from the device under monitoring includes: The system acquires raw operating data of the monitored equipment in real time, including electrical parameters, mechanical parameters, and environmental parameters. The original operating data is optimized for data quality to obtain multimodal data. The data quality optimization includes filtering and normalization.

[0016] A second aspect of the present invention provides a fault equipment analysis apparatus based on multimodal data, the apparatus comprising: The monitoring module is used to collect multimodal data of the device under monitoring in real time. The multimodal data is obtained by collecting different data from different monitoring devices while the device under monitoring is running in real time, and then optimizing the quality of the data. The feature module is used to call a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, and to optimize and transform the low-dimensional feature vectors to obtain modal feature vectors. An anomaly module is used to determine a state deviation value based on the modal feature vector, and to determine whether the monitored device is abnormal based on the magnitude of the state deviation value, wherein the state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state; The analysis module is used to perform fault analysis processing based on the modal feature vector if the monitored device is determined to be abnormal, and to obtain the analysis results. The fault analysis processing is the process of determining the fault type and fault severity based on the abnormal time nodes of the modal feature vector.

[0017] In conjunction with the second aspect, in one implementation, the step of performing fault analysis processing based on the modal feature vector to obtain the analysis result includes: Anomaly time points are obtained from the modal feature vector based on a sliding window, where the anomaly time point is the time point when the state deviation value meets the threshold. Obtain the abnormal duration data corresponding to the abnormal time node, wherein the abnormal duration data is the abnormal data within a preset duration before and after the abnormal time node. The preset anomaly analysis model is invoked to determine the fault type and severity of the monitored equipment based on the anomaly duration data.

[0018] In conjunction with the second aspect, in one implementation, the step of calling a preset anomaly analysis model to determine the fault type and severity of the monitored device based on the anomaly duration data includes: The fault type is obtained by classifying the abnormal duration data using the loss function of a preset anomaly analysis model. Based on the key feature parameters corresponding to the fault type, abnormal parameter values ​​are extracted from the abnormal duration data, and the severity of the fault is determined by comparing the values ​​of the abnormal parameters with several preset fault thresholds.

[0019] In conjunction with the second aspect, in one implementation, the preset anomaly analysis model is a model trained using historical data prior to the anomaly time point; before training the preset anomaly analysis model, the method further includes: Historical modal data of the device to be monitored is acquired, and the historical modal data is integrated and verified to obtain historical training data; If the amount of historical training data meets the requirements, then the historical training data is used for model training. If the amount of historical training data is insufficient, the sample size of the historical training data is expanded by data augmentation and supplementary data collection methods, and the expanded historical training data is used for model training.

[0020] In conjunction with the second aspect, in one implementation, determining the state deviation value based on the modal feature vector and determining whether the monitored device is abnormal based on the magnitude of the state deviation value includes: The Euclidean distance and Mahalanobis distance between the modal feature vector and the preset state baseline are calculated respectively, wherein the preset state baseline is a vector generated based on the normal operation data of the device under health monitoring. The state deviation value is calculated using the Euclidean distance and the Mahalanobis distance; If the state deviation value is greater than the preset deviation threshold, the monitored device is determined to be abnormal. If the state deviation value is less than or equal to the preset deviation threshold, the device under monitoring is determined to be normal.

[0021] In conjunction with the second aspect, in one implementation, the step of invoking a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, and optimizing and transforming the low-dimensional feature vectors to obtain modal feature vectors, includes: A preset encoding architecture is invoked to extract low-dimensional feature vectors from the multimodal data, wherein the preset encoding architecture is obtained by setting the number of layers in the encoder and decoder and training them. The low-dimensional feature vectors are optimized and transformed to obtain modal feature vectors that conform to a custom scaling ratio. The optimization and transformation process includes feature alignment, multiple scaling and evaluation, and accuracy verification.

[0022] In conjunction with the second aspect, in one implementation, the feature alignment operation is an operation that maps each of the low-dimensional feature vectors to the same dimensional space; The multi-group scaling and evaluation operation involves selecting feature vectors that meet custom conditions from a number of low-dimensional feature vectors. The custom conditions are a preset evaluation discrimination, a preset correlation between evaluation and device status, and a preset evaluation stability threshold for low-dimensional feature vectors. The accuracy verification operation is to calculate whether the accuracy retention rate of the low-dimensional feature vectors selected by the multiple scaling and evaluation operations meets a preset threshold.

[0023] In conjunction with the second aspect, in one implementation, the apparatus further includes: The image module is used to, after the step of performing fault analysis processing based on the modal feature vector and obtaining the analysis results, if it is determined that the device to be monitored is abnormal, call the camera at the site of the device to be monitored to obtain the real-time operating image of the device to be monitored. The structure detection module is used to extract image features from the real-time running image based on the fault type of the analysis results, and to analyze and process the structure of the device to be monitored based on the image features.

[0024] In conjunction with the second aspect, in one implementation, the real-time acquisition of multimodal data from the device under monitoring includes: The system acquires raw operating data of the monitored equipment in real time, including electrical parameters, mechanical parameters, and environmental parameters. The original operating data is optimized for data quality to obtain multimodal data. The data quality optimization includes filtering and normalization.

[0025] Compared to existing technologies, the fault equipment analysis method and apparatus based on multimodal data provided in this invention have the following advantages: This invention can collect multimodal data of the monitored equipment in real time; extract low-dimensional feature vectors from the multimodal data using a preset encoding architecture, and optimize and transform the low-dimensional feature vectors to obtain modal feature vectors; determine the state deviation value based on the modal feature vectors, and determine whether the monitored equipment is abnormal based on the magnitude of the state deviation value; if the monitored equipment is determined to be abnormal, determine the fault type and severity based on the abnormal time nodes of the modal feature vectors. This invention can monitor the actual operating status of equipment in real time using multimodal data collected by various sensors, and when the equipment is abnormal, perform equipment analysis based on the data from the abnormal time nodes in the multimodal data to achieve secondary verification of the abnormal state. This eliminates the need for technicians to spend a lot of time and effort on secondary equipment testing, improving monitoring accuracy and simplifying monitoring steps to enhance processing efficiency and meet existing application needs. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a fault equipment analysis method based on multimodal data provided in an embodiment of the present invention; Figure 2 This is an operation flowchart of a fault equipment analysis method based on multimodal data provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a fault equipment analysis device based on multimodal data provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] With the continuous development of the power industry, the power system infrastructure is also increasing. The operating environment of these infrastructures and equipment is complex. Once any piece of equipment fails, it will affect the stable operation of the entire power system. Therefore, it is necessary to monitor its operating status in real time and effectively to avoid safety problems.

[0029] To monitor equipment for anomalies in real time, one common approach is to deploy a variety of sensor devices, including partial discharge sensors, temperature and humidity sensors, accelerometers, and detectors, to monitor the equipment status through each device. Then, machine learning techniques, such as support vector machines (SVM) and random forests (RF), are introduced to determine whether the equipment is abnormal based on any of the monitoring data.

[0030] However, the above method has the following technical problems: Although the above method can determine whether the equipment is abnormal based on any data, the equipment is easily affected by the various connected devices and fluctuates during operation. Therefore, after the equipment is determined to be abnormal, technicians often need to spend a lot of time and energy to conduct a secondary test on the equipment in order to verify the actual status of the abnormal equipment and carry out subsequent maintenance. Therefore, the above method is not only difficult to determine the actual operating status of the equipment, but also has cumbersome steps, low processing efficiency, and is difficult to meet the current application needs.

[0031] To address the aforementioned issues, the following specific embodiments will provide a detailed description and explanation of a fault equipment analysis method, apparatus, device, and medium based on multimodal data provided in this application.

[0032] To address the technical problems of existing technologies being unable to determine the actual operating status of equipment and having low processing efficiency, referencing Figure 1 The diagram shows a flowchart of a fault equipment analysis method based on multimodal data provided in an embodiment of the present invention.

[0033] In one embodiment, the fault equipment analysis method based on multimodal data is applicable to the monitoring or management system of a power system.

[0034] As an example, the fault equipment analysis method based on multimodal data may include: S11. Real-time acquisition of multimodal data of the device under monitoring, wherein the multimodal data is obtained by acquiring different data using different monitoring devices while the device under monitoring is running in real time, and then optimizing the quality of the data.

[0035] In one embodiment, multiple sensor devices can be deployed on the device to be monitored, including partial discharge sensors, temperature and humidity sensors, accelerometers, and detectors, etc., to collect different data from the device in real time. The data from each sensor device is then optimized to obtain multimodal data of the device under monitoring. Comprehensive monitoring using multiple different data sources determines the real-time operating status of the device, facilitating fault analysis and identifying the cause and severity of any anomalies.

[0036] As an example, the real-time acquisition of multimodal data from the device under monitoring may include the following sub-steps: S111. Acquire the raw operating data of the device to be monitored in real time. The raw operating data includes electrical parameters, mechanical parameters, and environmental parameters.

[0037] S112. Perform data quality optimization on the original running data to obtain multimodal data. The data quality optimization includes filtering and normalization.

[0038] In one embodiment, this method is applicable to any complex equipment with multimodal operating data that requires condition monitoring and fault diagnosis. The equipment to be monitored includes, but is not limited to, industrial equipment (such as motors, pumps, compressors, etc.), energy and power equipment (such as transformers, generators, distribution cabinets, etc.), manufacturing and automation equipment (such as CNC machine tools, etc.), and infrastructure and heavy equipment (such as wind turbine generators, construction machinery (excavators, cranes, etc.).

[0039] In one embodiment, multiple sensors are deployed on the device to acquire raw operating data of the device body and the surrounding environment. The raw operating data includes, but is not limited to, vibration data, temperature data, pressure data, and current data. Data consistency is ensured by synchronizing with timestamps to form a comprehensive multimodal dataset. The data acquisition target and location are clearly defined, and multimodal data during the device operation process is collected in real time by placing corresponding sensors on different parts of the device body and the surrounding environment. In one operating mode, the sampling rate of the acquisition device can be configured for each sensor according to the operating frequency and fault characteristics of the equipment; and a unified timestamp synchronization mechanism can be configured for all sensors to ensure the consistency of data of different modes in time. Specifically, the time reference of all sensors is aligned through a synchronization signal; a precise timestamp is added to each data point, and the time difference of data of different modes is calibrated through an algorithm.

[0040] The beneficial effects of the above design scheme are: configuring the sampling rate for each sensor according to the equipment operating frequency and fault characteristics, ensuring the targeted and efficient data acquisition, timely capturing minute changes and potential fault characteristics in equipment operation, and effectively avoiding misjudgments caused by time deviation through a unified timestamp synchronization mechanism, thus providing a reliable guarantee for subsequent data fusion and analysis.

[0041] The acquired raw operational data undergoes preprocessing, including filtering and normalization. Filtering removes noise and interference while preserving key features and effective information of the original data to the greatest extent possible, ensuring the authenticity and integrity of the data. Normalization normalizes the amplitudes of vibration, temperature, pressure, and current data to the range of [0,1] or [-1,1] to eliminate the influence of different dimensions and ranges, giving the data a uniform scale and facilitating subsequent feature extraction.

[0042] S12. Invoke the preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, and optimize and transform the low-dimensional feature vectors to obtain modal feature vectors.

[0043] In one embodiment, multimodal data can be input into a preset encoding architecture. The preset encoding architecture extracts corresponding feature data from the multimodal data and converts it into vectors to obtain low-dimensional feature vectors. The low-dimensional feature vectors are then optimized and transformed to obtain modal feature vectors.

[0044] Modal feature vectors are used to represent the health status of a device at different points in time. For real-time monitored multimodal data, features are extracted from the encoder part of the autoencoder to obtain the modal feature vectors representing the real-time health status.

[0045] In an optional embodiment, the step of invoking a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data and optimizing and transforming the low-dimensional feature vectors to obtain modal feature vectors may include the following sub-steps: S121. Invoke the preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, wherein the preset encoding architecture is obtained by setting the number of layers of the encoder and decoder and training them.

[0046] S122. The low-dimensional feature vector is optimized and transformed to obtain modal feature vectors that meet the custom scaling ratio. The optimization and transformation process includes feature alignment operation, multiple scaling and evaluation, and accuracy verification.

[0047] Based on the complexity of the data and the requirements for feature extraction, select an appropriate autoencoder architecture. For multimodal data, use a multi-layer autoencoder to gradually extract deeper features. Determine the number of layers in the encoder and decoder of the autoencoder, the number of neurons in each layer, and the activation function, and set the hyperparameters such as learning rate, batch size, and number of training epochs. In the trained autoencoder, the preprocessed multimodal data is input into the encoder part to obtain a low-dimensional feature vector, which is used to represent the health status of the device at different time points.

[0048] Specifically, a suitable autoencoder architecture can be selected based on the complexity of the data and the requirements for feature extraction. For multimodal data, a multi-layer autoencoder can be used to progressively extract deeper features. The number of layers in the encoder and decoder, the number of neurons in each layer, and the activation function can be determined, and hyperparameters such as learning rate, batch size, and number of training epochs can be set to obtain the preset encoding architecture.

[0049] In the trained autoencoder, the preprocessed multimodal data is input into the encoder part to obtain a low-dimensional feature vector, which is used to represent the health status of the device at different time points.

[0050] In one embodiment, the focus is on monitoring the condition of a wind turbine generator set. A wind turbine generator set is a typical complex device, and the health status of its key components (such as the main shaft, gearbox, and generator) can be monitored using various sensors, for example: Vibration sensors: installed on gearboxes and generators to monitor mechanical wear, imbalance, misalignment and other faults. The data is usually high-frequency time-series signals and is very complex. Temperature sensor: Monitors gearbox oil temperature, bearing temperature, etc., and is used as an indirect indicator of overload or poor lubrication; Current / voltage sensor: Monitors the output power quality of the generator and can reflect electrical faults or load changes.

[0051] Considering the complexity of the data and the requirements for feature extraction, a stacked denoising autoencoder was selected. The architecture of the stacked denoising autoencoder includes: Stacking (depth): Employing multi-layer (depth) encoders, such as a 4-layer encoder. Input layer: Receives raw or preprocessed multimodal data (e.g., normalized vibration spectrum, temperature, current value); Hidden layer 1: Learns local features (such as specific frequency components) in vibration data; Hidden layer 2: Learn more abstract combinations of features (such as the set of feature frequencies corresponding to a certain wear pattern); Hidden layer 3: Forms a highly compressed bottleneck layer that represents the health status of the equipment.

[0052] Denoising: During training, random noise (such as Gaussian noise) is added to the input data (especially vibration data), requiring the autoencoder to learn and reconstruct clean original data.

[0053] Training phase: Input the multimodal data (including vibration data with noise, temperature, and current) collected during normal operation of the wind turbine into the stacked denoising autoencoder; Stacked denoising autoencoders learn a low-dimensional feature space that can effectively reconstruct normal data through multi-layer nonlinear transformations; in the feature space, data points in the normal state will cluster in a specific region to form an initial healthy state baseline.

[0054] Real-time monitoring phase: Real-time multimodal data is input into the encoder part of the pre-trained stacked denoising autoencoder. The encoder outputs a low-dimensional feature vector, which is used to represent the real-time health status of the device.

[0055] The beneficial effects of the above design scheme are: by gradually extracting deep features from multimodal data through an autoencoder, the data dimensionality can be effectively reduced while retaining key information. The extracted low-dimensional feature vectors can accurately reflect the health status of the equipment at different points in time, thereby constructing an initial health status baseline. Furthermore, the real-time monitored multimodal data can be quickly extracted through the autoencoder to generate real-time health status vectors, which can promptly detect equipment anomalies and provide strong support for early fault warning and preventive maintenance.

[0056] After obtaining the low-dimensional feature vectors, they can be optimized and transformed to select modal feature vectors that meet a custom scaling ratio. This optimization and transformation process includes feature alignment, multiple scaling and evaluation operations, and accuracy verification. Since there are multiple datasets, there are multiple low-dimensional feature vectors, each corresponding to a modality. Therefore, feature alignment, multiple scaling and evaluation operations, and accuracy verification can be performed on these multiple low-dimensional feature vectors to select the vectors that meet the requirements as the modal feature vectors.

[0057] In one embodiment, the feature alignment operation is an operation that maps each of the low-dimensional feature vectors to the same dimensional space.

[0058] The multi-group scaling and evaluation operation involves selecting feature vectors that meet custom conditions from a number of low-dimensional feature vectors. The custom conditions are a preset evaluation discrimination, a preset correlation between evaluation and device status, and a preset evaluation stability threshold for low-dimensional feature vectors.

[0059] The accuracy verification operation is to calculate whether the accuracy retention rate of the low-dimensional feature vectors selected by the multiple scaling and evaluation operations meets a preset threshold.

[0060] In one operation mode, the feature alignment operation may include: counting the dimensions of each modality feature and analyzing the differences between the dimensions of different modality features; and selecting an appropriate alignment method based on the dimensional differences of each feature vector to map the feature vectors of different modalities to the same dimensional space.

[0061] Specifically, the feature alignment operation is as follows: for modalities with small dimensional differences, a simple truncation or zero-padding method is used; for modalities with large dimensional differences, a feature transformation method based on deep learning is used; according to the selected alignment method, the feature vectors are aligned to make the dimensions of each feature vector consistent.

[0062] The beneficial effects of the above design scheme are: by statistically analyzing the dimensions of each modal feature and the differences, and by selecting an alignment method accordingly, the problem of inconsistent dimensions of different feature vectors is effectively solved. After ensuring that the dimensions of each feature vector are consistent, the effective fusion of different modal data in the same dimensional space can be achieved.

[0063] In an optional implementation, the multiple scaling and evaluation operations may be: obtaining the dimensional environment corresponding to each feature vector, optimizing the dimensional environment based on dimensional consistency, determining the environmental feature difference between the initial environmental features and the optimized environmental features, and determining the dimensional scaling ratio range for each feature vector based on the environmental feature difference. Optionally, multiple scaling ratios can be customized for each feature vector based on the dimension scaling ratio, and the feature vector of the new modality under each scaling ratio can be obtained. The feature vector of the new modality is evaluated based on the discriminability, the correlation with the device state, and the stability, and a comprehensive evaluation value of the feature vector of each new modality is obtained. The new feature vector with the largest evaluation value is selected as the evaluated feature vector.

[0064] In an optional implementation, the evaluation of multiple scaling ratios can calculate the evaluation discrimination, the correlation between the evaluation and the device status, and the evaluation stability, respectively. Then, based on the three dimensions of indicators, the vectors that meet the requirements are selected from the evaluated modal feature vectors to obtain the evaluated feature vectors.

[0065] Specifically, the ability to distinguish between different states can be evaluated by calculating the ratio of inter-class distance to intra-class distance of the new modal feature vector under different device state categories (such as normal, fault A, fault B), using Fisher's criterion or calculating the ratio of inter-class divergence to intra-class divergence. The inter-class distance uses the Euclidean distance of the mean values ​​of different class features, and the intra-class distance uses the standard deviation or average variance of each class feature, thereby quantifying the ability of the feature to distinguish between different states. The correlation between features and equipment status can be assessed by calculating the correlation coefficient between the new modality feature vector and the equipment status label (such as the Pearson correlation coefficient for continuous states) or by using mutual information methods to measure the statistical dependence between features and status. The higher the correlation, the stronger the feature's ability to represent the equipment status. Stability can be assessed by collecting feature data over multiple time periods under the same equipment conditions and similar operating conditions, calculating the coefficient of variation or standard deviation between these feature vectors, and examining the consistency of their feature values ​​at different sampling times. If the features change little over different time periods, the stability is high. After completing the evaluation of the above three dimensions, each evaluation indicator is normalized and weights are assigned to each indicator (such as a discrimination weight of 0.4, a correlation weight of 0.4, and a stability weight of 0.2). The comprehensive evaluation value of each new feature vector is obtained by weighted summation, and finally the feature vector with the largest comprehensive evaluation value is selected as the evaluated feature vector. Next, accuracy verification can be performed. Specifically, the accuracy verification operation may involve: obtaining the pre-alignment distribution features of the evaluated feature vectors and the post-alignment distribution features of the initial aligned modal feature vectors; calculating the accuracy retention rate of the post-alignment distribution features based on the pre-alignment distribution features; and determining whether the accuracy retention rate meets a preset threshold. The accuracy retention rate is calculated as follows: ; Where ARR represents the calculated accuracy retention rate; F before F represents the target modal feature vector before alignment, indicating the original feature distribution; aligned The initial aligned modal feature vector represents the feature distribution after alignment; Sim(X,Y) is a similarity function used to measure the similarity between two feature distributions. If the accuracy retention rate meets the preset threshold, the initial aligned modal feature vector is used as the final modal feature vector; If the accuracy retention rate does not meet the preset threshold, the initial aligned modal feature vector based on the pre-alignment distribution features is optimized until the accuracy retention rate meets the preset threshold, resulting in the final modal feature vector, specifically: Based on the distribution features before alignment, key statistical attributes are extracted, including the distribution of mean vector, covariance matrix and eigenvalues. A distribution consistency loss function is constructed, which is composed of the KL divergence of the feature distribution before and after alignment, the Euclidean distance of the difference in eigenvalues ​​and the cosine value of the angle between the principal components. The gradient descent method is used to iteratively adjust the initial alignment modal feature vector. In each iteration, the distribution feature of the current alignment vector is calculated and compared with the distribution before alignment. The alignment vector is updated through backpropagation to minimize the distribution consistency loss. If the accuracy retention rate still does not meet the preset threshold after multiple iterations, an adversarial training mechanism is introduced. A pre-trained discriminator network is used to distinguish the difference in feature distribution before and after alignment. The adversarial optimization between the alignment vector generator and the discriminator further narrows the distribution before and after alignment. Optimization stops when the accuracy retention rate meets the target or the maximum number of iterations is reached, and the optimized final modal feature vector is output.

[0066] In this embodiment, the criteria for evaluating discrimination include inter-class / intra-class distance and statistical significance test.

[0067] In this embodiment, the new modal feature vector needs to remain stable under the same equipment state and similar operating conditions, so the stability index is introduced for evaluation.

[0068] In this embodiment, high-quality features need to be strongly correlated with the device status, so the correlation with device status is introduced as an indicator for evaluation.

[0069] In this embodiment, the dimensional environment includes the high-frequency characteristics of vibration data, the temporal continuity of temperature data, etc.

[0070] The beneficial effects of the above design scheme are: Instead of directly adopting fixed scaling rules, the scaling ratio range is determined through dimensional environment optimization. This step considers the original dimensional environment of different modal feature vectors, making dimensional scaling more closely match the physical meaning of each modal data and the correlation characteristics of device status. This avoids feature distortion caused by fixed scaling, laying a more reasonable foundation for subsequent alignment. By evaluating and selecting target modal feature vectors through multi-dimensional assessments of discriminability, correlation with device status, and stability, it ensures that the features have strong discriminative ability, such as effectively distinguishing between normal and faulty device states and different fault types, strengthening the correlation between features and the actual operating status of the device. Improved feature stability results in higher quality target feature vectors. By comparing the distribution features before and after alignment and verifying the accuracy retention rate, the focus is on whether the aligned features retain the core distribution patterns and key information of the original features. When the accuracy retention rate is not up to standard, optimization ensures that the final aligned features still fully reflect the information content of the original features. This solves the problem of sacrificing key feature information in traditional alignment methods in pursuit of dimensional consistency, ensuring that the extracted features still have high information content and guaranteeing the quality of the final modal feature vectors.

[0071] S13. Determine the state deviation value based on the modal feature vector, and determine whether the monitored device is abnormal according to the magnitude of the state deviation value, wherein the state deviation value is the deviation value between the modal feature vector and the state baseline of the monitored device in the monitoring state.

[0072] Next, the state deviation value can be calculated based on the modal feature vector, and the magnitude of the state deviation value can be used to determine whether the monitored device is abnormal. The state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state.

[0073] Specifically, the modal feature vector can be calculated to calculate the state deviation value, and the magnitude of the state deviation value can be used to determine whether the monitored device is abnormal. The state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state.

[0074] The real-time modal feature vector is compared with the initial health state baseline to determine the deviation between the two. If the deviation is large, it indicates that the real-time state of the device deviates significantly from the initial health state, and the device may be malfunctioning.

[0075] In one embodiment, determining the state deviation value based on the modal feature vector and determining whether the monitored device is abnormal based on the magnitude of the state deviation value may include the following sub-steps: S131. Calculate the Euclidean distance and Mahalanobis distance between the modal feature vector and the preset state baseline, respectively, wherein the preset state baseline is a vector generated based on the normal operation data of the device under health monitoring.

[0076] S132. Calculate the state deviation value using the Euclidean distance and the Mahalanobis distance.

[0077] S133. If the state deviation value is greater than the preset deviation threshold, the device to be monitored is determined to be abnormal.

[0078] S134. If the state deviation value is less than or equal to the preset deviation threshold, then the device to be monitored is determined to be normal.

[0079] In one operating mode, the Euclidean distance between the modal feature vector and the preset state baseline can be calculated, as well as the Mahalanobis distance between the modal feature vector and the preset state baseline can be calculated.

[0080] The preset state baseline involves collecting multimodal data from the device under monitoring in a normal state, extracting low-dimensional feature vectors from the multimodal data in the normal state according to the above steps using a preset encoding architecture, and optimizing and transforming the low-dimensional feature vectors to obtain normal state feature vectors, which are then used as the preset state baseline.

[0081] Specifically, the Euclidean distance D e Mahalanobis distance D is used to measure the absolute difference between vectors and reflects the overall deviation. f Considering the covariance structure between features can better identify anomalous patterns. It should be noted that conventional techniques in this field can be used to calculate the Euclidean distance D between two vectors. e And Mahalanobis distance D f .

[0082] Next, we can consider the Euclidean distance D. e And Mahalanobis distance D fWeighted fusion is performed to obtain the state deviation value D. Specifically, the state deviation value D is calculated as follows: ; Where α and β are weighting coefficients, and α+β=1, and α and β∈[0,1]; The status deviation value D is compared with the preset deviation threshold T. If the status deviation value is less than or equal to the preset deviation threshold, i.e., D≤T, the equipment is determined to be in normal operating condition. If the status deviation value is greater than the preset deviation threshold, that is, D is greater than T, the device is determined to be abnormal; In this embodiment, the state deviation value D is compared with a preset deviation threshold T. If it exceeds the threshold, it is determined to be abnormal. The preset deviation threshold T can be set according to the historical data of equipment operation or expert experience, or it can be dynamically updated based on statistical methods (such as percentiles).

[0083] In this embodiment, α and β are adjustable weights, which can be initially set to α=0.5 and β=0.5. The weights can be dynamically adjusted according to the signal-to-noise ratio, feature stability, or historical diagnostic accuracy of each modality data. For example, if the noise of a certain modality data is large, the weight of its corresponding distance will be reduced.

[0084] The beneficial effects of the above design scheme are as follows: By comprehensively using Euclidean distance and Mahalanobis distance for weighted fusion to construct the state deviation value, the Euclidean distance can effectively capture the overall absolute deviation between the real-time state and the baseline, while the Mahalanobis distance enhances the sensitivity to the identification of abnormal patterns by introducing the covariance structure between features. This dual distance measurement mechanism allows the state comparison to not only focus on global changes but also keenly capture specific anomalies, thereby enhancing the comprehensiveness and accuracy of the judgment of equipment state anomalies. At the same time, the adjustable weight coefficients α and β enable flexible configuration of the contribution of different distance measurements, which can be adaptively optimized according to data characteristics (such as signal-to-noise ratio, feature stability) or historical performance, further improving the robustness and accuracy of the diagnostic system. In addition, comparing the state deviation value with a threshold that can be set and dynamically updated based on historical data or expert experience makes the judgment logic clear and reliable, effectively distinguishing between normal and abnormal equipment states, thus providing timely and reliable decision-making basis for early fault detection and preventive maintenance.

[0085] S14. If the monitored device is determined to be abnormal, fault analysis processing is performed based on the modal feature vector to obtain the analysis results. The fault analysis processing is the process of determining the fault type and severity based on the abnormal time nodes of the modal feature vector.

[0086] After determining that there is an anomaly in the equipment, abnormal data can be extracted from the modal feature vector and input into the pre-built anomaly analysis model. The anomaly analysis model can then determine the type and severity of the fault and output the diagnostic results.

[0087] The anomaly analysis model is trained by collecting historical data from the equipment as training samples. Before training, the total number of samples collected is assessed to determine whether it is sufficient to support the effective training of the model, based on the data requirements of the anomaly analysis model and the probability factors of equipment failure.

[0088] In one embodiment, the fault analysis processing based on the modal feature vector to obtain the analysis result may include the following sub-steps: S141. Obtain abnormal time nodes from the modal feature vector based on a sliding window. The abnormal time nodes are the time nodes when the state deviation value meets the threshold.

[0089] S142. Obtain the abnormal duration data corresponding to the abnormal time node. The abnormal duration data is the abnormal data within a preset duration before and after the abnormal time node.

[0090] S143. Call the preset anomaly analysis model to determine the fault type and severity of the monitored equipment based on the anomaly duration data.

[0091] In one embodiment, since the modal feature vector is acquired in real time, a method based on comparing a sliding window with a dynamic threshold is used: continuously calculating the state deviation value between the modal feature vector and the initial health state baseline, and setting a dynamic threshold; when the state deviation value first exceeds the preset deviation threshold, the monitored device is determined to be abnormal, and this time point is recorded as the initial abnormal time point; Starting from the time point identified as abnormal, multimodal data for that time point and a period before and after it are extracted as abnormal duration data. Specifically, the data for a period before the time point (e.g., the first 5 seconds) is reviewed to confirm whether there is a continuous deviation trend; and combined with the data for the next period (e.g., the last 5 seconds), the start and duration of the abnormality are comprehensively judged. Finally, the midpoint of the time period in which the state deviation value continuously exceeds the threshold and has a significant deviation trend is determined as the abnormal time point, and the abnormal duration data corresponding to the abnormal time point is obtained.

[0092] Next, the abnormal duration data can be input into the preset anomaly analysis model so that the preset anomaly analysis model can determine the fault type and severity of the monitored equipment based on the abnormal duration data.

[0093] Specifically, an anomaly analysis model can be pre-built and its parameters initialized. Historical data from the equipment can be collected and checked to ensure it meets the requirements for sample data. The checked historical data is then used as sample data to train the anomaly analysis model. This sample data includes samples with different types of faults and samples with different fault severity levels, and the samples are labeled. The anomaly analysis model is trained using the sample data, and its parameters are updated using the backpropagation algorithm. The model structure and parameters are adjusted to minimize prediction error and loss function, and hyperparameter tuning is performed.

[0094] Optionally, the preset anomaly analysis model is a model trained using historical data prior to the anomaly time point; before training, it can also be checked whether the historical data meets the requirements for being used as sample data.

[0095] As an example, before training the preset anomaly analysis model, the method may further include the following sub-steps: S21. Obtain historical modal data of the device to be monitored, and integrate and verify the historical modal data to obtain historical training data.

[0096] S22. If the amount of historical training data meets the requirements, then the historical training data is used for model training.

[0097] S23. If the amount of historical training data does not meet the requirements, the sample size of the historical training data is expanded by data augmentation and supplementary data collection methods, and the expanded historical training data is used for model training.

[0098] Specifically, historical data of different modes can be collected from the equipment's operation history. The collected historical data of various modes can be integrated to ensure consistency and correspondence in timestamps and equipment identification information. The time range of the collected historical data can be verified to ensure that it covers different failure modes of the equipment. Data from the most recent year should be selected first, and 3-6 days of data should be selected each month, with 1-2 days of data from the beginning, middle and end of each month.

[0099] Meanwhile, the overall sample size collected is assessed to determine whether it is sufficient to support effective training of the model, based on the data requirements of the anomaly analysis model and the probability factors of equipment failure. If the sample size is insufficient, it is expanded by data augmentation and supplementary data collection.

[0100] The beneficial effects of the above design scheme are: the above process provides a clear and standardized standard for checking whether historical data meets the sample data requirements, which can efficiently evaluate and screen the collected historical data and reduce the time spent on repeated processing and debugging due to data problems.

[0101] In an optional embodiment, the overall sample size collected can be assessed as sufficient to support effective training of the model, based on the data requirements of the anomaly analysis model and the probability of equipment failure.

[0102] Specifically, the number of model parameters of the anomaly analysis model can be obtained, and based on the model type of the anomaly analysis model, the first coefficient and the second coefficient can be determined, and the number of multi-dimensional features can be determined. Based on the number of model parameters, the first coefficient, the second coefficient, and the multidimensionality of multimodal features, the amount of data required for the anomaly analysis model is calculated. Specifically, the required amount of data is calculated as follows: ; Where M represents the amount of data required by the anomaly analysis model. Indicates the first coefficient. H represents the second coefficient, H represents the number of model parameters in the anomaly analysis model, and D represents the number of dimensions of the multimodal features. Based on the probability of each equipment failure type occurring and the proportion of the current equipment failure type in the total sample size, the fit between the current equipment failure type and the total sample size is calculated. Specifically, the fit is calculated as follows: ; Where K represents the fit between the current equipment fault type and the overall sample size, and C represents a constant. This represents the proportion of the current equipment failure type in the total sample size; N represents the total sample size related to the current equipment failure type, ensuring both normal and failure data are included; n represents the failure data size for the current equipment failure type; and m represents the total number of all equipment failure types. This represents the ratio of the total number of faults in the overall sample to the total sample size. When the total sample size meets the data requirements of the anomaly analysis model, and the fit between all equipment fault types and the total sample size is greater than the preset fit, the total sample size is determined to be sufficient to support the effective training of the model; otherwise, the total sample size is determined to be insufficient to support the effective training of the model.

[0103] In this embodiment, the first and second coefficients are adjusted according to the model type, such as in a deep learning model. =10-100, =500-1000; In traditional machine learning models =1-10, =100-500.

[0104] In this embodiment, The larger the difference, the more unreasonable the distribution of the current equipment failure type in the overall sample size, and the smaller the fit.

[0105] The beneficial effects of the above design scheme are as follows: The scheme dynamically calculates the amount of data required by the model through three dimensions: model parameter quantity, multimodal feature dimension, and coefficient adjustment. Compared with a fixed sample size threshold, this is more scientific and more in line with the model complexity. It evaluates the degree of matching between the sample size of each fault type and the actual probability of occurrence, and focuses on solving the common problem of too many normal samples and scarce fault samples in industrial scenarios. By incorporating the multimodal feature dimension into the data quantity calculation, it ensures that the contribution of each modality of data to model training is balanced. By balancing the model requirements and fault probabilities, the model trained based on the final samples has stronger generalization ability. This scheme constructs a complete sample size evaluation system, solves the data bottleneck problem of large multimodal models in equipment diagnosis, and improves model performance and diagnostic reliability from the source.

[0106] In one embodiment, the step of invoking a preset anomaly analysis model to determine the fault type and severity of the monitored device based on the anomaly duration data may include the following sub-steps: S1431. Using the loss function of the preset anomaly analysis model, classify the anomaly duration data to obtain the fault type.

[0107] S1432. Extract abnormal parameter values ​​from the abnormal duration data based on the key feature parameters corresponding to the fault type, and determine the severity of the fault based on the comparison results of the abnormal parameter values ​​with several preset fault thresholds.

[0108] In the anomaly analysis model, the loss function of the preset anomaly analysis model is used to classify the anomaly duration data to obtain the fault type.

[0109] Specifically, the cross-entropy loss function is used for fault type classification, and its formula is shown below: ; In the formula, N is the number of samples; C is the number of fault categories; Let i be the true label of the i-th sample; Let be the predicted probability of the model that the i-th sample belongs to category C; The model parameters θ are then iteratively updated using gradient descent, as shown in the following equation: ; In the formula, The learning rate; This represents the gradient of the loss function with respect to the parameters. The hyperparameter tuning process is as follows: Define the hyperparameter search space, including learning rate (e.g., 0.001 to 0.1), batch size (e.g., 32, 64, 128), regularization coefficient λ (e.g., 0.0001 to 0.01), number of hidden layer nodes, etc. Select optimization method: Use grid search or Bayesian optimization method to systematically select hyperparameter combinations within the search space; Set evaluation metrics: Use macro F1 score or weighted accuracy on the validation set as the evaluation standard for model performance; Perform cross-validation: For each set of hyperparameters, perform k-fold cross-validation (e.g., k=5) to ensure the robustness of the evaluation results; Training and Evaluation: For each set of hyperparameters, train the model, record its performance on the validation set, and select the best-performing combination of hyperparameters. Final determination and validation: The model was retrained on the full training set using the optimal hyperparameters, and its generalization performance was validated on an independent test set; The extracted abnormal data is input into a trained anomaly analysis model. The model outputs the fault type, including but not limited to mechanical faults (such as bearing damage, gear wear or broken teeth), electrical faults (such as insulation aging, short circuits or open circuits), and thermal faults (such as overheating, abnormal temperature distribution). Combining the information from the fault type and characteristics, the severity of the fault is assessed, quantifying the severity into minor, moderate, and severe faults. Determine the corresponding key characteristic parameters based on the fault type, including vibration amplitude, temperature deviation, current harmonic content, and pressure fluctuation range; Extract the values ​​of key feature parameters from the abnormal data and compare them with preset fault thresholds at each level: If the parameter value is in the first range (e.g., exceeding the normal value but within the mild threshold), it is judged as a minor fault; If the parameter value is in the second range (e.g., exceeding the mild threshold but not reaching the severe threshold), it is judged as a moderate fault; If the parameter value is in the third interval (such as exceeding the severe threshold or showing a sharp change), it is judged as a severe fault; The output results are presented to the user through a graphical interface and text reports, including the current status of the device, the type of fault, and the severity of the fault.

[0110] The beneficial effects of the above design scheme are as follows: by constructing an anomaly analysis model based on machine learning and training it with historical data containing different types and severity of faults, it is possible to accurately identify the fault types of the equipment and quantitatively assess the faults (minor, moderate, severe), which significantly improves the accuracy and reliability of fault diagnosis. Furthermore, through accurate fault diagnosis and severity assessment, it provides a scientific basis for preventive maintenance of the equipment, thereby helping users to plan maintenance plans in advance.

[0111] When an anomaly is detected, further verification of the equipment is conducted to determine whether there is physical damage or structural problems.

[0112] As an example, after the step of performing fault analysis processing based on the modal feature vector to obtain the analysis result, the method may further include the following sub-steps: S15. If it is determined that the device to be monitored is abnormal, call the camera at the site of the device to be monitored to obtain real-time operating images of the device.

[0113] S16. Extract image features from the real-time running image according to the fault type of the analysis results, and analyze and process the structure of the device to be monitored according to the image features.

[0114] In one embodiment, when an anomaly is detected, the camera is triggered to retrieve image information from the equipment site to obtain real-time operating images. The image data is combined to enhance fault feature extraction, and the presence of physical damage or structural problems is confirmed by analyzing the equipment's response behavior characteristics.

[0115] Specifically, when an abnormality is detected in the equipment, the image acquisition function of the camera is immediately triggered, and the camera acquires on-site images according to the preset parameters; Based on the equipment type and fault characteristics, an appropriate image feature extraction method is selected to extract the image features on site. Specifically, based on prior knowledge of equipment type and typical fault modes, an equipment type-fault feature-image feature mapping table is established; based on the fault features detected in real time, the possible fault types are initially judged, and the image feature categories to be extracted are determined according to the mapping table. Evaluate the applicability of different image feature extraction methods to target features: for structural problems, edge detection or contour extraction algorithms should be preferred; For surface damage, texture analysis or local binary mode should be selected; For thermal faults, temperature distribution features are extracted using infrared images. Taking into account image quality, environmental interference, and real-time requirements, a feature extraction method that balances robustness and efficiency was selected from the candidate methods. Based on the operating principle and failure mode of the equipment, the response behavior characteristics of the equipment are defined, and then the response behavior characteristics of the equipment are extracted from multimodal data; By combining image features, multimodal data features, and device response behavior features, a comprehensive analysis is conducted to determine whether there is physical damage or structural problems.

[0116] The technical effects of the above solution are as follows: when an equipment abnormality is detected, the camera is immediately triggered to capture on-site images. By combining image data, multimodal data features, and equipment response behavior features for comprehensive analysis, the equipment status can be comprehensively evaluated from multiple perspectives. This multi-dimensional diagnostic method not only improves the accuracy of fault detection but also more accurately locates the type of fault, especially physical damage or structural problems. It can effectively prevent further deterioration of equipment faults, reduce the risk of safety accidents caused by equipment faults, and ensure the safety and stability of equipment operation.

[0117] Working principle: By collecting multimodal data from the device itself and the environment through multiple sensors and synchronizing the data with timestamps, the comprehensiveness and consistency of the data can be ensured, thus providing a reliable data foundation for accurate diagnosis. By using an autoencoder to extract features and build an initial health status baseline, the normal and abnormal states of the device can be efficiently identified. At the same time, through feature alignment operations, the fusion effect of multimodal data is further improved. In the device status diagnosis stage, by comparing the real-time health status vector with the baseline, it is possible to quickly determine whether the device is abnormal. Combined with the anomaly analysis model, the fault type and severity can be accurately determined, providing a scientific basis for the preventive maintenance of the device. In addition, when an anomaly is detected, camera image information is retrieved and combined with the device's response behavior characteristics, which further enhances the fault feature extraction capability and can effectively confirm physical damage or structural problems.

[0118] Reference Figure 2 The diagram shows an operation flowchart of a fault equipment analysis method based on multimodal data provided by an embodiment of the present invention.

[0119] Specifically, the operation flow of the fault equipment analysis method based on multimodal data includes the following steps: The first step is to acquire multimodal data of the device itself and its surrounding environment by deploying multiple sensors on the device, including but not limited to vibration data, temperature data, pressure data and current data, and to ensure data consistency by synchronizing with timestamps, thus forming a comprehensive multimodal dataset.

[0120] The second step is to preprocess the acquired multimodal data, including filtering and normalization.

[0121] The third step involves using an autoencoder algorithm to extract features from multimodal data, constructing an initial health status baseline, and generating a real-time health status vector. Specifically, based on the features of the extracted multimodal data, an alignment method is selected to align the feature vectors of each modality, ensuring that the dimensions of each feature vector are consistent.

[0122] The fourth step involves comparing the real-time health status vector with the initial health status baseline through real-time health status comparison and fault identification, calculating the difference between the two, and determining that the device is abnormal if the difference exceeds a preset threshold.

[0123] The fifth step involves extracting the abnormal data from the equipment after determining that an anomaly exists, and inputting it into a pre-built anomaly analysis model. The anomaly analysis model then determines the type and severity of the equipment malfunction and outputs the diagnostic results. The anomaly analysis model is trained using historical data collected from the equipment as training samples. Before training, the total sample size collected is assessed to determine whether it is sufficient to support effective training of the model, based on the data requirements of the anomaly analysis model and the probability of equipment malfunctions.

[0124] The sixth step is to trigger the camera to retrieve on-site image information when an anomaly is detected, combine the image data to enhance the extraction of fault features, and confirm whether there is physical damage or structural problems by analyzing the response behavior characteristics of the equipment.

[0125] This invention collects multimodal data from the equipment itself and its surrounding environment using multiple sensors and synchronizes the data with timestamps. This comprehensively and accurately reflects the actual operating status of the equipment, providing a rich and consistent data foundation for subsequent diagnosis. Secondly, it uses an autoencoder algorithm to extract features from the multimodal data, constructs an initial health status baseline, and generates a real-time health status vector. This enables real-time health status comparison and fault identification, allowing for rapid detection of equipment anomalies. Furthermore, the abnormal data allows for precise determination of the fault type and severity, providing a strong basis for equipment maintenance. In addition, by combining this with on-site image information retrieved from cameras, the fault feature extraction capability is further enhanced, helping to confirm physical damage or structural problems and improving the comprehensiveness and accuracy of diagnosis.

[0126] In this embodiment, the present invention provides a fault equipment analysis method based on multimodal data. Its advantages are as follows: the present invention can collect multimodal data of the device under monitoring in real time; extract low-dimensional feature vectors from the multimodal data using a preset encoding architecture, and optimize and transform the low-dimensional feature vectors to obtain modal feature vectors; determine the state deviation value based on the modal feature vectors, and determine whether the device under monitoring is abnormal based on the magnitude of the state deviation value; if the device under monitoring is determined to be abnormal, determine the fault type and severity based on the abnormal time nodes of the modal feature vectors. The present invention can monitor the actual operating status of the equipment in real time using multimodal data collected by various sensors, and when the equipment is abnormal, perform equipment analysis based on the data from the abnormal time nodes in the multimodal data to achieve secondary verification of the abnormal state. This eliminates the need for technicians to spend a lot of time and effort on secondary equipment testing, improving monitoring accuracy and simplifying monitoring steps, thereby improving processing efficiency and meeting existing application needs.

[0127] This invention also provides a fault equipment analysis device based on multimodal data, see [link to relevant documentation]. Figure 3 The diagram shows a structural schematic of a fault equipment analysis device based on multimodal data provided in an embodiment of the present invention.

[0128] As an example, the fault equipment analysis device based on multimodal data may include: The monitoring module 201 is used to collect multimodal data of the device under monitoring in real time. The multimodal data is obtained by collecting different data from different monitoring devices while the device under monitoring is running in real time, and then optimizing the quality of the data. Feature module 202 is used to call a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, and to optimize and transform the low-dimensional feature vectors to obtain modal feature vectors; Anomaly module 203 is used to determine a state deviation value based on the modal feature vector, and to determine whether the monitored device is abnormal based on the magnitude of the state deviation value, wherein the state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state; The analysis module 204 is used to perform fault analysis processing based on the modal feature vector if the monitored device is determined to be abnormal, and to obtain the analysis results. The fault analysis processing is the process of determining the fault type and fault severity based on the abnormal time nodes of the modal feature vector.

[0129] Optionally, the step of performing fault analysis processing based on the modal feature vector to obtain the analysis result includes: Anomaly time points are obtained from the modal feature vector based on a sliding window, where the anomaly time point is the time point when the state deviation value meets the threshold. Obtain the abnormal duration data corresponding to the abnormal time node, wherein the abnormal duration data is the abnormal data within a preset duration before and after the abnormal time node. The preset anomaly analysis model is invoked to determine the fault type and severity of the monitored equipment based on the anomaly duration data.

[0130] Optionally, the step of calling a preset anomaly analysis model to determine the fault type and severity of the monitored device based on the anomaly duration data includes: The fault type is obtained by classifying the abnormal duration data using the loss function of a preset anomaly analysis model. Based on the key feature parameters corresponding to the fault type, abnormal parameter values ​​are extracted from the abnormal duration data, and the severity of the fault is determined by comparing the values ​​of the abnormal parameters with several preset fault thresholds.

[0131] Optionally, the preset anomaly analysis model is a model trained using historical data prior to the anomaly time point; before training the preset anomaly analysis model, the method further includes: Historical modal data of the device to be monitored is acquired, and the historical modal data is integrated and verified to obtain historical training data; If the amount of historical training data meets the requirements, then the historical training data is used for model training. If the amount of historical training data is insufficient, the sample size of the historical training data is expanded by data augmentation and supplementary data collection methods, and the expanded historical training data is used for model training.

[0132] Optionally, determining the state deviation value based on the modal feature vector and determining whether the monitored device is abnormal based on the magnitude of the state deviation value includes: The Euclidean distance and Mahalanobis distance between the modal feature vector and the preset state baseline are calculated respectively, wherein the preset state baseline is a vector generated based on the normal operation data of the device under health monitoring. The state deviation value is calculated using the Euclidean distance and the Mahalanobis distance; If the state deviation value is greater than the preset deviation threshold, the monitored device is determined to be abnormal. If the state deviation value is less than or equal to the preset deviation threshold, the device under monitoring is determined to be normal.

[0133] Optionally, the step of invoking a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data and optimizing and transforming the low-dimensional feature vectors to obtain modal feature vectors includes: A preset encoding architecture is invoked to extract low-dimensional feature vectors from the multimodal data, wherein the preset encoding architecture is obtained by setting the number of layers in the encoder and decoder and training them. The low-dimensional feature vectors are optimized and transformed to obtain modal feature vectors that conform to a custom scaling ratio. The optimization and transformation process includes feature alignment, multiple scaling and evaluation, and accuracy verification.

[0134] Optionally, the feature alignment operation is an operation that maps each of the low-dimensional feature vectors to the same dimensional space; The multi-group scaling and evaluation operation involves selecting feature vectors that meet custom conditions from a number of low-dimensional feature vectors. The custom conditions are a preset evaluation discrimination, a preset correlation between evaluation and device status, and a preset evaluation stability threshold for low-dimensional feature vectors. The accuracy verification operation is to calculate whether the accuracy retention rate of the low-dimensional feature vectors selected by the multiple scaling and evaluation operations meets a preset threshold.

[0135] Optionally, the device further includes: The image module is used to, after the step of performing fault analysis processing based on the modal feature vector and obtaining the analysis results, if it is determined that the device to be monitored is abnormal, call the camera at the site of the device to be monitored to obtain the real-time operating image of the device to be monitored. The structure detection module is used to extract image features from the real-time running image based on the fault type of the analysis results, and to analyze and process the structure of the device to be monitored based on the image features.

[0136] Optionally, the real-time acquisition of multimodal data from the monitored device includes: The system acquires raw operating data of the monitored equipment in real time, including electrical parameters, mechanical parameters, and environmental parameters. The original operating data is optimized for data quality to obtain multimodal data. The data quality optimization includes filtering and normalization.

[0137] Those skilled in the art will understand that, for ease of description and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] Furthermore, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fault device analysis method based on multimodal data as described in the above embodiments.

[0139] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to execute the fault device analysis method based on multimodal data as described in the above embodiments.

[0140] In the description of the embodiments of the present invention, it should be noted that the terms "above," "below," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. When an element such as a layer, region, or substrate is referred to as being "above" or "on top of" another element, it may be directly on the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "below" or "under" another element, it may be directly below or under the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly below" or "under" another element, there is no intermediate element. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0141] Those skilled in the art will understand that embodiments of this application may also include computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), devices, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A fault equipment analysis method based on multimodal data, characterized in that, The method includes: Real-time acquisition of multimodal data from the device under monitoring, wherein the multimodal data is obtained by collecting different data from different monitoring devices while the device under monitoring is running in real time, and the data quality is optimized; A preset encoding architecture is invoked to extract low-dimensional feature vectors from the multimodal data, and the low-dimensional feature vectors are optimized and transformed to obtain modal feature vectors. The state deviation value is determined based on the modal feature vector, and the magnitude of the state deviation value is used to determine whether the monitored device is abnormal. The state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state. If the monitored device is determined to be abnormal, fault analysis processing is performed based on the modal feature vector to obtain the analysis results. The fault analysis processing is the process of determining the fault type and severity based on the abnormal time nodes of the modal feature vector.

2. The fault equipment analysis method based on multimodal data according to claim 1, characterized in that, The fault analysis process based on the modal feature vectors, to obtain the analysis results, includes: Anomaly time points are obtained from the modal feature vector based on a sliding window, where the anomaly time point is the time point when the state deviation value meets the threshold. Obtain the abnormal duration data corresponding to the abnormal time node, wherein the abnormal duration data is the abnormal data within a preset duration before and after the abnormal time node. The preset anomaly analysis model is invoked to determine the fault type and severity of the monitored equipment based on the anomaly duration data.

3. The fault equipment analysis method based on multimodal data according to claim 2, characterized in that, The step of calling a preset anomaly analysis model to determine the fault type and severity of the monitored device based on the anomaly duration data includes: The fault type is obtained by classifying the abnormal duration data using the loss function of a preset anomaly analysis model. Based on the key feature parameters corresponding to the fault type, abnormal parameter values ​​are extracted from the abnormal duration data, and the severity of the fault is determined by comparing the values ​​of the abnormal parameters with several preset fault thresholds.

4. The fault equipment analysis method based on multimodal data according to claim 2, characterized in that, The preset anomaly analysis model is a model trained using historical data prior to the anomaly time point; before training the preset anomaly analysis model, the method further includes: Historical modal data of the device to be monitored is acquired, and the historical modal data is integrated and verified to obtain historical training data; If the amount of historical training data meets the requirements, then the historical training data is used for model training. If the amount of historical training data is insufficient, the sample size of the historical training data is expanded by data augmentation and supplementary data collection methods, and the expanded historical training data is used for model training.

5. The fault equipment analysis method based on multimodal data according to claim 1, characterized in that, The step of determining the state deviation value based on the modal feature vector and determining whether the monitored device is abnormal based on the magnitude of the state deviation value includes: The Euclidean distance and Mahalanobis distance between the modal feature vector and the preset state baseline are calculated respectively, wherein the preset state baseline is a vector generated based on the normal operation data of the device under health monitoring. The state deviation value is calculated using the Euclidean distance and the Mahalanobis distance; If the state deviation value is greater than the preset deviation threshold, the monitored device is determined to be abnormal. If the state deviation value is less than or equal to the preset deviation threshold, the device under monitoring is determined to be normal.

6. The fault equipment analysis method based on multimodal data according to claim 1, characterized in that, The step of invoking a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data and optimizing and transforming the low-dimensional feature vectors to obtain modal feature vectors includes: A preset encoding architecture is invoked to extract low-dimensional feature vectors from the multimodal data, wherein the preset encoding architecture is obtained by setting the number of layers in the encoder and decoder and training them. The low-dimensional feature vectors are optimized and transformed to obtain modal feature vectors that conform to a custom scaling ratio. The optimization and transformation process includes feature alignment, multiple scaling and evaluation, and accuracy verification.

7. The fault equipment analysis method based on multimodal data according to claim 6, characterized in that, The feature alignment operation is the operation of mapping each of the low-dimensional feature vectors to the same dimensional space. The multi-group scaling and evaluation operation involves selecting feature vectors that meet custom conditions from a number of low-dimensional feature vectors. The custom conditions are a preset evaluation discrimination, a preset correlation between evaluation and device status, and a preset evaluation stability threshold for low-dimensional feature vectors. The accuracy verification operation is to calculate whether the accuracy retention rate of the low-dimensional feature vectors selected by the multiple scaling and evaluation operations meets a preset threshold.

8. The fault equipment analysis method based on multimodal data according to any one of claims 1-7, characterized in that, After the step of performing fault analysis processing based on the modal feature vector to obtain the analysis result, the method further includes: If the device under monitoring is determined to be abnormal, the camera at the site of the device under monitoring is used to obtain real-time images of the device's operation. Based on the fault type identified in the analysis results, image features are extracted from the real-time operating images, and the structure of the device to be monitored is analyzed and processed based on these image features.

9. The fault equipment analysis method based on multimodal data according to any one of claims 1-7, characterized in that, The real-time acquisition of multimodal data from the monitored device includes: The system acquires raw operating data of the monitored equipment in real time, including electrical parameters, mechanical parameters, and environmental parameters. The original running data is optimized for data quality to obtain multimodal data. The data quality optimization includes filtering and normalization.

10. A fault equipment analysis device based on multimodal data, characterized in that, The device includes: The monitoring module is used to collect multimodal data of the device under monitoring in real time. The multimodal data is obtained by collecting different data from different monitoring devices while the device under monitoring is running in real time, and then optimizing the quality of the data. The feature module is used to call a preset encoding architecture to extract low-dimensional feature vectors from the multimodal data, and to optimize and transform the low-dimensional feature vectors to obtain modal feature vectors. An anomaly module is used to determine a state deviation value based on the modal feature vector, and to determine whether the monitored device is abnormal based on the magnitude of the state deviation value, wherein the state deviation value is the deviation between the modal feature vector and the state baseline of the monitored device in the monitoring state; The analysis module is used to perform fault analysis processing based on the modal feature vector if the monitored device is determined to be abnormal, and to obtain the analysis results. The fault analysis processing is the process of determining the fault type and fault severity based on the abnormal time nodes of the modal feature vector.

Images