Equipment fault diagnosis and analysis system and method based on multiple sensors

Abstract

The invention relates to the field of equipment fault diagnosis, and particularly discloses an equipment fault diagnosis and analysis system and method based on multiple sensors. The method comprises the following steps: firstly, acquiring an industrial equipment operation video collected by a video collector, an industrial equipment vibration signal collected by a vibration sensor and load values of the industrial equipment at a plurality of preset time points collected by a load sensor, and then performing feature extraction and correlation analysis on the three by using a deep learning technology; and finally, a classification result is obtained through a classifier to judge whether the industrial equipment breaks down or not, so that potential problems are captured before the equipment breaks down seriously, the maintenance cost is reduced, the maintenance time is shortened, and the overall efficiency and safety of industrial production are improved.

Description

Technical Field

[0001] The present application relates to the field of equipment fault diagnosis, and more specifically, to a multi-sensor based equipment fault diagnosis and analysis system and method. Background Art

[0002] With the rapid progress of economy and technology, the degree of mechanization in industry has been continuously improved, and large-scale mechanical equipment has become a core component of production. However, once the equipment fails, it may cause the entire production process to stagnate, causing huge economic losses to the factory. Therefore, during the operation of the equipment, it is particularly important to determine its overall or partial normal operation status. Early detection and analysis of the cause of the failure, and rapid repair are of great significance to modern industry.

[0003] Currently, factories usually rely on experts' years of maintenance experience to predict equipment failures and perform health management. Although this method of relying on expert experience is effective, it requires a lot of manpower and time to conduct comprehensive inspections of equipment to determine the cause of the failure and repair it.

[0004] Therefore, a multi-sensor based equipment fault diagnosis and analysis system and method is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a multi-sensor based equipment fault diagnosis and analysis system and method, which first obtains the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally obtains the classification result through the classifier to determine whether the industrial equipment has failed, so as to capture potential problems before the equipment fails seriously, reduce maintenance costs and maintenance time, and thus improve the overall efficiency and safety of industrial production.

[0006] According to one aspect of the present application, a multi-sensor based equipment fault diagnosis and analysis system is provided, which includes:

[0007] The equipment fault diagnosis data acquisition module is used to acquire the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor;

[0008] An equipment fault diagnosis data extraction module is used to extract a multimodal management feature vector of the industrial equipment operation state and a time series correlation feature vector of the industrial equipment load value from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor;

[0009] The industrial equipment fault judgment module is used to judge whether the industrial equipment has a fault based on the multi-modal management feature vector of the industrial equipment operation status and the time series correlation feature vector of the industrial equipment load value.

[0010] According to another aspect of the present application, a multi-sensor based equipment fault diagnosis and analysis method is provided, which includes:

[0011] Acquire the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor;

[0012] Extracting the multimodal management feature vector of the industrial equipment operation state and the time series correlation feature vector of the industrial equipment load value from the industrial equipment operation state collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, the industrial equipment video collected by the load sensor, and the load value of the industrial equipment at multiple predetermined time points;

[0013] Based on the multi-modal management feature vector of the industrial equipment operating status and the time series correlation feature vector of the industrial equipment load value, it is determined whether the industrial equipment fails.

[0014] Compared with the prior art, the present application provides a multi-sensor based equipment fault diagnosis and analysis system and method, which first obtains the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally obtains the classification result through the classifier to determine whether the industrial equipment has a fault, thereby capturing potential problems before the equipment has a serious fault, reducing repair costs and maintenance time, and thus improving the overall efficiency and safety of industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 4 is a block diagram of a multi-sensor based equipment fault diagnosis and analysis system according to an embodiment of the present application.

[0017] Figure 2 4 is a block diagram of an equipment fault diagnosis data extraction module in a multi-sensor based equipment fault diagnosis and analysis system according to an embodiment of the present application.

[0018] Figure 3 It is a block diagram of an industrial equipment operation video feature extraction unit in a multi-sensor based equipment fault diagnosis and analysis system according to an embodiment of the present application.

[0019] Figure 4 4 is a block diagram of an industrial equipment fault judgment module in a multi-sensor based equipment fault diagnosis and analysis system according to an embodiment of the present application.

[0020] Figure 5 The present invention is a flowchart of a method for diagnosing and analyzing equipment faults based on multiple sensors according to an embodiment of the present application.

[0021] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 FIG. 1 is a block diagram of a multi-sensor equipment fault diagnosis and analysis system according to an embodiment of the present application. Figure 1 As shown, according to the multi-sensor based equipment fault diagnosis and analysis system 100 of the embodiment of the present application, it includes: an equipment fault diagnosis data acquisition module 110, which is used to obtain the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; an equipment fault diagnosis data extraction module 120, which is used to extract the industrial equipment operation status multimodal management feature vector and the industrial equipment load value time series correlation feature vector from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; an industrial equipment fault judgment module 130, which is used to judge whether the industrial equipment has a fault based on the industrial equipment operation status multimodal management feature vector and the industrial equipment load value time series correlation feature vector.

[0024] In the above-mentioned multi-sensor-based equipment fault diagnosis and analysis system 100, the equipment fault diagnosis data acquisition module 110 is used to obtain the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor. It should be understood that with the rapid development of economy and technology, the mechanical equipment in industrial production is increasingly automated and has become an indispensable and important part of the production process. However, once the equipment fails, it may cause the interruption of the entire production chain and cause serious economic losses to the factory. Therefore, it is particularly important to accurately judge the normal operating status of the equipment. It is of vital importance to modern industry to promptly discover and analyze the cause of the failure and quickly take repair measures. At present, many factories still rely on experienced experts to predict equipment failures and perform health management. Although this method is effective to a certain extent, it requires a lot of manpower and time to determine potential failure factors and repair them through a comprehensive inspection of the equipment. Therefore, in the technical solution of the present application, by obtaining the operation video of industrial equipment collected by the video collector, the vibration signal of industrial equipment collected by the vibration sensor, and the load values ​​of industrial equipment at multiple predetermined time points collected by the load sensor, and combining with deep learning technology, it is possible to accurately judge whether the industrial equipment has a fault, so as to timely discover potential hidden dangers before the problem further deteriorates, thereby reducing maintenance costs and maintenance time, and then improving the overall efficiency and safety of industrial production.

[0025] Specifically, video data provides visual information that can be used to detect abnormal behavior or suspicious activities in equipment operation, such as abnormal movement or heat distribution. Vibration sensors record equipment vibration frequencies and patterns, which are critical for detecting abnormal operation of mechanical components and can reveal possible mechanical wear, imbalance or other structural problems. Load sensors measure changes in the equipment's workload, which is critical for evaluating the equipment's working condition and load-bearing capacity. By comprehensively analyzing this data, the system can promptly identify and predict possible equipment failures, thereby achieving preventive maintenance and minimizing equipment downtime, improving production efficiency and equipment reliability.

[0026] In the above-mentioned multi-sensor-based equipment fault diagnosis and analysis system 100, the equipment fault diagnosis data extraction module 120 is used to extract the multimodal management feature vector of the industrial equipment operation status and the time series correlation feature vector of the industrial equipment load value from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor. It should be understood that in this way, the operating status of the industrial equipment can be fully understood and evaluated by comprehensively analyzing different types of sensor data. These data combined can reveal the overall picture and details of the equipment operation, help identify potential fault signs and abnormal behaviors, help to promptly discover possible problems with the equipment, and provide a scientific basis for equipment maintenance to ensure the long-term reliable operation of the equipment and maximize production efficiency.

[0027] Figure 2 FIG. 1 is a block diagram of a device fault diagnosis data extraction module in a multi-sensor based device fault diagnosis and analysis system according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the equipment fault diagnosis data extraction module 120 includes: an industrial equipment operation video feature extraction unit 121, which is used to extract features from the industrial equipment operation video collected by the video collector to obtain an industrial equipment operation-related feature vector; an industrial equipment vibration signal feature extraction unit 122, which is used to extract features from the industrial equipment vibration signal collected by the vibration sensor to obtain an industrial equipment vibration signal feature vector; an industrial equipment operation status feature association unit 123, which is used to associate the industrial equipment operation-related feature vector with the industrial equipment vibration signal feature vector to obtain the industrial equipment operation status multimodal management feature vector; an industrial equipment load value feature extraction unit 124, which is used to extract features from the load values ​​of the industrial equipment at multiple predetermined time points collected by the load sensor to obtain the industrial equipment load value time series association feature vector.

[0028] It should be understood that the industrial equipment operation video is collected by the video collector and contains visual information of the equipment during operation, such as movement, shape and environmental background. These visual data contain rich operation characteristics, such as movement patterns, changes in the working area, and possible abnormal activities or signs of failure. Through the feature extraction process, the system can extract numerical descriptors that can represent these features from the video, such as movement direction, speed change, object recognition, etc. These descriptors constitute the components of the industrial equipment operation related feature vector.

[0029] Furthermore, feature extraction of industrial equipment vibration signals to obtain vibration signal feature vectors is to deeply understand the operating status and health status of the equipment by analyzing the spectrum, amplitude, waveform and other characteristics of the vibration signal. The vibration signals collected by the vibration sensor reflect the mechanical vibration generated by the equipment during operation. These vibration signals contain rich information, such as the operating frequency, vibration amplitude, periodic changes and possible abnormal vibration modes of the equipment. Through feature extraction, complex vibration signals can be converted into feature vectors with practical significance and analysis capabilities. These feature vectors usually include frequency domain features (such as spectrum distribution, frequency components), time domain features (such as root mean square, kurtosis, waveform shape), and statistical features (such as mean, variance, correlation, etc.). These features can effectively describe the basic characteristics and changing trends of equipment vibration.

[0030] Furthermore, associating the associated feature vectors of industrial equipment operation with the feature vectors of vibration signals is to combine the data collected by different types of sensors to comprehensively and accurately describe and evaluate the operating status and health of the equipment. The operation of industrial equipment involves many aspects. The video collector records the visual information of the equipment, while the vibration sensor captures the mechanical vibration signal of the equipment. These data sources contain their own unique characteristics. By associating these feature vectors, the visual operation characteristics and vibration characteristics of the equipment can be comprehensively considered to form a more comprehensive and expressive description of the operating status of industrial equipment. By comprehensively analyzing the associated feature vectors of different sensor data, the system can achieve more advanced monitoring and management functions, such as real-time operation status monitoring, predictive maintenance, and intelligent decision support. This approach can not only improve the reliability and safety of equipment, but also reduce maintenance costs and the risk of production interruptions.

[0031] In particular, feature extraction is performed on the load values ​​of industrial equipment collected by load sensors at multiple predetermined time points, aiming to obtain time-series correlation feature vectors that can effectively describe the equipment's operating status and load change trends by analyzing and processing these load data. Load sensors record the load conditions of equipment in different time periods. These load data reflect the equipment's workload, energy consumption changes, and possible load fluctuations. Through the feature extraction process, the system can extract a series of numerical descriptors from the load data, such as average load, peak load, load distribution characteristics (such as variance or standard deviation), and load change trends and periodicity.

[0032] Figure 3 FIG. 1 is a block diagram of an industrial equipment operation video feature extraction unit in a multi-sensor based equipment fault diagnosis and analysis system according to an embodiment of the present application. Figure 3As shown, in a specific embodiment of the present application, the industrial equipment operation video feature extraction unit 121 includes: an industrial equipment operation video key frame extraction subunit 1211, which is used to extract key frames from the industrial equipment operation video collected by the video collector to obtain multiple industrial equipment operation images; an equipment operation image convolution encoding subunit 1212, which is used to pass the multiple industrial equipment operation images through an industrial equipment operation convolutional neural network model with multi-feature extraction capabilities to obtain multiple industrial equipment operation feature graphs; an industrial equipment operation video feature encoding subunit 1213, which is used to feature encode the multiple industrial equipment operation feature graphs to obtain the industrial equipment operation associated feature vector.

[0033] It should be understood that keyframe extraction is a method of selecting representative and informative frames from a video sequence. By analyzing the motion, changes, and activities in the video, the system can select keyframes that best represent the operating status of the equipment. These keyframes can include different postures of the equipment, key operation stages, or key time points when abnormal situations occur, thus providing comprehensive visual data support for engineers and analysts.

[0034] Furthermore, multiple industrial equipment operation images are processed by an industrial equipment operation convolutional neural network model with multi-feature extraction capabilities, aiming to extract rich and advanced operation feature maps from these images to deeply analyze and understand the operation status, performance characteristics and possible abnormal conditions of the equipment. Industrial equipment operation images contain visual information of the equipment at different time points, such as the movement trajectory, working posture, operation process and surrounding environment of the equipment. These images can be processed by a convolutional neural network (CNN) model, which has excellent feature extraction capabilities and can learn and extract representative features from complex image data. Through these feature extractions, key feature maps of multiple industrial equipment during operation can be obtained, such as the movement mode of the equipment, changes in working status, and the operating status of equipment components. By inputting the operation images of multiple industrial equipment into a unified CNN model for processing, comparative analysis and performance evaluation between equipment can also be achieved. This data-driven approach not only helps to optimize the design and operation process of equipment, but also improves the efficiency and quality level of industrial production. Specifically, the industrial equipment with multi-feature extraction capability runs a convolutional neural network model module, including: a first convolutional layer, a second convolutional layer parallel to the first convolutional layer, and a cascade layer connected to the first convolutional layer and the second convolutional layer, wherein the first convolutional layer uses a one-dimensional convolutional kernel with a first scale, and the second convolutional layer uses a one-dimensional convolutional kernel with a second scale.

[0035] Furthermore, the industrial equipment operation feature map is obtained by processing and analyzing the equipment operation video or image data, which contains information such as the equipment's motion trajectory, changes in working status, and the activity of equipment components. These image data themselves have a rich amount of information, but direct use may result in excessive data dimensions and information redundancy, which is not conducive to subsequent data processing and analysis. Therefore, through the process of feature encoding, each industrial equipment operation feature map can be mapped to a lower-dimensional and more representative feature space. This process can be achieved using techniques such as autoencoders, convolutional neural networks (CNNs), or other feature extraction techniques. The goal of feature encoding is to extract the key features that best characterize the equipment's operating status and characteristics, such as motion patterns, frequency components, shape features, etc. These features can retain important information about the equipment's operation while reducing data redundancy.

[0036] In a specific embodiment of the present application, the industrial equipment operation video feature encoding subunit 1213 includes: passing the multiple industrial equipment operation feature maps through an industrial equipment operation target detection network to obtain multiple industrial equipment operation regions of interest; passing the multiple industrial equipment operation regions of interest through an industrial equipment operation significant target detector to obtain the industrial equipment operation associated feature vector.

[0037] It should be understood that the industrial equipment operation feature map contains the operation status and behavior characteristics of the equipment at different time points, such as the equipment's posture, action, motion trajectory, etc. These image data can automatically identify and locate the key components or operation characteristics of the equipment through the target detection network, such as the position of the robot arm, the operation status of the conveyor belt, the working area of ​​the equipment, etc. The industrial equipment operation target detection network is usually based on deep learning technology, such as the target detection model based on the convolutional neural network (CNN) (such as Faster R-CNN, YOLO, etc.), which can effectively locate and identify the key parts of different equipment in complex industrial scenes. Through these networks, it is possible to realize automatic detection and analysis of multiple devices in the equipment operation image, improving the efficiency and accuracy of data processing.

[0038] Furthermore, the regions of interest for industrial equipment operation are key parts of the equipment or operating feature areas that are accurately identified and located by the aforementioned target detection network, such as the end effector of the robot arm, the key motion area of ​​the conveyor belt, etc. These areas are considered to be the most informative and decision-critical parts of the equipment operation process, and the representation and recognition capabilities of these areas can be further enhanced through salient target detectors. Salient target detectors for industrial equipment operation are usually a special target detection technology that is specifically designed to extract and highlight salient parts of equipment operation from complex industrial scenes. This technology may be based on visual attention mechanisms, feature enhancement algorithms, or specific deep learning structures, and aims to improve the accuracy of positioning and identifying key features of equipment. Specifically, each layer of the industrial equipment operation salient target detector is used to perform the following on multiple industrial equipment operation interest regions in the forward pass of the layer: use a first convolution kernel to perform convolution processing on the multiple industrial equipment operation interest regions to obtain a first convolution feature map; use a second convolution kernel to perform convolution processing on the first convolution feature map to obtain a second convolution feature map, wherein the size of the first convolution kernel is larger than the size of the second convolution kernel; perform pooling processing on the second convolution feature map to obtain a pooling feature map; perform activation processing on the pooling feature map to obtain an activation feature map; wherein the output of the last layer of the industrial equipment operation salient target detector is the industrial equipment operation associated feature vector, and the input of the first layer of the industrial equipment operation salient target detector is the multiple industrial equipment operation interest regions.

[0039] In a specific embodiment of the present application, the industrial equipment vibration signal feature extraction unit 122 includes: performing a short-time Fourier transform on the industrial equipment vibration frequency image to obtain an industrial equipment vibration signal spatial frequency image; performing a two-dimensional Fourier transform on the industrial equipment vibration signal spatial frequency image to obtain an industrial equipment vibration signal spatial frequency image; and performing an industrial equipment vibration signal spatial frequency image through an industrial equipment vibration signal spatial frequency feature encoder to obtain a feature vector of the industrial equipment vibration signal.

[0040] It should be understood that the purpose of processing the vibration signal of industrial equipment through short-time Fourier transform (STFT) is to convert from the time domain to the frequency domain to obtain the frequency image of the equipment vibration. This transformation can effectively analyze the spectral characteristics of the equipment vibration, help identify the frequency components and periodic changes in the vibration signal, and thus achieve in-depth monitoring and evaluation of the equipment's operating status and health. The vibration signal of industrial equipment contains the vibration waveform caused by mechanical movement or operation during the operation of the equipment. These waveforms will show complex changes in time, and the frequency characteristics of vibration are often one of the important indicators for evaluating the health status and performance stability of the equipment. Through STFT, the vibration signal can be decomposed into frequency components in different time periods to generate a time-frequency image, i.e., a spectrogram. STFT allows the observation of the spectral information of the vibration signal in different time periods by performing Fourier transform within the time window. This method allows the capture of instantaneous changes in vibration frequency, and is particularly suitable for analyzing the vibration characteristics of the equipment under different operating states or load conditions. For example, resonance phenomena of different frequencies, abnormal vibrations, or signs of mechanical failure that occur during equipment operation can be detected.

[0041] Furthermore, the purpose of performing a two-dimensional Fourier transform on the vibration frequency image of industrial equipment is to convert the frequency information from the time-frequency domain to the space-frequency domain to obtain a more detailed and comprehensive spatial frequency image of the vibration signal. This transformation can reveal the frequency characteristic distribution of the vibration signal at different positions and directions in space, and help further understand and analyze the complex structure and dynamic characteristics of the equipment vibration. The vibration frequency image of industrial equipment reflects the vibration of the equipment at different time points and frequencies, and is obtained by converting from the time domain to the frequency domain through a short-time Fourier transform. The two-dimensional Fourier transform further transforms this frequency domain information into the space-frequency domain, that is, describes the distribution characteristics of the vibration signal through the dual changes of spatial position and frequency. Through the two-dimensional Fourier transform, the vibration characteristics at each time point and frequency in the vibration frequency image can be mapped to the frequency distribution image in the spatial domain. This transformation allows the observation of the frequency components of the equipment vibration signal at different positions and directions, such as the distribution of vibration modes on the surface of the equipment or between different components. This is particularly helpful for detecting and analyzing local vibration anomalies or frequency resonance phenomena that may exist in the equipment.

[0042] Furthermore, the spatial frequency image of the vibration signal of the industrial equipment reflects the distribution of the vibration frequency of the equipment at different positions in space, which is obtained by two-dimensional Fourier transform. These images are high-dimensional data, in which each pixel corresponds to the vibration frequency characteristics on the surface of the equipment or on different components. The spatial frequency feature encoder of the vibration signal of the industrial equipment can effectively encode and extract features of these images through a specific algorithm or deep learning model. The vibration signal feature vector is obtained by processing the spatial frequency image of the vibration signal through the spatial frequency feature encoder. These feature vectors not only contain the comprehensive information of the frequency distribution of the vibration signal at different spatial positions, but also reflect the structural characteristics and dynamic behavior of the vibration of the equipment. This is of great significance for identifying and analyzing the local vibration anomalies, frequency resonance phenomena that may exist in the equipment, or predicting the health status of the equipment. Specifically, each layer of the industrial equipment load value temporal coding model is used to perform convolution processing, mean pooling processing based on the local feature matrix, and nonlinear activation processing on the input data in the forward transfer of the layer to output the industrial equipment vibration signal feature vector from the last layer of the industrial equipment load value temporal coding model, wherein the input of the industrial equipment load value temporal coding model is the spatial frequency image of the industrial equipment vibration signal.

[0043] In a specific embodiment of the present application, the industrial equipment load value feature extraction unit 124 includes: constructing the load values ​​of the industrial equipment at multiple predetermined time points collected by the load sensor into an industrial equipment load input vector; and passing the industrial equipment load input vector through an industrial equipment load value timing encoding model to obtain the industrial equipment load value timing associated feature vector.

[0044] It should be understood that load sensors are usually installed at key locations of industrial equipment to monitor the load carried by the equipment in real time. These sensors collect load data at predetermined time intervals and record the load changes of the equipment under different working conditions. By constructing an industrial equipment load input vector, these multi-dimensional time series data can be integrated into a vector form with a lower dimension but still containing key information. The industrial equipment load input vector reflects the load status of the equipment at multiple time points, and each vector element represents a load value at a predetermined time point. This vectorized data form facilitates the application of statistical analysis, pattern recognition, and machine learning algorithms, thereby revealing the load characteristics and change patterns of the equipment under different operating conditions.

[0045] Furthermore, the industrial equipment load input vector is processed by the industrial equipment load value time series coding model, aiming to extract the dynamic characteristics of load changes from the static load data and generate a more information-rich and time-correlated load value time series correlation feature vector. This model can capture the evolution law and correlation of equipment load in time series, and provide a more in-depth and accurate analysis basis for equipment status prediction, anomaly detection and operation optimization. The industrial equipment load input vector contains the load values ​​at different time points, which are static data collected and sorted by load sensors. However, a simple static vector cannot reflect the trend and periodicity of load changes over time, nor can it reveal the possible time series correlation and influence between loads. The industrial equipment load value time series coding model can perform time series coding and feature extraction on the load input vector through a specific algorithm or deep learning structure. Specifically, the industrial equipment load input vector is fully-connectedly encoded using the fully-connected layer of the industrial equipment load value temporal coding model to extract high-dimensional implicit features of the eigenvalues ​​at each position in the industrial equipment load input vector; and, the industrial equipment load input vector is one-dimensionally encoded using the one-dimensional convolutional layer of the industrial equipment load value temporal coding model to extract high-dimensional implicit correlation features of the correlation between the eigenvalues ​​at each position in the industrial equipment load input vector.

[0046] In the above-mentioned multi-sensor-based equipment fault diagnosis and analysis system 100, the industrial equipment fault judgment module 130 is used to judge whether the industrial equipment has a fault based on the multi-modal management feature vector of the industrial equipment operation state and the time-series correlation feature vector of the industrial equipment load value. It should be understood that the multi-modal management feature vector of the industrial equipment operation state covers the comprehensive information of the equipment under multiple key parameters and working conditions. These feature vectors may include multi-dimensional data such as temperature, pressure, vibration, current, etc. of the equipment, which are collected and processed in real time by various sensors. Abnormal equipment operation state feature vectors can reflect possible operational abnormalities or performance degradation of the equipment, and are an important basis for fault prediction and diagnosis. The time-series correlation feature vector of the industrial equipment load value can capture the change pattern and trend of the equipment load over time. These vectors can identify periodic fluctuations, abnormal peaks or irregular changes in the load by time-series encoding and analyzing the equipment load data. The abnormal performance of the load value time-series correlation feature vector usually means that the equipment has problems in load carrying capacity or working stability, which may be a precursor to a fault. The process of comprehensively using these two types of feature vectors for fault judgment usually involves machine learning algorithms or pattern recognition technology. By training the model, we can establish the correlation pattern between the normal operation and fault status of the equipment, so that the system can automatically monitor and identify abnormal conditions. This fault diagnosis method based on multimodal feature vectors has high accuracy and real-time performance, which can help enterprises effectively reduce maintenance costs, improve equipment reliability and production efficiency. By timely discovering and handling equipment failures, we can also avoid production losses and safety risks caused by equipment downtime, thereby achieving continuous production operation optimization and management.

[0047] Figure 4 FIG. 1 is a block diagram of an industrial equipment fault judgment module in a multi-sensor equipment fault diagnosis and analysis system according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the industrial equipment fault judgment module 130 includes: an industrial equipment fault data feature fusion unit 131, which is used to fuse the industrial equipment operation status multimodal management feature vector and the industrial equipment load value time series correlation feature vector to obtain an industrial equipment fault judgment feature vector; an industrial equipment fault data feature optimization unit 132, which is used to perform backward learning-type attention optimization based on feature-level error evaluation on the industrial equipment fault judgment feature vector to obtain an optimized industrial equipment fault judgment feature vector; an industrial equipment fault occurrence judgment classification unit 133, which is used to pass the optimized industrial equipment fault judgment feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the industrial equipment has a fault.

[0048] It should be understood that by fusing the multimodal management feature vector of the industrial equipment operating status and the time series association feature vector of the industrial equipment load value, an industrial equipment fault judgment feature vector can be generated through appropriate data integration and model combination. The industrial equipment fault judgment feature vector not only comprehensively considers the various characteristics of the equipment operating status, but also combines the time series change characteristics of the load value, so as to more comprehensively describe the working status of the equipment and possible fault signs. For example, the simultaneous appearance of abnormal operating status and load time series association in the feature vector can provide a more reliable basis for fault prediction and diagnosis. This method of fusing feature vectors is often implemented and optimized through machine learning algorithms or deep learning models. These models can learn the relationship between normal and abnormal equipment states from historical data, and then realize the prediction and early warning of possible future fault conditions.

[0049] Furthermore, in the technical solution of the present application, it is considered that the data collected by the video collector, vibration sensor and load sensor may be affected by factors such as equipment accuracy, environmental interference, and installation location, resulting in uneven quality of the original data. The process of extracting key frames from the video may not fully represent the operating status of the entire industrial equipment, and some important transient behaviors may not be captured. The industrial equipment operation convolutional neural network model may not capture all features related to the equipment status, especially when the equipment status changes are subtle or the model training is insufficient. The industrial equipment operation target detection network and the significant target detector may not accurately identify all key areas of interest or significant targets, resulting in incomplete feature extraction. The industrial equipment vibration signal spatial frequency feature encoder may not fully and accurately encode the features of the vibration signal, resulting in a deviation between the feature vector and the actual equipment state. Fusion of feature vectors from different sensors is a complex process that may introduce additional noise or errors, which in turn causes the industrial equipment fault judgment feature vector to have a class harmony offset relative to the predetermined class label of the classifier. In order to reduce the class harmony offset, in the technical solution of the present application, the industrial equipment fault judgment feature vector is subjected to backward learning-like attention optimization based on feature-level error evaluation.

[0050] Among them, the industrial equipment fault judgment feature vector is subjected to backward learning-like attention optimization based on feature-level error evaluation to obtain an optimized industrial equipment fault judgment feature vector, including: performing probabilistic activation based on Sigmoid function on the industrial equipment fault judgment feature vector to obtain a probabilistic industrial equipment fault judgment feature vector; inputting the industrial equipment fault judgment feature vector into a classifier to obtain an initial class label probability value; calculating the absolute value of the difference between the feature value at each position in the probabilistic industrial equipment fault judgment feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error metric vector; using the backward machine learning fine-grained The error metric vector is used as a key vector, the industrial equipment fault judgment feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning type attention optimized industrial equipment fault judgment feature vector; the ratio between the sum of the backward machine learning fine-grained error metric vector and the length of the backward machine learning fine-grained error metric vector is calculated, and the natural exponential function value with the ratio as a power is calculated as an optimization scaling factor; the product between the optimization scaling factor and the backward learning type attention optimized industrial equipment fault judgment feature vector is calculated to obtain the optimized industrial equipment fault judgment feature vector.

[0051] The optimization steps are specifically expressed as follows:

[0052]

[0053] Among them, V c represents the probabilistic industrial equipment fault judgment feature vector, P represents the initial class label probability value, Indicates subtraction by position, V e represents the fine-grained error metric vector for backward machine learning, v i represents the eigenvalue of the i-th position of the backward machine learning fine-grained error metric vector, L represents the length of the backward machine learning fine-grained error metric vector, T represents the transpose of the vector, represents vector multiplication, Softmax represents the normalized exponential function, ⊙ represents point multiplication by position, V c ' represents the optimized industrial equipment fault judgment feature vector.

[0054] Based on this, in the technical solution of the present application, the industrial equipment fault judgment feature vector is subjected to backward learning-type attention optimization based on feature-level error evaluation, which first implements probabilistic activation based on the Sigmoid function on the industrial equipment fault judgment feature vector to obtain a probabilistic industrial equipment fault judgment feature vector, and inputs the industrial equipment fault judgment feature vector into the classifier to obtain the initial class label probability value. Then, the error between the probabilistic industrial equipment fault judgment feature vector and the initial class label probability value is calculated through backward error analysis to generate a fine-grained error metric vector, which quantifies the viewpoint metric contribution of each feature position to the classification error.

[0055] Then, the backward machine learning fine-grained error metric vector is used as a key vector, the industrial equipment fault judgment feature vector is used as a query vector and a value vector, and the key vector, the query vector and the value vector are input into a transformer structure to perform backward machine learning based on the error metric-like attention mechanism on the industrial equipment fault judgment feature vector through a Transformer-like architecture. In this way, the industrial equipment fault judgment feature vector is subjected to a backward interpretability analysis so that the obtained backward learning-like attention-optimized industrial equipment fault judgment feature vector has a backward learning sensitive preference relative to the predetermined class label of the classifier. And further, the ratio of the sum of the error metric vector and the vector length is calculated, and the optimized scaling factor obtained by calculating the natural exponential function value with this ratio as an exponent is used to perform backward scaling optimization on the backward learning-like attention-optimized industrial equipment fault judgment feature vector. In this way, the sensitive preference and robustness of the industrial equipment fault judgment feature vector relative to the predetermined class label of the classifier are improved, thereby improving the accuracy of the classification result.

[0056] Furthermore, the optimized industrial equipment fault judgment feature vector is classified by a classifier to obtain the classification result for judging whether the industrial equipment has failed. This is to use machine learning and data analysis technology to extract key features from a large amount of complex equipment operation data and make accurate judgments. The failure of industrial equipment is usually manifested as abnormal changes in multidimensional data, such as sudden temperature increase, abnormal vibration frequency, etc. These changes often indicate that the equipment may have problems or failures. Classifying the optimized industrial equipment fault judgment feature vector by a classifier is one of the important technical means in modern industrial production management. It can not only help enterprises realize intelligent monitoring and management of equipment, improve production efficiency and product quality, but also reduce maintenance costs and risks, and ensure long-term stable operation of equipment.

[0057] In summary, the embodiment of the present application first obtains the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three. Finally, a classifier is used to obtain a classification result to determine whether the industrial equipment has a fault, thereby capturing potential problems before the equipment has a serious fault, reducing repair costs and maintenance time, and thereby improving the overall efficiency and safety of industrial production.

[0058] As described above, the multi-sensor-based device fault diagnosis and analysis system 100 according to the embodiment of the present application can be implemented in various terminal devices. In one example, the multi-sensor-based device fault diagnosis and analysis system 100 can be integrated into the terminal device as a software module and / or a hardware module. For example, the multi-sensor-based device fault diagnosis and analysis system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the multi-sensor-based device fault diagnosis and analysis system 100 can also be one of the many hardware modules of the terminal device.

[0059] Alternatively, in another example, the multi-sensor based equipment fault diagnosis and analysis system 100 and the terminal device may also be separate devices, and the multi-sensor based equipment fault diagnosis and analysis system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0060] Figure 5 FIG. 1 is a flow chart of a multi-sensor equipment fault diagnosis and analysis method according to an embodiment of the present application. Figure 5 As shown, according to the multi-sensor based equipment fault diagnosis and analysis method of the embodiment of the present application, it includes: S110, acquiring the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; S120, extracting the industrial equipment operation status multimodal management feature vector and the industrial equipment load value time series correlation feature vector from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; S130, judging whether the industrial equipment has a fault based on the industrial equipment operation status multimodal management feature vector and the industrial equipment load value time series correlation feature vector.

[0061] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned multi-sensor based equipment fault diagnosis and analysis method have been referred to above. Figures 1 to 4 The multi-sensor based equipment fault diagnosis and analysis system has been introduced in detail, and therefore, its repeated description will be omitted.

[0062] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0063] Figure 6 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0064] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 14 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0065] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0066] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the multi-sensor based equipment fault diagnosis and analysis methods described in the above embodiments.

[0067] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0068] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0069] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A multi-sensor based equipment fault diagnosis and analysis system, characterized in that: include: The equipment fault diagnosis data acquisition module is used to acquire the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; An equipment fault diagnosis data extraction module is used to extract a multimodal management feature vector of the industrial equipment operation state and a time series correlation feature vector of the industrial equipment load value from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; The industrial equipment fault judgment module is used to judge whether the industrial equipment has a fault based on the multi-modal management feature vector of the industrial equipment operation status and the time series correlation feature vector of the industrial equipment load value.

2. The multi-sensor based equipment fault diagnosis and analysis system according to claim 1, characterized in that: The equipment fault diagnosis data extraction module comprises: An industrial equipment operation video feature extraction unit, used for extracting features from the industrial equipment operation video collected by the video collector to obtain an industrial equipment operation related feature vector; An industrial equipment vibration signal feature extraction unit, used for extracting features from the industrial equipment vibration signal collected by the vibration sensor to obtain an industrial equipment vibration signal feature vector; An industrial equipment operation status feature association unit, used to associate the industrial equipment operation association feature vector with the industrial equipment vibration signal feature vector to obtain the industrial equipment operation status multimodal management feature vector; The industrial equipment load value feature extraction unit is used to extract features from the load values ​​of the industrial equipment at multiple predetermined time points collected by the load sensor to obtain a time series correlation feature vector of the industrial equipment load value.

3. The multi-sensor based equipment fault diagnosis and analysis system according to claim 2, characterized in that: The industrial equipment runs a video feature extraction unit, comprising: An industrial equipment operation video key frame extraction subunit, used to extract key frames from the industrial equipment operation video collected by the video collector to obtain a plurality of industrial equipment operation images; An equipment operation image convolution encoding subunit, used for passing the plurality of industrial equipment operation images through an industrial equipment operation convolution neural network model with multi-feature extraction capability to obtain a plurality of industrial equipment operation feature graphs; The industrial equipment operation video feature encoding subunit is used to perform feature encoding on the multiple industrial equipment operation feature graphs to obtain the industrial equipment operation associated feature vector.

4. The multi-sensor based equipment fault diagnosis and analysis system according to claim 3, characterized in that: The industrial equipment runs a video feature encoding subunit, including: Passing the plurality of industrial equipment operation characteristic graphs through an industrial equipment operation target detection network to obtain a plurality of industrial equipment operation regions of interest; The multiple industrial equipment operation interest regions are passed through an industrial equipment operation salient target detector to obtain the industrial equipment operation associated feature vector.

5. The multi-sensor based equipment fault diagnosis and analysis system according to claim 4, characterized in that: The industrial equipment vibration signal feature extraction unit comprises: The said is subjected to short-time Fourier transformation to obtain a vibration frequency image of the industrial equipment; Performing a two-dimensional Fourier transform on the industrial equipment vibration frequency image to obtain an industrial equipment vibration signal spatial frequency image; The industrial equipment vibration signal spatial frequency image is passed through an industrial equipment vibration signal spatial frequency feature encoder to obtain the industrial equipment vibration signal feature vector.

6. The multi-sensor based equipment fault diagnosis and analysis system according to claim 5, characterized in that: The industrial equipment load value feature extraction unit comprises: constructing the load values ​​of the industrial equipment at a plurality of predetermined time points collected by the load sensor into an industrial equipment load input vector; The industrial equipment load input vector is passed through an industrial equipment load value time series encoding model to obtain the industrial equipment load value time series associated feature vector.

7. The multi-sensor based equipment fault diagnosis and analysis system according to claim 6, characterized in that: The industrial equipment fault judgment module comprises: An industrial equipment fault data feature fusion unit, used to fuse the industrial equipment operating state multimodal management feature vector and the industrial equipment load value time series correlation feature vector to obtain an industrial equipment fault judgment feature vector; An industrial equipment fault data feature optimization unit, used for performing backward learning type attention optimization based on feature level error evaluation on the industrial equipment fault judgment feature vector to obtain an optimized industrial equipment fault judgment feature vector; The industrial equipment fault occurrence judgment classification unit is used to pass the optimized industrial equipment fault judgment feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the industrial equipment has a fault.

8. The multi-sensor based equipment fault diagnosis and analysis system according to claim 7, characterized in that: The industrial equipment fault data feature optimization unit comprises: Performing probabilistic activation based on a Sigmoid function on the industrial equipment fault judgment feature vector to obtain a probabilistic industrial equipment fault judgment feature vector; Inputting the industrial equipment fault judgment feature vector into a classifier to obtain an initial class label probability value; Calculating the absolute value of the difference between the feature value of each position in the probabilistic industrial equipment fault judgment feature vector and the initial class label probability value to obtain a backward machine learning fine-grained error measurement vector; Using the backward machine learning fine-grained error metric vector as a key vector and the industrial equipment fault judgment feature vector as a query vector and a value vector, the key vector, the query vector and the value vector are input into a converter structure to obtain a backward learning class attention optimized industrial equipment fault judgment feature vector; Calculating a ratio between a sum of the backward machine learning fine-grained error metric vector and a length of the backward machine learning fine-grained error metric vector, and calculating a natural exponential function value raised to a power of the ratio as an optimization scaling factor; The product of the optimized scaling factor and the backward learning type attention optimized industrial equipment fault judgment feature vector is calculated to obtain the optimized industrial equipment fault judgment feature vector.

9. A multi-sensor based equipment fault diagnosis and analysis method, characterized in that: include: Acquire the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; Extracting a multimodal management feature vector of the industrial equipment operation state and a time series correlation feature vector of the industrial equipment load value from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor; Based on the multimodal management feature vector of the industrial equipment operating status and the time series correlation feature vector of the industrial equipment load value, it is determined whether the industrial equipment fails.

10. The multi-sensor based equipment fault diagnosis and analysis method according to claim 9, characterized in that: Extracting the multimodal management feature vector of the industrial equipment operation state and the time series correlation feature vector of the industrial equipment load value from the industrial equipment operation video collected by the video collector, the industrial equipment vibration signal collected by the vibration sensor, and the load value of the industrial equipment at multiple predetermined time points collected by the load sensor, including: Extracting features from the industrial equipment operation video collected by the video collector to obtain an industrial equipment operation-related feature vector; Performing feature extraction on the industrial equipment vibration signal collected by the vibration sensor to obtain a feature vector of the industrial equipment vibration signal; Associating the industrial equipment operation association feature vector with the industrial equipment vibration signal feature vector to obtain the industrial equipment operation state multimodal management feature vector; Feature extraction is performed on the load values ​​of the industrial equipment at multiple predetermined time points collected by the load sensor to obtain a time series correlation feature vector of the load value of the industrial equipment.

Images