A device health status monitoring method and system based on acoustic-optical fusion technology
By using sound-light fusion technology, sensor arrays, and deep learning models, the problems of monitoring blind spots and accuracy in existing equipment health status monitoring methods have been solved. This enables comprehensive and accurate monitoring of equipment health status, improves the accuracy of fault identification and location, and reduces the false alarm rate.
Patent Information
- Application Number
- CN202411254669.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-09
AI Technical Summary
Existing equipment health status monitoring methods, such as acoustic monitoring and thermal imaging, each have monitoring blind spots, cannot be cross-validated, and cannot accurately monitor non-thermal driven faults and minor mechanical faults, resulting in reduced accuracy and reliability of fault prediction in complex operating environments.
An equipment health status monitoring method based on acoustic-optical fusion technology is adopted. Sound data, thermal images and operating condition data are collected through a sensor array. Acoustic features are extracted using a first model, and potential temperature anomaly areas are identified by analyzing the thermal images. A second model is then constructed to monitor the equipment health status. Data fusion is performed by combining graph convolutional layers, temporal convolutional layers and multi-head attention mechanisms.
It significantly improves the accuracy and reliability of equipment health status monitoring, eliminates the influence of environmental noise, extracts pure acoustic information, optimizes thermal imaging data, dynamically adjusts data weights, ensures the comprehensiveness and accuracy of monitoring results, reduces false alarm rate, and improves the accuracy of fault identification and location.
Smart Images

Figure CN118758379B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent diagnosis technology, and in particular to a method and system for monitoring the health status of equipment based on acoustic-optical fusion technology. Background Art
[0002] The health status monitoring of industrial equipment is a key link to ensure production safety, improve equipment utilization and reduce maintenance costs. With the advent of the Industrial 4.0 era, equipment health status monitoring technology continues to develop, gradually evolving from traditional single parameter monitoring to multi-parameter fusion analysis. Acoustic monitoring technology has been widely used in equipment fault diagnosis due to its advantages such as non-invasiveness, real-time and high sensitivity. At the same time, infrared thermal imaging technology has shown unique advantages in abnormal hotspot monitoring because it can intuitively reflect the temperature distribution of equipment. However, a single technology is often difficult to fully and accurately reflect the health status of equipment in a complex industrial environment, which has prompted researchers to explore new methods for multi-source information fusion.
[0003] Although the existing acoustic and thermal imaging monitoring technologies have achieved certain applications in their respective fields, they also have many shortcomings in actual use. Acoustic monitoring technology is sensitive to the ambient noise around the equipment, prone to false alarms or missed alarms, and has limited ability to identify some weak mechanical fault characteristics. Although thermal imaging technology can effectively identify abnormal temperature areas on the surface of the equipment, it relies solely on temperature distribution information and cannot make accurate judgments on some non-thermally driven mechanical faults. Therefore, it is necessary to integrate acoustic monitoring technology and thermal imaging technology to cross-validate the monitoring results, significantly reduce the false alarm rate, and improve the accuracy and reliability of monitoring. The integration of the two can expand the monitoring range and monitor weak mechanical faults and non-thermally driven faults. In addition, the existing equipment monitoring methods are usually based on static data analysis, which fails to fully consider the operating differences of the equipment under different working conditions, resulting in reduced accuracy and reliability of fault prediction in complex operating environments. Summary of the invention
[0004] In view of the problems existing in the existing health status monitoring technology, the present invention is proposed.
[0005] Therefore, the problem to be solved by the present invention is how to provide a method and system for monitoring the health status of equipment based on acoustic and optical fusion technology, so as to solve the technical problem that in the existing equipment health status monitoring methods, the acoustic monitoring technology and thermal imaging technology each have monitoring blind spots, which not only cannot be cross-verified, but also cannot monitor non-thermally driven faults and even some weak mechanical faults?
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] In a first aspect, an embodiment of the present invention provides a device health status monitoring method based on acoustic-optical fusion technology, which includes:
[0008] Collecting data based on a sensor array; the data includes sound data, thermal images and operating condition data;
[0009] Inputting the sound data into a first model to obtain acoustic features;
[0010] Analyzing the thermal map and identifying potential temperature anomaly areas, and determining thermal imaging features based on the potential temperature anomaly areas;
[0011] A second model is constructed according to the acoustic features, the thermal imaging features and the operating condition data, and the health status of the equipment is monitored by using the second model.
[0012] As a preferred solution of the equipment health status monitoring method based on the acoustic-optical fusion technology of the present invention, wherein: the sensor array includes a patch-type sound sensor, an infrared thermal imaging sensor and a working condition sensor;
[0013] The acoustic feature is obtained by spatially filtering the sound data through the first model and adjusting the time delay and amplitude weight of the sensor array signal, as shown in the following formula:
[0014] ;
[0015] in, is the output of the first model, i is the target direction angle, f is the frequency, T is the temperature, M is the number of patch sound sensors, w i For the i The weight of a patch sound sensor, For the i The frequency domain representation of the signal received by a patch sound sensor, t i (i) Based on the target direction angle i Calculated i The time delay of a patch sound sensor.
[0016] As a preferred solution of the device health status monitoring method based on the acoustic-optical fusion technology of the present invention, wherein: analyzing the thermal map and identifying potential temperature abnormality areas includes the following steps:
[0017] Determine the threshold range based on the current operating status of the device;
[0018] Calculating the between-class variance of each threshold in the threshold range;
[0019] Selecting a threshold that maximizes the inter-class variance as the optimal threshold;
[0020] The heat Figure 2 The image is binarized to obtain a binary image, and the area above the optimal threshold is marked as a potential temperature anomaly area.
[0021] As a preferred solution of the device health status monitoring method based on the acoustic-optical fusion technology of the present invention, the threshold range is as follows:
[0022] ;
[0023] in, m T is the average gray value, s T is the standard deviation, k is an adjustable parameter, t min is the minimum threshold, t max is the maximum threshold;
[0024] The inter-class variance of each threshold in the threshold range is calculated by the following formula:
[0025] s B 2 ( t ) = w 0 ( t ) w 1 ( t )[ m 0 ( t ) − m 1 ( t )] 2 ;
[0026] in, w 0 (t) is the probability of background class pixels, w 1 (t) is the probability of foreground class pixels, m 0 (t) is the average gray value of background pixels, m 1 (t) The average gray value of foreground pixels, is the between-class variance;
[0027] The optimal threshold is determined as follows:
[0028] ;
[0029] in, t* is the optimal threshold, s is the spatial scale, S is the total number of spatial scales, w s is the weight of each spatial scale s, s B2 (t,s) is the between-class variance calculated for threshold t at spatial scale s.
[0030] As a preferred solution of the device health status monitoring method based on the acoustic-optical fusion technology of the present invention, determining the thermal imaging feature includes the following steps:
[0031] Using a closing operation to fill the small holes inside the binary image and smooth the outer boundary of the potential temperature anomaly area;
[0032] The potential temperature anomaly areas of the same temperature anomaly source are combined to form a temperature anomaly area;
[0033] Calculating the area, average temperature and maximum temperature of the temperature anomaly region;
[0034] Filter out temperature anomaly areas smaller than a preset area threshold to obtain thermal imaging features;
[0035] The preset area threshold is determined according to a relative proportion of the total area of the heat map.
[0036] As a preferred solution of the device health status monitoring method based on the acoustic-optical fusion technology of the present invention, wherein: the second model includes a convolution layer, a temporal convolution layer and a multi-head attention mechanism;
[0037] The convolutional layer is used to process the spatial relationship between nodes, as shown in the following formula:
[0038] ;
[0039] in, For the l Layer Node i The hidden state of No. l+1 Layer Node i The hidden state of W s (l) For the l The structural weight matrix of the layer, h j (l) For the l Layer Node j The hidden state of W d (l) For the l The data weight matrix of the layer, N(i) For Node i The neighbor set of d i For Node i The degree, dj Neighbor node j degree, σ is the activation function, x j For Node j The eigenvector of
[0040] The temporal convolution layer is used to process the time series data of acoustic features, thermal imaging features and working condition data, as shown in the following formula:
[0041] ;
[0042] in, z i t For time t Time Node i The timing characteristics of x i (t-k:t) For Node i In the time window The original feature sequence in h i (t-k:t) For Node i In the time window The hidden state sequence within, TCN is a temporal convolutional layer.
[0043] As a preferred solution of the equipment health status monitoring method based on the acoustic-optical fusion technology of the present invention, the multi-head attention mechanism adaptively adjusts the importance weights of acoustic features, thermal imaging features and working condition data, as shown in the following formula:
[0044] a ij k = soft max j ( a k T [ W k h i || W k h j ] h i ′ = || k = 1 K s ( ∑ j ∈ N ( i ) a ij k W k h j ) ;
[0045] in, α ij k For Node j For Node i The attention weight, a k For the k The learnable vector of the attention heads, W k For the k The weight matrix of the attention head, h i ,h j For Node i and j The feature vector of , || represents the vector concatenation operation, h i ' For Node iThe new feature representation is K is the number of attention heads, s is the activation function, W k For the k The weight matrix of the attention head, S j∈N(i) For the node i All neighbors j Sum, h j Neighbor node j The characteristic vector of || k=1 K For the general K The outputs of the attention heads are concatenated together.
[0046] In a second aspect, an embodiment of the present invention provides an ultra-large-scale power grid production planning intelligent monitoring system for an electric power system, which includes:
[0047] A data acquisition module, used for collecting data based on a sensor array;
[0048] A feature extraction module is used to input the sound data into the first model to obtain acoustic features; analyze the thermal map and identify potential temperature anomaly areas, and determine thermal imaging features based on the potential temperature anomaly areas;
[0049] A status monitoring module is used to construct a second model based on the acoustic characteristics, the thermal imaging characteristics and the operating condition data, and to monitor the health status of the equipment through the second model.
[0050] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the processor executes the computer program, it implements any step of the above-mentioned device health status monitoring method based on acoustic-optical fusion technology.
[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the above-mentioned device health status monitoring method based on the acoustic-optical fusion technology is implemented.
[0052] The beneficial effect of the present invention is that the present invention more accurately reflects the actual health status of the equipment through multimodal feature fusion technology, greatly improving the accuracy and reliability of monitoring. In terms of acoustic feature extraction, the present invention effectively eliminates adverse factors such as environmental noise and extracts purer and more accurate acoustic information. This process cleverly combines the spatial distribution and time delay characteristics of the sensor, significantly enhances the accuracy of the acoustic features, and thus improves the sensitivity and monitoring ability of the entire system to equipment abnormalities. In terms of thermal imaging feature extraction, the present invention realizes the optimization and refinement of thermal imaging data, eliminates noise areas that have no practical significance, and retains temperature abnormality areas that are truly of diagnostic value. This feature extraction and optimization process greatly improves the validity of the data, making the final equipment health status monitoring more accurate and reliable. The application of the multi-head attention mechanism dynamically adjusts the importance of different data and components to ensure that the model can reasonably allocate weights according to actual conditions when processing multimodal data, thereby optimizing the monitoring effect. This method effectively avoids information overload or excessive influence of a certain data source on the monitoring results, and ensures the comprehensiveness and accuracy of the monitoring results. In general, this multimodal fusion technology provides a more comprehensive and accurate solution for equipment health monitoring, greatly improving the overall performance and reliability of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0054] Figure 1 The figure is a flow chart of the equipment health status monitoring method based on the acoustic-optical fusion technology.
[0055] Figure 2 This is a technical structure diagram of the equipment health status monitoring method based on the acoustic-optical fusion technology. DETAILED DESCRIPTION
[0056] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0058] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0059] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.
[0060] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.
[0061] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0062] Example 1
[0063] Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a device health status monitoring method based on the acoustic-optical fusion technology, comprising:
[0064] Multimodal sensor deployment and data collection aims to capture multi-dimensional information about the equipment's operating status.
[0065] First, highly sensitive patch-type sound sensor arrays are arranged at key locations of the equipment. These sensors use advanced piezoelectric or MEMS technology to accurately capture the audible frequency from 20Hz to the ultrasonic band above 100kHz, covering a variety of acoustic signals generated during the operation of the equipment.
[0066] At the same time, high-resolution infrared thermal imaging sensor modules are integrated. These sensors can capture the thermal distribution map of the equipment surface in real time with a resolution of up to 0.05°C, effectively monitoring abnormal hot spots and temperature change trends. In addition, three-axis acceleration sensors are deployed to monitor the vibration characteristics of the equipment. These sensors can capture tiny vibration changes with a frequency response range from 0.1Hz to 10kHz.
[0067] A series of working condition sensors are integrated, including Hall effect current sensors, high-precision voltage sensors, power analyzers and photoelectric speed sensors, which can monitor the electrical parameters and operating status of the equipment in real time. All these sensors are synchronously sampled through a high-speed data acquisition system with a sampling rate of up to 1MHz, ensuring the time consistency of the data.
[0068] In order to process such a large amount of data, edge computing technology is adopted to perform preliminary data compression and feature extraction locally, and then the processed data is transmitted to the central server via industrial Ethernet for further analysis. The preprocessing here is common knowledge in the field and conventional technical means are used, so the present invention will not go into details.
[0069] Furthermore, the collected sound signal is processed, and the abnormal sound of the device is located according to the processed sound signal.
[0070] Preferably, based on beamforming theory, a delay-sum beamforming algorithm is used to perform spatial filtering on sound signals collected by multiple patch sound sensor arrays, and a directional sound field model is formed by adjusting the time delay and amplitude weight of each patch sound sensor signal.
[0071] First, a basic delay-sum model is built to enhance the sound coming from a specific direction by time-aligning and summing the signals from different patch-type sound sensors. For example, if a bearing starts to make an unusual sound, this delay-sum model determines the approximate source of the sound, as shown in the following equation:
[0072] ;
[0073] In actual industrial environments, patch sound sensors at different locations are subject to different degrees of interference or noise. For example, if a patch sound sensor is close to a continuous noise source (such as a cooling fan), its weight is reduced to reduce the impact on abnormal sound monitoring, as shown in the following formula:
[0074] ;
[0075] Since different parts of the equipment emit abnormal sounds in different frequency ranges, for example, a gearbox failure produces high-frequency noise, while a bearing problem causes low-frequency vibration. The delay-sum model is converted to the frequency domain to better analyze the sound characteristics of different frequencies. By analyzing the sound in the frequency domain, different types of equipment abnormalities can be more accurately identified and located.
[0076] ;
[0077] Finally, considering that abnormal sounds come from different directions, the above model is modified into a model that can "scan" different directions to accurately locate faulty components in large equipment. At the same time, the output is normalized to eliminate the influence of signal strength and balance the contribution of different patch-type sound sensors. Even if different patch-type sound sensors are assigned different weights, the final output will not be disproportionately affected by the change in weight. It is expressed as follows:
[0078] ;
[0079] in, is the output of the first model, i is the target direction angle, f is the frequency, T is the temperature, M is the number of patch sound sensors, w i For the i The weight of a patch sound sensor, For the i The frequency domain representation of the signal received by a patch sound sensor, t i (i) Based on the target direction angle i Calculated i The time delay of a patch sound sensor.
[0080] For example, in a feasible embodiment, the first model is used to accurately identify and locate abnormal sounds of the device as follows:
[0081] The original time domain signal collected X i (t) Perform noise reduction and then convert the processed time domain signal into frequency domain representation X i (f) Here, noise reduction and frequency domain conversion are common knowledge in the art and are performed using conventional technical means, so the present invention will not elaborate on them.
[0082] Determine the frequency range that needs to be focused on and the angle range that needs to be scanned based on the characteristics of the device and known failure modes I, To cover all sound source positions around the device, and set the angular resolution, for example, to perform calculations every 5° or 10°.
[0083] For each scanning angle i , calculate the time delay of the sound wave from the assumed sound source position to each microphone t i (i) The initial weight of all patch sound sensors is 1, and the weight is adjusted later according to the location of the patch sound sensor or the environmental noise.
[0084] For every angle i and the frequency of each attention f The first model is used for calculation and the output value of the model is plotted into a two-dimensional heat map, with the horizontal axis being the angle and the vertical axis being the frequency. The high-intensity area in the two-dimensional heat map indicates the location and frequency of the sound source. A threshold is set and the area exceeding the threshold is found in the two-dimensional heat map. The beamforming technology is used to extract the sound in that direction from the original time domain signal. The direction of the identified abnormal sound source is corresponded to the physical structure of the equipment. The fault type is inferred based on the frequency characteristics of the abnormal sound and the known fault mode of the equipment.
[0085] Furthermore, the thermal images collected by the infrared thermal imaging sensor are deeply analyzed and the temperature abnormality areas are identified.
[0086] First, the original thermal map needs to be preprocessed, including denoising, contrast enhancement and geometric correction, to ensure image quality and spatial accuracy. The preprocessing here is common knowledge in the art and conventional technical means are used, so the present invention will not be described in detail.
[0087] Then, the Otsu threshold segmentation algorithm is applied to the preprocessed heat map. The Otsu method automatically determines the optimal threshold by maximizing the inter-class variance and divides the image into foreground (potential temperature anomaly area) and background. The specific steps are as follows:
[0088] Determine the threshold range based on the current operating status of the device;
[0089] Calculate the between-class variance for each threshold in the threshold range;
[0090] Select the threshold that maximizes the inter-class variance as the optimal threshold;
[0091] Will heat Figure 2 The image is binarized and the areas above the optimal threshold are marked as potential temperature anomaly areas.
[0092] Adaptively adjust the threshold range based on the current operating status of the device:
[0093] ;
[0094] Among them, μ T is the average gray value, σ T is the standard deviation, k is an adjustable parameter, t min is the minimum threshold, t max is the maximum threshold.
[0095] For each possible threshold in the threshold range t , calculate the between-class variance between foreground and background:
[0096] s B 2 ( t ) = w 0 ( t ) w 1 ( t )[ m 0 ( t ) − m 1 ( t )] 2 ;
[0097] in, w 0 (t) is the probability of background class pixels, w 1 (t) is the probability of foreground class pixels, m 0 (t) is the average gray value of background pixels, m 1 (t) The average gray value of foreground pixels, is the between-class variance.
[0098] The threshold that maximizes the inter-class variance is selected as the optimal threshold, and a multi-scale analysis is performed to capture large-scale temperature changes and local hot spots, as shown in the following formula:
[0099] ;
[0100] in, t* is the optimal threshold, s is the spatial scale, S is the total number of spatial scales, w s is the weight of each spatial scale s, s B 2 (t,s) is the threshold value at the spatial scale s t Compute the between-class variance.
[0101] Using this threshold, the heat Figure 2The binarized image is then binarized, where the area above the threshold is marked as a potential temperature anomaly area. Next, morphological operations are applied to the binarized image to combine the potential temperature anomaly areas of the same temperature anomaly source to form a temperature anomaly area; the area, average temperature and maximum temperature of the temperature anomaly area are calculated; and the temperature anomaly areas smaller than the preset area threshold are filtered out to obtain thermal imaging features.
[0102] The preset area threshold is determined according to the relative proportion of the total area of the heat map, and the preset area threshold is defined as follows:
[0103] Connected areas that are smaller than α% of the total area of the heat map are considered "too small" and need to be filtered. α is an adjustable parameter, usually set between 0.1% and 1%, and the specific value can be adjusted according to the application scenario and image resolution.
[0104] Furthermore, the output of the first model is used as the acoustic feature, the degree of temperature anomaly identified by the Otsu method is used as the thermal feature, and various operating condition information collected by the operating condition sensor is used to construct a graph neural network based on these data.
[0105] First, the various components of the equipment are regarded as nodes in the graph, and the physical connections or logical relationships between the components are edges. Each node contains the acoustic characteristics of the component (the output of the directional sound field model), thermal imaging characteristics (the degree of temperature anomaly identified by the Otsu method), and related working condition information (operating speed, load, ambient temperature and humidity, etc.). The graph convolution layer is used to capture the spatial relationship and information transfer between nodes, and the time convolution layer is introduced to process the time series data of each feature.
[0106] Preferably, the device is viewed as a complex graph structure G=(V,E), where the node set V represents the various components of the device and each node v i Contains the acoustic characteristics of the component f acoustici , Thermal imaging features f thermali and working condition characteristics f operationali The edge set E represents the physical connection or logical relationship between components and is determined according to the structure diagram of the device.
[0107] The second model is a multi-layer graph neural network including a graph convolution layer, a temporal convolution layer and a multi-head attention mechanism. The core of the second model of the present invention is the multi-head attention mechanism, which can adaptively adjust the weights of acoustic features, thermal imaging features and working condition data.
[0108] Specifically, the graph convolution layer is used to process the spatial relationship between nodes, allowing information to be transferred between connected components and capturing the mutual influence between components. It can be expressed by the following formula:
[0109] ;
[0110] in, For the l Layer Node i The hidden state of For the l+1 Layer Node i The hidden state of W s (l) For the l The structural weight matrix of the layer, h j (l) For the l Layer Node j The hidden state of W d (l) For the l The data weight matrix of the layer, N(i) For Node i The neighbor set of d i For Node i The degree, d j Neighbor node j degree, σ is the activation function, x j For Node j The feature vector of .
[0111] The temporal convolutional layer processes time series data using the TCN structure, allowing the model to identify patterns in device status changes over time, such as early signs of gradual failures, as shown in the following formula:
[0112] ;
[0113] in, z i t For time t Time Node i The timing characteristics of x i (t-k:t) For Node i In the time window The original feature sequence in h i (t-k:t) For Node i In the time window The hidden state sequence within, TCN is a temporal convolutional layer.
[0114] The multi-head attention mechanism enables the model to adaptively adjust the importance weights of different types of data and different components, which can be expressed as follows:
[0115] ;
[0116] in, α ij k For Node j For Node i The attention weight, a k For the k The learnable vector of the attention heads, W k For the k The weight matrix of the attention head, h i ,h j For Node i and j The feature vector of , || represents the vector concatenation operation, h i ' For Node i The new feature representation is K is the number of attention heads, s is the activation function, W k For the k The weight matrix of the attention head, S j∈N(i) For the node i All neighbors j Sum, h j Neighbor node j The characteristic vector of || k=1 K For the general K The outputs of the attention heads are concatenated together.
[0117] It should be noted that the first formula of the multi-head attention mechanism calculates the attention weight, which measures the k In the attention head, the neighbor nodes j For the current node i The second formula uses these weights to aggregate neighbor information. It first performs a weighted summation on each attention head separately, and then concatenates the results of all attention heads to form a node i New representation of .
[0118] Furthermore, an autoencoder structure is introduced into the graph neural network for abnormal monitoring and health status assessment. Specifically, the autoencoder structure compresses the input into a latent space representation, samples it through the reparameterization technique, and finally reconstructs the input. ||h-h'|| 2 and KL divergence, and monitor abnormal conditions that deviate from normal operating conditions. It can be expressed as follows:
[0119] ;
[0120] in, H (l) is the output of the last layer of graph neural network, W μ , W σ To generate the weight matrix for the mean and log variance, b μ ,b σ is the corresponding bias term, m is the mean vector output by the encoder, log σ 2 is the logarithmic variance vector of the encoder output, z is the potential representation obtained by sampling, e is random noise from a standard normal distribution.
[0121] In summary, this GNN-based fusion model cleverly integrates multi-dimensional heterogeneous data (including acoustics, thermal imaging and working condition information), uses graph structure to accurately represent the complex topological relationship and functional dependency between the various components of the equipment, and deeply captures the nonlinear interactions and dynamic influences between these components through multi-layer graph convolution and attention mechanism. At the same time, it combines variational autoencoders to achieve accurate modeling of normal operating conditions and sensitive monitoring of abnormal patterns. Finally, it comprehensively characterizes the dynamic evolution of the health status of the equipment in both time and space dimensions. It can not only timely identify potential faults and performance degradation trends, but also formulate differentiated maintenance strategies according to the importance of different components and the degree of fault impact. It provides engineers and decision makers with an integrated and intelligent predictive maintenance solution, significantly improves the accuracy, foresight and economic benefits of equipment management, and plays a key role in improving production efficiency, reducing maintenance costs and extending equipment life. It provides strong technical support and decision-making basis for the intelligent transformation and sustainable development of the modern industrial field.
[0122] Furthermore, this embodiment also provides an ultra-large-scale power grid production planning intelligent monitoring system for an electric power system, including:
[0123] A data acquisition module, used for collecting data based on a sensor array;
[0124] A feature extraction module is used to input the sound data into the first model to obtain acoustic features; analyze the thermal map and identify potential temperature anomaly areas, and determine thermal imaging features based on the potential temperature anomaly areas;
[0125] A status monitoring module is used to construct a second model based on the acoustic characteristics, the thermal imaging characteristics and the operating condition data, and to monitor the health status of the equipment through the second model.
[0126] This embodiment also provides a computer device, which is suitable for the device health status monitoring method based on the sound and light fusion technology, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the device health status monitoring method based on the sound and light fusion technology proposed in the above embodiment.
[0127] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.
[0128] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the device health status monitoring method based on the sound-light fusion technology proposed in the above embodiment is implemented.
[0129] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0130] Example 2
[0131] This is the second embodiment of the present invention. This embodiment provides a method for monitoring the health status of equipment based on the acoustic-optical fusion technology. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.
[0132] In order to verify the effectiveness of the equipment health status monitoring method based on the acoustic-optical fusion technology, this example selected a large centrifugal pump of a petrochemical plant as the test object. The centrifugal pump is a key production equipment, and its normal operation is crucial to the entire production process. The experimental process is as follows:
[0133] First, 10 highly sensitive patch-type sound sensors were installed at key locations of the centrifugal pump to form a sensor array. These sensors use advanced MEMS technology and are able to capture sound waves from 20Hz to 120kHz, covering a variety of acoustic signals generated during the operation of the centrifugal pump. At the same time, four high-resolution infrared thermal imaging sensors with a resolution of 0.05°C were installed on the surface of the pump body to monitor the surface temperature distribution of the pump body in real time. In addition, three triaxial acceleration sensors with a frequency response range of 0.1Hz to 15kHz were installed on the pump bearing seat to monitor vibration characteristics.
[0134] In order to fully monitor the working condition of the pump, Hall effect current sensors, high-precision voltage sensors, power analyzers and photoelectric speed sensors were installed. All sensors were synchronously sampled by a high-speed data acquisition system with a sampling rate set to 500kHz to ensure the time consistency of the data.
[0135] Edge computing devices are used for data preprocessing, including noise reduction and feature extraction, and then the processed data is transmitted to the central server via industrial Ethernet. On the server side, the improved delay-sum beamforming algorithm is first applied to process the sound signal. The algorithm takes into account the location of each sensor and the influence of environmental noise, and assigns different weights to different sensors. By analyzing the sound in the frequency domain and "scanning" different directions, the abnormal sound source is successfully identified and located.
[0136] For thermal imaging data, the improved Otsu threshold segmentation algorithm was applied. The temperature sensitivity parameter β=1.5 was introduced to enhance the monitoring capability of high temperature anomalies. Through multi-scale analysis (considering three spatial scales with weights of 0.3, 0.4, and 0.3 respectively), large-scale temperature changes and local hot spots were successfully captured.
[0137] Finally, a 7-layer graph neural network was constructed, treating the 9 main components of the centrifugal pump (such as impeller, bearing, seal, etc.) as nodes. The network contains 3 graph convolution layers, 2 temporal convolution layers, and 2 multi-head graph attention layers (8 attention heads per layer). In the autoencoder structure, the latent space dimension is set to 32.
[0138] The experiment lasted for 30 days, collecting normal operation data and data on various artificially introduced faults (such as bearing failure, impeller imbalance, seal leakage, etc.). The experimental results are shown in the following table:
[0139] Table 1 Experimental data table
[0140]
[0141] As shown in Table 1, by comparing the experimental results, it can be clearly seen that the present invention is significantly superior to the traditional method and the single technical method in multiple key indicators.
[0142] Fault monitoring rate: The fault monitoring rate of the present invention reaches 97.3%, which is 12.1 percentage points higher than the traditional method, and 5.8 and 7.5 percentage points higher than the single acoustic method and the single thermal imaging method, respectively. This shows that the acoustic-optical fusion technology can more comprehensively capture the abnormal state of the equipment and greatly improve the accuracy of monitoring.
[0143] False alarm rate: The false alarm rate of the present invention is only 2.8%, which is much lower than other methods. This result fully demonstrates the advantages of multimodal data fusion, which effectively reduces the misjudgment caused by a single data source by comprehensively analyzing acoustic, thermal imaging and working condition data.
[0144] Average monitoring time: The present invention shortens the average fault monitoring time to 6.8 minutes, which is nearly 12 minutes faster than the traditional method and 5.5-8.9 minutes faster than the single technology method. This significant time advantage is crucial for preventing faults.
[0145] Positioning accuracy: 94.7% positioning accuracy proves the effectiveness of the improved delay-sum beamforming algorithm. Accurate fault location can significantly reduce maintenance time and cost.
[0146] Early fault recognition rate: The present invention performs particularly well in early fault recognition, reaching an 89.5% recognition rate. This is 27.2 percentage points higher than the traditional method, fully demonstrating the advantage of graph neural networks in capturing small changes in equipment status.
[0147] System response time: The system response time of 320ms demonstrates the advantages of the present invention in real-time performance, which is due to the application of edge computing technology and efficient data processing algorithms.
[0148] In general, the present invention achieves comprehensive, accurate and real-time monitoring of the health status of large centrifugal pumps through multimodal sensor deployment, innovative signal processing algorithms and deep learning models. In particular, its outstanding performance in early fault identification and precise positioning provides strong technical support for predictive maintenance. This can not only significantly reduce the unplanned downtime of equipment, but also optimize maintenance strategies and extend the life of equipment, thus bringing considerable economic benefits to enterprises.
[0149] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A device health status monitoring method based on acoustic-optical fusion technology, characterized in that: include, Collecting data based on a sensor array; the data includes sound data, thermal images and operating condition data; Inputting the sound data into a first model to obtain acoustic features; Analyzing the thermal map and identifying potential temperature anomaly areas, and determining thermal imaging features based on the potential temperature anomaly areas; constructing a second model according to the acoustic features, the thermal imaging features and the operating condition data, and performing equipment health status monitoring through the second model; The sensor array includes a patch-type sound sensor, an infrared thermal imaging sensor and an operating condition sensor; The acoustic feature is obtained by spatially filtering the sound data through the first model and adjusting the time delay and amplitude weight of the sensor array signal, as shown in the following formula: ; in, is the output of the first model, θ is the target direction angle, f is the frequency, T is the temperature, M is the number of patch sound sensors, w i For the i The weight of a patch sound sensor, For the i The frequency domain representation of the signal received by a patch sound sensor, τ i (θ) Based on the target direction angle θ Calculated i The time delay of a patch sound sensor; Analyzing the thermal map and identifying potential temperature anomaly areas includes the following steps: Determine the threshold range based on the current operating status of the device; Calculating the between-class variance of each threshold in the threshold range; Selecting a threshold that maximizes the inter-class variance as the optimal threshold; Binarizing the thermal map to obtain a binary image, and marking areas above the optimal threshold as potential temperature anomaly areas; The threshold range is shown in the following formula: ; in, μ T is the average gray value, σ T is the standard deviation, k is an adjustable parameter, t min is the minimum threshold, t max is the maximum threshold; The inter-class variance of each threshold in the threshold range is calculated by the following formula: ; in, w 0 (t) is the probability of background class pixels, w 1 (t) is the probability of foreground class pixels, μ 0 (t) is the average gray value of background pixels, μ 1 (t) The average gray value of foreground pixels, is the between-class variance; The determination of the optimal threshold is shown in the following formula: ; in, t* is the optimal threshold, s is the spatial scale, S is the total number of spatial scales, w s is the weight of each spatial scale s, σ B 2 (t,s) is the threshold value at the spatial scale s t Compute the between-class variance.
2. The device health status monitoring method based on the acoustic-optical fusion technology according to claim 1 is characterized in that: Determining the thermal imaging feature comprises the following steps: Using a closing operation to fill the small holes inside the binary image and smooth the outer boundary of the potential temperature anomaly area; The potential temperature anomaly areas of the same temperature anomaly source are combined to form a temperature anomaly area; Calculating the area, average temperature and maximum temperature of the temperature anomaly region; Filter out temperature anomaly areas smaller than a preset area threshold to obtain thermal imaging features; The preset area threshold is determined according to a relative proportion of the total area of the heat map.
3. The device health status monitoring method based on the acoustic-optical fusion technology according to claim 2 is characterized in that: The second model includes a convolutional layer, a temporal convolutional layer, and a multi-head attention mechanism; The convolutional layer is used to process the spatial relationship between nodes, as shown in the following formula: ; in, For the l Layer Node i The hidden state of For the l+1 Layer Node i The hidden state of W s (l) For the l The structural weight matrix of the layer, h j (l) For the l Layer Node j The hidden state of W d (l) For the l The data weight matrix of the layer, N(i) For Node i The neighbor set of d i For Node i The degree, d j Neighbor node j degree, σ is the activation function, x j For Node j The eigenvector of The temporal convolution layer is used to process the time series data of acoustic features, thermal imaging features and working condition data, as shown in the following formula: ; in, z i t For time t Time Node i The timing characteristics of x i (t-k:t) For Node i In the time window The original feature sequence in h i (t-k:t) For Node i In the time window The hidden state sequence within, TCN is a temporal convolutional layer.
4. The device health status monitoring method based on the acoustic-optical fusion technology according to claim 3 is characterized in that: The multi-head attention mechanism adaptively adjusts the importance weights of acoustic features, thermal imaging features, and working condition data, as shown in the following formula: ; in, α ij k For Node j For Node i The attention weight, a k For the k The learnable vector of the attention heads, W k For the k The weight matrix of the attention head, h i ,h j For Node i and j The feature vector of , || represents the vector concatenation operation, h i ' For Node i The new feature representation is K is the number of attention heads, σ is the activation function, W k For the k The weight matrix of the attention head, Σ j∈N(i) For the node i All neighbors j Sum, h j Neighbor node j The characteristic vector of || k=1 K For the general K The outputs of the attention heads are concatenated together.
5. A device health status monitoring system based on acoustic-optical fusion technology, based on the device health status monitoring method based on acoustic-optical fusion technology according to any one of claims 1 to 4, characterized in that: include, A data acquisition module, used for collecting data based on a sensor array; A feature extraction module, used for inputting sound data into the first model to obtain acoustic features; Analyzing the thermal map and identifying potential temperature anomaly areas, and determining thermal imaging features based on the potential temperature anomaly areas; A status monitoring module is used to construct a second model based on the acoustic characteristics, the thermal imaging characteristics and the operating condition data, and to monitor the health status of the equipment through the second model.
6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the device health status monitoring method based on the acoustic-optical fusion technology described in any one of claims 1 to 4 are implemented.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the device health status monitoring method based on the acoustic-optical fusion technology described in any one of claims 1 to 4 are implemented.
