A device and method for diagnosing defects in electric power equipment based on multi-source data fusion
Through the power equipment defect diagnosis device fusion with multi-source data, deep learning technology is used to fuse sound, visible light and infrared thermal image data, and the missed detection and false detection problems caused by single source dependence in the existing technology are solved, achieving high accuracy and stable diagnosis of power equipment defects.
Patent Information
- Application Number
- CN202011411155.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-03
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-12-03
AI Technical Summary
Existing power equipment defect diagnosis devices rely on a single source data, resulting in high leakage detection and error detection rates, poor robustness, unable to fully utilize the connection between multiple source data, and it is difficult to detect and characterize imprecise defects in the early stage.
Using a power equipment defect diagnosis device based on multi-source data fusion, combined with a data acquisition module and an artificial intelligence processing module, a convolutional neural network designed by deep learning and a fully connected neural network are fused to generate sound, visible light images and infrared thermal image data. Decision-level fusion diagnostic results include visible-infrared image diagnostic subnetwork, sound diagnostic subnetwork and decision-making fusion network.
Improves diagnostic accuracy, can provide accurate diagnosis in early or insignificant defects, has high stability, can maintain high diagnostic accuracy in the absence or distortion of source data, and is suitable for complex sound field environments .
Smart Images

Figure CN112419301B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of power equipment diagnosis, and in particular, to a power equipment defect diagnosis device based on multi-source data fusion and a power equipment defect diagnosis method based on multi-source data fusion. Background Art
[0002] Power equipment is the hub and channel in the transmission and distribution network. During the use of equipment, there will be aging, disrepair, hidden faults, etc. Therefore, regular defect diagnosis of power equipment can early identify hidden dangers of safe operation of each equipment and effectively ensure the normal operation of the transmission and distribution network.
[0003] When the power equipment is running, it will be accompanied by different degrees of heating and sound conditions, and abnormal sound and heating conditions are the most obvious signs of power equipment defects. Therefore, based on the heating and sound conditions of the power equipment when it is working, the defects of each device can be diagnosed.
[0004] In the prior art, most of the power equipment defect diagnosis devices perform defect diagnosis based on a single source of data, such as sound-based defect diagnosis and infrared thermal imaging-based defect diagnosis. If you want to jointly diagnose equipment defects through multiple sources, you usually diagnose the equipment through a single source first, obtain the diagnostic information of this type of source, and then perform a comprehensive analysis to obtain the final diagnosis result. These devices cannot fully consider the connection between multiple source data when the power equipment is working. Therefore, the aforementioned diagnostic devices are prone to false detection and missed detection of early or unclear equipment defects.
[0005] Moreover, these devices have poor robustness and are greatly affected by the surrounding working environment. Summary of the invention
[0006] The purpose of this application is to overcome the problems in the existing power equipment defect diagnosis technology, such as reliance on a single data source, high defect missed detection rate and false detection rate, and poor robustness, and to provide a power equipment defect diagnosis device and method based on multi-source data fusion.
[0007] The technical solution of the first aspect of the present application is: providing an electric power equipment defect diagnosis device based on multi-source data fusion, the device comprising at least a data acquisition module and an artificial intelligence processing module, the data acquisition module being used to acquire first multi-source data of the electric power equipment, the artificial intelligence processing module comprising at least: a data preprocessing module, a defect diagnosis module; the data preprocessing module being used to preprocess the acquired first multi-source data, and generate second multi-source data of the electric power equipment in a specified format, wherein the second multi-source data of the electric power equipment in the specified format comprises at least: an acoustic image Simg, a second visible light image Vimg, a second infrared thermal image Iimg and a sound feature vector Smfcc; the defect diagnosis module being used to construct an electric power equipment defect diagnosis model, and generating a decision-level fusion diagnosis result according to the second multi-source data of the electric power equipment in the specified format, wherein the decision-level fusion diagnosis result is generated by fusion of the visible-infrared image diagnosis result VI-result and the sound source defect diagnosis result S-result.
[0008] In any of the above technical solutions, further, the power equipment defect diagnosis model includes at least: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision fusion network.
[0009] In any of the above technical solutions, further, a squeeze excitation SE module and a DW convolution module are provided in the visible light image-infrared thermal image defect diagnosis subnetwork, and the defect diagnosis module generates a calculation process of the visible-infrared image diagnosis result VI-result, which specifically includes: using the squeeze excitation SE module and the DW convolution module to calculate the visible light feature map FM-v3 and the visible light feature map FM-v5 of the second visible light image Vimg, and the infrared feature map FM-i3 and the infrared feature map FM-i5 of the second infrared thermal image Iimg; based on the position information in the visible light feature map FM-v3 and the visible light feature map FM-v5, respectively, the temperature information in the infrared thermal feature map FM-i3 and the infrared thermal feature map FM-i5 is non-uniformly extracted to generate a fused feature map output_1 and a fused feature map output_2; according to the fused feature map output_1 and the fused feature map output_2, a non-maximum suppression NMS operation is performed to generate a visible-infrared image diagnosis result VI-result.
[0010] In any of the above technical solutions, further, the defect diagnosis module generates a calculation process for generating a sound source defect diagnosis result S-result, specifically including: performing 4 layers of DW convolution and pooling operations according to the acoustic image Simg and the sound feature vector Smfcc, and flattening the obtained acoustic image feature map of the acoustic image Simg and the sound feature vector feature map of the sound feature vector Smfcc; determining the nonlinear relationship between the features through 2 layers of full connection operations; and outputting the prediction result after processing by the softmax layer to generate the sound source defect diagnosis result S-result.
[0011] In any of the above technical solutions, further, the first multi-source data includes: sound data, a first visible light image and a first infrared thermal image, and the defect diagnosis module is also used to: mark the decision-level fusion diagnosis result in the second visible light image Vimg.
[0012] The technical solution of the second aspect of the present application is: a method for diagnosing defects in power equipment based on multi-source data fusion is provided, the method comprising: step 1, preprocessing the collected first multi-source data to generate second multi-source data of power equipment in a specified format, wherein the second multi-source data of power equipment in the specified format at least includes: an acoustic image Simg, a second visible light image Vimg, a second infrared thermal image Iimg and a sound feature vector Smfcc; step 2, constructing a power equipment defect diagnosis model, and generating a decision-level fusion diagnosis result based on the second multi-source data of power equipment in the specified format, wherein the decision-level fusion diagnosis result is generated by fusing the visible-infrared image diagnosis result VI-result with the sound source defect diagnosis result S-result.
[0013] In any of the above technical solutions, further, the power equipment defect diagnosis model includes at least: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision fusion network.
[0014] In any of the above technical solutions, further, a squeeze excitation SE module and a DW convolution module are provided in the visible light image-infrared thermal image defect diagnosis subnetwork. In step 2, the calculation process of the visible-infrared image diagnosis result VI-result specifically includes: step 21, using the squeeze excitation SE module and the DW convolution module to calculate the visible light feature map FM-v3 and the visible light feature map FM-v5 of the second visible light image Vimg, and the infrared feature map FM-i3 and the infrared feature map FM-i5 of the second infrared thermal image Iimg; step 22, based on the position information in the visible light feature map FM-v3 and the visible light feature map FM-v5, respectively, the temperature information in the infrared thermal feature map FM-i3 and the infrared thermal feature map FM-i5 is non-uniformly extracted to generate a fused feature map output_1 and a fused feature map output_2; step 23, according to the fused feature map output_1 and the fused feature map output_2, a non-maximum suppression NMS operation is performed to generate a visible-infrared image diagnosis result VI-result.
[0015] In any of the above technical solutions, further, in step 2, the calculation process of the sound source defect diagnosis result S-result specifically includes: performing 4 layers of DW convolution and pooling operations according to the acoustic image Simg and the sound feature vector Smfcc, and flattening the obtained acoustic image feature map of the acoustic image Simg and the sound feature vector feature map of the sound feature vector Smfcc; determining the nonlinear relationship between the features through 2 layers of full connection operations; and outputting the prediction result after processing by the softmax layer to generate the sound source defect diagnosis result S-result.
[0016] In any of the above technical solutions, further, the first multi-source data includes: sound data, a first visible light image and a first infrared thermal image, and after step 2, it also includes: marking the decision-level fusion diagnosis result in the second visible light image Vimg.
[0017] The beneficial effects of this application are:
[0018] The present invention discloses a defect diagnosis device and method based on multi-source data fusion. The device can simultaneously collect first multi-source data of power equipment, including sound data, infrared thermal images and visible light images. The device performs defect diagnosis on power equipment based on the collected first multi-source data, and the diagnosis effect is good and can meet the requirements of practical applications.
[0019] Compared with the currently common power equipment defect diagnosis devices, such as the power equipment defect diagnosis device based on sound and the power equipment defect diagnosis device based on infrared thermal images, the advantages of the present invention are mainly reflected in the following aspects:
[0020] 1. Realize multi-source data fusion. The defect diagnosis model in the present invention is designed based on convolutional neural networks and fully connected neural networks in deep learning, and fuses sound, visible light images and infrared thermal images, providing a method for data fusion of sound data, visible light images and infrared thermal images.
[0021] 2. The defect diagnosis device based on multi-source data fusion has a high diagnostic accuracy. It combines the three source data to jointly obtain the diagnostic results. It has a high diagnostic accuracy for early defects or defects with unclear characteristics, and is suitable for actual diagnostic tasks.
[0022] 3. The defect diagnosis device based on multi-source data fusion has good stability and can still guarantee relatively accurate diagnosis results in the case of missing or serious data distortion of certain source data. For example, if the equipment to be diagnosed is in a complex sound field with multiple sound sources and high reverberation, and the sound data has large noise, this device can still diagnose equipment defects based on visible light images and infrared thermal images.
[0023] 4. Multi-source data and single-source data acquisition. This device is based on a microphone array, a visible light image acquisition module and an infrared thermal image acquisition module. For target power equipment, it can not only collect multiple source data of the target equipment, but also collect any one or two of the three source data separately as required. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:
[0025] Figure 1 is a schematic diagram of a power equipment defect diagnosis device based on multi-source data fusion according to an embodiment of the present application;
[0026] Figure 2 is a schematic diagram of a visible light image-infrared thermal defect diagnosis subnetwork model structure according to an embodiment of the present application;
[0027] Figure 3 is a schematic diagram of a sound defect diagnosis subnetwork model structure according to an embodiment of the present application;
[0028] Figure 4 It is a flowchart of model training and testing according to an embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0030] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.
[0031] Embodiment 1:
[0032] like Figure 1 As shown, this embodiment provides a power equipment defect diagnosis device based on multi-source data fusion, which can perform a first multi-source data collection and data fusion on the power equipment to complete the task of diagnosing the defects of the power equipment.
[0033] The data acquisition module in the device is connected to the data transmission module and the artificial intelligence processing module respectively. The data acquisition module transmits the collected first multi-source data to the artificial intelligence processing module for data preprocessing and defect diagnosis, and at the same time, transmits it to the data transmission module for data download and backup.
[0034] The data acquisition module is provided with a variety of acquisition devices, including at least: a microphone array sensor, a visible light image acquisition module, and an infrared thermal image acquisition module. Accordingly, the first multi-source data collected includes at least: the sound of the power equipment (sound data), the first visible light image, and the first infrared thermal image.
[0035] The artificial intelligence processing module is mainly used to perform equipment defect diagnosis on the first multi-source data collected by the data acquisition module. The module includes at least: a data preprocessing module and a defect diagnosis module, wherein the data preprocessing module performs data preprocessing operations on the collected first multi-source data, and the defect diagnosis module performs equipment defect diagnosis on the second multi-source data in a specified format after preprocessing, and transmits the diagnosis results to the visualization and alarm module so as to visualize the diagnosis results.
[0036] In this embodiment, the data preprocessing module performs preprocessing operations on the collected sound data, the first visible light image, and the first infrared thermal image, and the preprocessing operations at least include: denoising, image enhancement, image alignment, and extraction of MFCC (Mel-frequency cepstral coefficient) feature parameters of the sound signal. The aligned second visible light image Vimg, the second infrared thermal image Iimg, the acoustic image Simg, and the extracted sound feature vector Smfcc are obtained through the preprocessing operation, wherein the acoustic image Simg displays the distribution of the sound source in space in the form of an image, and is displayed in the form of a color cloud map, which can display the location of the sound source, and the higher the sound pressure, the brighter the color at the corresponding position.
[0037] The defect diagnosis module includes computing hardware and a deep learning-based defect diagnosis method running in the computing hardware. Specifically, the computing hardware is an artificial intelligence computing module that can provide powerful computing power for the edge system, run modern neural networks in parallel, and process high-resolution data from multiple sensors at the same time. In the defect diagnosis method based on deep learning, a multi-source data fusion defect diagnosis model based on deep learning is mainly proposed. The model can be used to diagnose defects in power equipment, and at least includes: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork, and a decision fusion network. The defect diagnosis model fully integrates the sound of the working power equipment, the first infrared thermal image, and the first visible light image information, and performs equipment defect diagnosis based on the first multi-source data collected by the above-mentioned data acquisition module.
[0038] The visualization and alarm module mainly includes a touch screen, an alarm buzzer, and corresponding peripheral circuits. The touch screen can be used to visually operate the equipment defect diagnosis device. When performing defect diagnosis, the touch screen can display the second visible light image Vimg marking the diagnosis result in real time and generate an equipment diagnosis report. At the same time, when the equipment is diagnosed to be abnormal, the alarm buzzer is triggered to alert the maintenance personnel to perform timely maintenance on the equipment diagnosed to be defective.
[0039] The data transmission module in this embodiment includes a USB transmission module and a serial communication module. The above-mentioned defect diagnosis model can be updated and the device diagnosis report can be downloaded through the USB transmission module.
[0040] Furthermore, a battery management module may be provided in the above device, and the battery management module uses a lithium battery power supply module to power the device to meet the power supply requirements of various components of the device. The battery management module provides charge and discharge protection and overload protection functions, which can extend the service life of the lithium battery power supply module. At the same time, the lithium battery power supply module is detachable, and a backup battery can be replaced for heavy diagnostic tasks to ensure long-term diagnostic operations.
[0041] Embodiment 2:
[0042] On the basis of the above embodiment, this embodiment further provides a method for diagnosing defects in electric power equipment based on multi-source data fusion, the method comprising:
[0043] S1 collects the first multi-source data under the working state of the power equipment. Specifically, after starting the defect diagnosis task, the first multi-source data under the working state of the power equipment is collected by using the microphone array sensor, the visible light image acquisition module and the infrared thermal image acquisition module, and is saved in the first multi-source data buffer, wherein the first multi-source data includes sound data, the first visible light image and the first infrared thermal image.
[0044] S2 performs preprocessing operations on the first multi-source data to generate an acoustic image Simg, a second visible light image Vimg, and a second infrared thermal image Iimg, and extracts feature parameters from the sound data to obtain a sound feature vector Smfcc, i.e., the second multi-source data in a specified format. The specific process is as follows:
[0045] S2.1 uses beamforming technology based on sound data to obtain the acoustic image Simg of the power equipment.
[0046] Specifically, the sound signals collected by each array element in the microphone array sensor are weighted and summed to form a directional beam. The directional beam is guided by searching for the possible location of the sound source, and the weight is modified to maximize the output signal (sound signal) power of each array element. The position distribution of the sound source in space is displayed in the form of an image. The image is displayed in the form of a color cloud map, which will show the location of the sound source. The higher the sound pressure, the brighter the color. This picture is the acoustic image Simg obtained based on the sound data.
[0047] S2.2 uses the Meshflow image alignment and calibration method to divide the acoustic image Simg, the first visible light image, and the first infrared thermal image into multiple small grids of equal size, calculate the feature points in each small grid, and determine the image alignment matrix based on the calculated feature points to align the acoustic image Simg, the first visible light image, and the first infrared thermal image. Finally, the aligned images are resized to obtain the acoustic image Simg, the second visible light image Vimg, and the second infrared thermal image Iimg, all of which are 416×416 in size.
[0048] S2.3 extracts MFCC feature parameters from the sound data to obtain a sound feature vector Smfcc.
[0049] S3 inputs the preprocessed, specified-format second multi-source data into a multi-source data fusion defect diagnosis model based on deep learning, performs data fusion and defect diagnosis on the acoustic image Simg, the second visible light image Vimg, the second infrared thermal image Iimg and the sound feature vector Smfcc, and obtains the final diagnosis result to achieve equipment defect diagnosis, wherein the multi-source data fusion defect diagnosis model based on deep learning includes at least: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision-level fusion layer.
[0050] Specifically, the equipment defect diagnosis in this embodiment mainly adopts a multi-source data fusion defect diagnosis model based on deep learning. Based on the second multi-source data of the power equipment in a specified format, the back propagation algorithm is used in the defect diagnosis model training process to adjust the weights and thresholds of the defect diagnosis model to obtain a defect diagnosis model that meets the practical diagnosis requirements of the power equipment.
[0051] Among them, the visible light image-infrared thermal image defect diagnosis subnetwork uses the feature map extracted from the second visible light image Vimg as a mask, and non-uniformly extracts the temperature features of the device location on the second infrared thermal image Iimg to achieve feature-level fusion of visible light images and infrared thermal images.
[0052] The sound defect diagnosis subnetwork uses the aligned acoustic image Simg and the second visible light image Vimg to display the final result of the diagnosis based on the acoustic image Simg on the second visible light image Vimg.
[0053] S4 visualizes the diagnosis results of the multi-source data fusion defect diagnosis model based on deep learning and generates an equipment diagnosis report. Specifically, the equipment defect category is marked in the second visible light image Vimg, and the diagnosis results are visualized on the touch screen, and the equipment diagnosis report is generated. If an equipment defect is diagnosed, an alarm signal is immediately generated to trigger a buzzer alarm and end the diagnosis task.
[0054] Embodiment three:
[0055] Based on the above embodiment, this embodiment shows a method for diagnosing defects in electric power equipment based on multi-source data fusion, the method comprising:
[0056] Step 1, preprocess the collected first multi-source data to generate second multi-source data of the power equipment in a specified format, wherein the second multi-source data of the power equipment in a specified format includes at least: an acoustic image Simg, a second visible light image Vimg, a second infrared thermal image Iimg and a sound feature vector Smfcc.
[0057] On the basis of the above embodiment, the first multi-source data of each power equipment when working, including sound data, first visible light image and first infrared thermal image, is collected through microphone array sensor, visible light image acquisition module, infrared thermal image acquisition module and other devices. The above first multi-source data is preprocessed to obtain the aligned second visible light image Vimg, acoustic image Simg, second infrared thermal image Iimg and sound feature vector Smfcc, that is, the second multi-source data in the specified format.
[0058] According to the settings, a total of 3,200 first multi-source data of power equipment were collected, among which the power equipment involved included three types: transformers, high-voltage circuit breakers and capacitors. At the same time, it covered the first multi-source data of multiple categories of power equipment in multiple scenarios.
[0059] Step 2, construct a power equipment defect diagnosis model based on the second multi-source data of the power equipment in the specified format, wherein the power equipment defect diagnosis model includes: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision fusion network, and the multi-source data fusion method adopted by the decision fusion network is: a combination of feature-level fusion and decision-level fusion. Based on the visible light image-infrared thermal image defect diagnosis subnetwork, the feature fusion of the visible light source and the infrared thermal source is realized, and then based on the sound defect diagnosis subnetwork, the diagnosis result of the sound source is obtained, and then the feature fusion result and the diagnosis result are fused by decision to obtain the final diagnosis result after the fusion of multi-source data.
[0060] This embodiment shows a calculation process of a visible-infrared image diagnosis result VI-result, which specifically includes:
[0061] Step 21, using the squeeze excitation SE module and the DW convolution module, calculate the visible light feature map FM-v3 and the visible light feature map FM-v5 of the second visible light image Vimg, and the infrared feature map FM-i3 and the infrared feature map FM-i5 of the second infrared thermal image Iimg;
[0062] Specifically, the preprocessed second visible light image Vimg and the second infrared thermal image Iimg are input into the visible light image-infrared thermal image defect diagnosis subnetwork to obtain a defect diagnosis result based on the fusion of visible light image and infrared thermal image features.
[0063] This sub-network is based on the improved lightweight YOLOv3 Tiny network design. On the basis of the original network, the squeeze excitation SE module is introduced to replace some convolution modules in the original network to realize the channel-level attention mechanism. At the same time, the feature map extracted from the second visible light image Vimg is used as a mask to non-uniformly extract the temperature features of the equipment location on the second infrared thermal image Iimg to realize feature-level data fusion. Based on this network, the type of equipment defects and their locations can be accurately identified.
[0064] Specifically, Figure 2 As shown in FIG. 1 , it is a model structure diagram of the visible light image-infrared thermal image defect diagnosis subnetwork. It should be noted that the colorless rectangular boxes in the figure are the extracted infrared thermal and visible light feature maps, and the gray rectangular boxes are the convolution, pooling and other operations performed on the feature maps.
[0065] Firstly, the second infrared thermal image Iimg and the second visible light image Vimg with a size of 416×416×3 are input into the network. After 4 layers of convolution and SE module processing, the infrared thermal feature map FM-i1 and the visible light feature map FM-v1 with a size of 26*26*256 are obtained. After 3 layers of convolution, the infrared thermal feature map FM-i2 and the visible light feature map FM-v2 with a size of 13*13*256 are obtained.
[0066] After that, the visible light feature map FM-v2 is upsampled and concatenated with the previous visible light feature map FM-v1 to obtain a visible light feature map FM-v4 with a size of 26*26*384 after fusion of different scales. The same operation is performed on the infrared thermal feature map FM-i2 to obtain the infrared thermal feature map FM-i4.
[0067] Then, the visible light feature maps FM-v2 and FM-v4 are subjected to two convolution and pooling operations respectively to obtain the visible light feature map FM-v3 with a size of 13*13*256 and the visible light feature map FM-v5 with a size of 26*26*256. The same operation is performed on the infrared thermal feature maps FM-i2 and FM-i4 to obtain the infrared thermal feature maps FM-i3 and FM-i5.
[0068] Step 22, performing non-uniform extraction on the temperature information in the infrared thermal feature map FM-i3 and the infrared thermal feature map FM-i5 based on the position information in the visible light feature map FM-v3 and the visible light feature map FM-v5, respectively, to generate a fused feature map output_1 and a fused feature map output_2;
[0069] Step 23, performing a non-maximum suppression NMS operation according to the fused feature map output_1 and the fused feature map output_2 to generate a visible-infrared image diagnosis result VI-result.
[0070] Specifically, at two scales with downsampling multiples of 16 and 32, the visible light feature maps FM-v3 and FM-v5 are multiplied with the infrared feature maps FM-i3 and FM-i5 at corresponding positions. Based on the position information of the visible light feature maps FM-v3 and FM-v5, the temperature information of the infrared thermal feature maps FM-i3 and FM-i5 are non-uniformly extracted to obtain fused feature maps output_1 and output_2 with sizes of 13*13*255 and 26*26*255, respectively. Among them, the fused feature map of 13*13*255 is used to predict large-size targets, while the fused feature map of 26*26*255 is used to predict small-size targets.
[0071] A non-maximum suppression NMS operation is performed based on the output fusion feature maps output_1 and output_2 to obtain the final diagnosis result VI-result based on the second visible light image Vimg and the second infrared thermal image Iimg.
[0072] In this embodiment, the category labels of the network diagnosis results are set to 6 categories, including normal transformer, abnormal transformer, normal capacitor, abnormal capacitor, normal high-voltage circuit breaker and abnormal high-voltage circuit breaker. In this embodiment, the number of channels of the feature map corresponds to the number of convolution kernels. Based on the above setting of 6 categories of category labels, combined with the characteristics of the yolo network, the number of convolution kernels in the network structure is determined, that is, the number of channels of the feature maps FM-v3, FM-i3, FM-v5 and FM-i5.
[0073] Therefore, the calculation formula for the number of channels of feature maps FM-v3, FM-i3, FM-v5 and FM-i5 is:
[0074] n channels =(1+4+n class )×2=(1+4+6)×2=22
[0075] Where n channels is the number of feature map channels, n class The number of categories of the predicted label. In the 22-dimensional channel of the feature map, 1 dimension is used to predict whether the grid contains an object, 4 dimensions are used to represent the coordinates of the predicted target box, and 6 dimensions represent the confidence of the target box corresponding to the 6 categories of labels.
[0076] At the same time, since defect diagnosis is performed at two downsampling scales of 16 times and 32 times, the number of convolution kernels needs to be multiplied by 2.
[0077] The squeeze excitation SE module in this embodiment first performs a squeeze operation to extract global features based on the input feature map to obtain a global feature map, and the squeeze method is global mean pooling. Based on the amount of information of each channel in the global feature map, the importance of each channel is learned. Then a squeeze operation is performed on the extracted global feature map, and a gate mechanism is implemented with a Sigmoid function to obtain the nonlinear relationship between each channel. Based on the SE module, the importance of each channel and the nonlinear relationship between channels are learned to realize a channel-level attention mechanism.
[0078] In the figure, the visible light feature map (FM-v5, FM-v3) and infrared feature map (FM-i5, FM-i3) are obtained at two scales with downsampling multiples of 16 and 32. Compared with the infrared feature map, the visible light feature map contains richer location information. Therefore, the diagnostic subnetwork adopts non-uniform infrared feature extraction to fuse the infrared thermal feature map and the visible light feature map. That is, the visible light feature map is used as a mask, such as Figure 2The visible light feature maps FM-v3 and FM-v5 in the figure contain the location information of the equipment. The closer to the location of the equipment defect, the larger the pixel value of the visible light feature map, and the brighter the location is when displayed. Multiply the visible light feature maps FM-v3 and FM-v5 with the corresponding positions of the infrared thermal feature maps FM-i3 and FM-i5 respectively, and use the pixel values of the visible light feature maps FM-v3 and FM-v5 as weight coefficients. The closer to the location of the equipment defect, the larger the weight. Perform non-uniform extraction of temperature information at the same position on the infrared feature map to obtain a fused feature map.
[0079] Based on the non-uniform infrared feature extraction method proposed in this method, we can focus on the temperature characteristics of the device location and ignore the influence of the background temperature characteristics. Thus, we can obtain a fusion feature map rich in device location information and temperature information, which can effectively improve the detection accuracy of the model.
[0080] This embodiment also shows a calculation process of a sound source defect diagnosis result S-result, which specifically includes: inputting the preprocessed acoustic image Simg and the sound feature vector Smfcc into a sound defect diagnosis subnetwork to obtain a defect diagnosis result based on the sound data.
[0081] Specifically, Figure 3 As shown in FIG. 1 , it is a model structure diagram of the sound defect diagnosis sub-network. It should be noted that the colorless rectangular box in the figure is the generated feature map, and the gray rectangular box is the convolution, pooling and other operations performed on the feature map.
[0082] The acoustic image Simg and sound feature vector Smfcc of size 416*416*3 are input into the sound defect diagnosis subnetwork. After 4 layers of DW convolution and pooling operations, the obtained acoustic image Simg feature map and sound feature vector Smfcc feature map are flattened and concatenated into a one-dimensional feature vector. Then, they are subjected to 2 layers of full connection operations to determine the nonlinear relationship between features. Finally, the prediction result is output after processing by the softmax layer to obtain the sound-based sound source defect diagnosis result S-result.
[0083] The sound defect diagnosis subnetwork uses depthwise and pointwise convolution based on the deep separable convolution (DW convolution) operation. Compared with general convolution operations, DW convolution can reduce the amount of calculation and parameters to be trained while ensuring the diagnostic effect, thus speeding up the training and reasoning of the model.
[0084] This embodiment further illustrates a method for generating a decision-level fusion diagnosis result, the method comprising:
[0085] For the first multi-source data collected, the visible-infrared image diagnosis result VI-result of the visible light image-infrared thermal image feature fusion is obtained based on the visible light image-infrared thermal image defect diagnosis subnetwork. Based on the sound defect diagnosis subnetwork, the sound source defect diagnosis result S-result of the acoustic image Simg-sound feature vector Smfcc feature fusion is obtained, and then the two diagnosis results are fused at the decision level. At the same time, since the data preprocessing module performs scaling and alignment operations on the acoustic data, the first visible light image and the first infrared thermal image, the diagnosis results of the two diagnosis subnetworks can be directly fused without coordinate transformation.
[0086] Finally, the fused diagnosis result is directly annotated in the second visible light image Vimg.
[0087] Specifically, the decision-level diagnosis result fusion rule is based on the intersection-over-union (IOU) of the prediction boxes in each diagnosis result. The specific formula is as follows:
[0088]
[0089] Where S is the area enclosed by the prediction box in the prediction result, Ri is any defect in the diagnosis result of the visible light image-infrared thermal image defect diagnosis subnetwork, and Rj is any defect in the diagnosis result of the sound defect diagnosis subnetwork.
[0090] After testing, when the IOU threshold of the prediction box in the two prediction results is set to 0.83, the decision fusion has the best diagnostic effect. When visualizing the defect diagnosis results of the same device, only a single prediction box is displayed, and the defect categories are marked on the prediction box to avoid repeated drawing of boxes for a single device defect.
[0091] It should be noted that after the yolo network detects an image, it will obtain the detection results for this image, namely the sound source defect diagnosis result S-result and the visible-infrared image diagnosis result VI-result in this embodiment. The detection results may include one or more detected targets, where R1 indicates that any detected target in the sound source defect diagnosis result S-resutl must be judged as to whether it is the same defect as the detected target in the visible-infrared image diagnosis result VI-result. Each target will contain some information, and its format can be (target category, target position), where the target position is represented by the pixel position of a rectangular box.
[0092] The above two diagnostic subnetworks include a visible light image-infrared thermal image defect diagnosis subnetwork and a sound defect diagnosis subnetwork, wherein the sound defect diagnosis subnetwork utilizes the alignment of the acoustic image Simg and the second visible light image Vimg, and then displays the final result on the second visible light image Vimg.
[0093] The two networks detect independently, and the detection results obtained may overlap. Considering that the final display is in the second visible light image Vimg, the above-mentioned "decision-level diagnosis result fusion rule" is: to ensure that the repeated detection results in the two networks are displayed only once. The basis for judgment is the previous target type and rectangular frame pixel position. If the target type is the same and the IOU rectangular frame overlap intersection and union ratio is > 0.83, it is considered to overlap and only one is displayed.
[0094] like Figure 4 As shown, in this embodiment, after constructing the power equipment defect diagnosis model, it is also necessary to train and reason it. The second multi-source data of the power equipment in the entire specified format is divided into a training set, a validation set, and a test set in a ratio of 6:2:2. Based on the training set, the weights and thresholds of the model are adjusted using the back propagation algorithm. Based on the validation set, the model hyperparameters are adjusted to select a suitable deep learning model structure. The test set is used to evaluate the generalization ability of the model.
[0095] Embodiment 4:
[0096] On the basis of the above embodiment, this embodiment further shows a power equipment defect diagnosis device based on multi-source data fusion, the device at least includes a data acquisition module and an artificial intelligence processing module, the data acquisition module is used to collect first multi-source data of the power equipment, the artificial intelligence processing module at least includes: a data preprocessing module, a defect diagnosis module;
[0097] The data preprocessing module is used to preprocess the collected first multi-source data to generate second multi-source data of the power equipment in a specified format, wherein the second multi-source data of the power equipment in the specified format at least includes: an acoustic image Simg, a second visible light image Vimg, a second infrared thermal image Iimg and a sound feature vector Smfcc;
[0098] The defect diagnosis module is used to construct a defect diagnosis model for power equipment and generate a decision-level fusion diagnosis result based on the second multi-source data of the power equipment in a specified format. The decision-level fusion diagnosis result is generated by fusing the visible-infrared image diagnosis result VI-result with the sound source defect diagnosis result S-result.
[0099] A further power equipment defect diagnosis model includes at least: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision fusion network.
[0100] Among them, the visible light image-infrared thermal image defect diagnosis subnetwork is provided with a squeeze excitation SE module and a DW convolution module. The defect diagnosis module generates the calculation process of the visible-infrared image diagnosis result VI-result, which specifically includes:
[0101] Using the squeeze excitation SE module and the DW convolution module, calculate the visible light feature map FM-v3 and the visible light feature map FM-v5 of the second visible light image Vimg, and the infrared feature map FM-i3 and the infrared feature map FM-i5 of the second infrared thermal image Iimg;
[0102] Based on the position information in the visible light feature map FM-v3 and the visible light feature map FM-v5, the temperature information in the infrared thermal feature map FM-i3 and the infrared thermal feature map FM-i5 is non-uniformly extracted to generate the fused feature map output_1 and the fused feature map output_2;
[0103] According to the fused feature map output_1 and the fused feature map output_2, a non-maximum suppression NMS operation is performed to generate a visible-infrared image diagnosis result VI-result.
[0104] Furthermore, the defect diagnosis module generates a calculation process of the sound source defect diagnosis result S-result, which specifically includes:
[0105] According to the acoustic image Simg and the sound feature vector Smfcc, four layers of DW convolution and pooling operations are performed to flatten the obtained acoustic image feature map and sound feature vector feature map;
[0106] After 2 layers of full connection operations, the nonlinear relationship between features is determined;
[0107] After being processed by the softmax layer, the prediction result is output to generate the sound source defect diagnosis result S-result.
[0108] In a preferred implementation of this embodiment, the first multi-source data includes: sound data, a first visible light image and a first infrared thermal image, and the defect diagnosis module is further used to: annotate the decision-level fusion diagnosis result in the second visible light image Vimg.
[0109] The technical solution of the present application is described in detail above in conjunction with the accompanying drawings. The present application proposes a device and method for diagnosing defects in electric power equipment based on multi-source data fusion. The device includes at least a data acquisition module and an artificial intelligence processing module. The data acquisition module is used to collect the first multi-source data of the electric power equipment. The artificial intelligence processing module includes at least: a data preprocessing module and a defect diagnosis module; the data preprocessing module is used to preprocess the collected first multi-source data to generate the second multi-source data of the electric power equipment in a specified format; the defect diagnosis module is used to construct a defect diagnosis model for the electric power equipment, and generate a decision-level fusion diagnosis result according to the second multi-source data of the electric power equipment in a specified format. The decision-level fusion diagnosis result is generated by fusion of the visible-infrared image diagnosis result VI-result and the sound source defect diagnosis result S-result. Through the technical solution in the present application, the problems of relying on a single data source, high defect missed detection rate and false detection rate, and poor robustness in the existing electric power equipment defect diagnosis technology are overcome.
[0110] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.
[0111] The units in the device of the present application can be combined, divided and deleted according to actual needs.
[0112] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.
Claims
1. A power equipment defect diagnosis device based on multi-source data fusion, characterized in that: The device at least comprises a data acquisition module and an artificial intelligence processing module, wherein the data acquisition module is used to collect first multi-source data of the power equipment, and the artificial intelligence processing module at least comprises: a data preprocessing module, a defect diagnosis module; The data preprocessing module is used to preprocess the collected first multi-source data to generate second multi-source data of the power equipment, wherein the second multi-source data of the power equipment at least includes: an acoustic image Simg, a second visible light image Vimg, a second infrared thermal image Iimg and a sound feature vector Smfcc; The defect diagnosis module is used to construct a power equipment defect diagnosis model, and generate a decision-level fusion diagnosis result according to the second multi-source data of the power equipment, wherein the decision-level fusion diagnosis result is generated by fusing the visible-infrared image diagnosis result VI-result and the sound source defect diagnosis result S-result; The electric power equipment defect diagnosis model at least includes: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision fusion network; The visible light image-infrared thermal image defect diagnosis subnetwork is provided with a squeeze excitation SE module and a DW convolution module. The defect diagnosis module generates a calculation process of a visible-infrared image diagnosis result VI-result, which specifically includes: Utilizing the squeeze excitation SE module and the DW convolution module, calculating the visible light feature map FM-v3 and the visible light feature map FM-v5 of the second visible light image Vimg, and the infrared feature map FM-i3 and the infrared feature map FM-i5 of the second infrared thermal image Iimg; Based on the position information in the visible light feature map FM-v3 and the visible light feature map FM-v5, respectively, the temperature information in the infrared feature map FM-i3 and the infrared feature map FM-i5 is non-uniformly extracted to generate a fused feature map output_1 and a fused feature map output_2; A non-maximum suppression (NMS) operation is performed according to the fused feature map output_1 and the fused feature map output_2 to generate the visible-infrared image diagnosis result VI-result.
2. The power equipment defect diagnosis device based on multi-source data fusion according to claim 1, characterized in that: The calculation process of the defect diagnosis module generating the sound source defect diagnosis result S-result specifically includes: According to the acoustic image Simg and the sound feature vector Smfcc, a 4-layer DW convolution and pooling operation is performed to flatten the obtained acoustic image feature map and sound feature vector feature map; After 2 layers of full connection operations, the nonlinear relationship between features is determined; After being processed by the softmax layer, the prediction result is output to generate the sound source defect diagnosis result S-result.
3. The power equipment defect diagnosis device based on multi-source data fusion according to any one of claims 1 to 2, characterized in that: The first multi-source data includes: sound data, a first visible light image and a first infrared thermal image, and the defect diagnosis module is further used for: The decision-level fusion diagnosis result is annotated in the second visible light image Vimg.
4. A method for diagnosing defects in power equipment based on multi-source data fusion, characterized in that: The method comprises: Step 1, preprocessing the collected first multi-source data to generate second multi-source data of the power equipment, wherein the second multi-source data of the power equipment at least includes: an acoustic image Simg, a second visible light image Vimg, a second infrared thermal image Iimg and a sound feature vector Smfcc; Step 2, constructing a power equipment defect diagnosis model, generating a decision-level fusion diagnosis result based on the second multi-source data of the power equipment, wherein the decision-level fusion diagnosis result is generated by fusing the visible-infrared image diagnosis result VI-result and the sound source defect diagnosis result S-result, The electric power equipment defect diagnosis model at least includes: a visible light image-infrared thermal image defect diagnosis subnetwork, a sound defect diagnosis subnetwork and a decision fusion network; The visible light image-infrared thermal image defect diagnosis subnetwork is provided with a squeeze excitation SE module and a DW convolution module. In the step 2, the calculation process of the visible-infrared image diagnosis result VI-result specifically includes: Step 21, using the squeeze excitation SE module and the DW convolution module, calculate the visible light feature map FM-v3 and the visible light feature map FM-v5 of the second visible light image Vimg, and the infrared feature map FM-i3 and the infrared feature map FM-i5 of the second infrared thermal image Iimg; Step 22, performing non-uniform extraction on the temperature information in the infrared feature map FM-i3 and the infrared feature map FM-i5 based on the position information in the visible light feature map FM-v3 and the visible light feature map FM-v5, respectively, to generate a fused feature map output_1 and a fused feature map output_2; Step 23, performing a non-maximum suppression (NMS) operation according to the fused feature map output_1 and the fused feature map output_2 to generate the visible-infrared image diagnosis result VI-result.
5. The method for diagnosing defects in electric power equipment based on multi-source data fusion according to claim 4, characterized in that: In step 2, the calculation process of the sound source defect diagnosis result S-result specifically includes: According to the acoustic image Simg and the sound feature vector Smfcc, a 4-layer DW convolution and pooling operation is performed to flatten the obtained acoustic image feature map and sound feature vector feature map; After 2 layers of full connection operations, the nonlinear relationship between features is determined; After being processed by the softmax layer, the prediction result is output to generate the sound source defect diagnosis result S-result.
6. The method for diagnosing defects in electric power equipment based on multi-source data fusion according to claim 5, characterized in that: The first multi-source data includes: sound data, a first visible light image and a first infrared thermal image. After step 2, the method further includes: The decision-level fusion diagnosis result is annotated in the second visible light image Vimg.
