Power equipment detection method and device based on image fusion

Abstract

The invention provides a power equipment detection method and device based on image fusion, and belongs to the field of power equipment defect detection. The method comprises the following steps: acquiring a visible light image, an infrared image and temperature information of power equipment; extracting first feature points of the two images; according to the first feature point, registering the infrared image through a registration model to obtain a registered image; extracting second feature points of the visible light image and the registration image; according to the second feature point, the registration image and the visible light image are fused through a fusion model; enhancing the feature information of the obtained fused image based on a feature fusion framework to obtain a to-be-detected feature image; inputting the to-be-detected feature image into a detection module to obtain preliminary detection information of the power equipment; and determining a final detection result of the power equipment in combination with the temperature information. According to the method, the image fusion and target detection technology is applied, and the thermal defects of the power equipment can be rapidly, accurately and intelligently detected.

Description

Technical Field

[0001] The present application belongs to the field of power equipment defect detection, and specifically relates to a power equipment detection method and device based on image fusion. Background Art

[0002] Infrared thermal imaging technology has always played an important role in the diagnosis of defects in power equipment. It has the characteristics of long distance, no contact, no sampling, and no contact with the body. There is no need to touch the power equipment directly when monitoring and troubleshooting, which effectively avoids the risk of accidents caused by direct contact with high-voltage equipment. Since there is no need to contact the object being tested, the infrared thermal imager can perform detection work when the power equipment is energized, without the need to cut off the power, thus ensuring the continuous operation of the power equipment and reducing the impact of the detection on the power supply. At the same time, since there is no need to cut off the power for detection, the economic losses caused by power outages are also reduced.

[0003] Visible light images have rich texture details and color information under good lighting conditions, but the target detection effect based on visible light images is poor under complex lighting conditions such as weak light, no light, and strong light at night. Infrared images are formed based on the target's thermal radiation information and are not affected by lighting conditions, but lack color information, local texture details are not rich, and there is more background noise.

[0004] Using single-modal images for substation equipment status monitoring is difficult to meet the detection needs under complex conditions in substations. Therefore, how to effectively combine the morphological features of visible light images and the thermal anomaly information of infrared images to conduct a more comprehensive assessment of the operating status of substation equipment has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for detecting power equipment based on image fusion, so as to solve the problem in the prior art in the field of power equipment defect detection that the target detection effect based on visible light images is poor under complex lighting conditions such as weak light, no light, and strong light at night, while the infrared images lack color information, have poor local texture details, and have more background noise, resulting in low target detection efficiency.

[0006] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a method for detecting electric power equipment based on image fusion, the method comprising: Acquire visible light images, infrared images and multiple temperature information in infrared images of power equipment; Based on the first neural network, extracting first feature points of the visible light image and the infrared image; wherein the first feature points are used to describe geometric texture information of the visible light image and the infrared image; According to the first feature point, based on the registration model and the visible light image, registering the infrared image to obtain a registered image; Based on the second neural network, extract the second feature points of the visible light image and the registered image; wherein the second feature points are used to describe the semantic information of the visible light image and the infrared image; according to the second feature points, based on the fusion model, fuse the registered image and the visible light image to obtain a fused image; A feature fusion framework including a feature extraction network, a scale sequence feature fusion module and a triple feature encoder is built. Based on the feature fusion framework, the feature information of the fused image is enhanced to obtain the feature image to be detected. Specifically, the following steps are used: based on the feature extraction network, preliminary feature images of different levels of the fused image are extracted. Some convolutions of the feature extraction network are modified to depth-wise over-parameterized convolutions. According to the preliminary feature image, based on the scale sequence feature fusion module and the triple feature encoder, the feature image to be detected is obtained; the feature image to be detected is input into the detection module for detection to obtain preliminary detection information of the power equipment; wherein the preliminary detection information includes preliminary defect location, bounding box, category and confidence; The final inspection result of the power equipment is determined based on the preliminary inspection information and multiple temperature information.

[0007] Compared with the prior art, the above technical solution provided by the present application includes at least the following beneficial effects: The present application first obtains the visible light, infrared images and temperature information of the power equipment, and then uses the first neural network to extract the first feature points of the two images. Then, according to these feature points, the infrared image is registered through the registration model and the visible light image to generate a registered image to ensure the accuracy of subsequent image fusion. Then, the second neural network is used to extract the second feature points of the image, and based on these feature points, the registered image is fused with the visible light image through the fusion model to generate a fused image to eliminate the problem of uneven matching of the background of some images during registration, while enhancing the visibility of the foreground of the target image and reducing the impact of the image background on the detection. The feature information of the fused image is then enhanced through the feature fusion framework to form a feature image to be detected. Finally, the feature image to be detected is input into the detection module to obtain preliminary detection information, and combined with the temperature information, the final detection result of the power equipment is determined. The present application uses image fusion and target detection technology to quickly, accurately and intelligently detect thermal defects of power equipment.

[0008] Preferably, according to the first feature point, based on the registration model and the visible light image, registering the infrared image to obtain the registered image, the specific steps include: According to the first feature point, the infrared image is spatially transformed to obtain a mapping image; wherein the mapping image can be aligned with the visible light image; Perform pixel interpolation processing on the new pixel points of the mapped image to obtain the registered image.

[0009] In this solution, the infrared image and the visible light image are registered so that the images can be accurately aligned according to the feature points, ensuring the accuracy of subsequent image fusion and making the fused images consistent in spatial position, avoiding misalignment, blur or discontinuity in image content.

[0010] Preferably, the image preprocessing function used by the scale sequence feature fusion module is: in, represents a two-dimensional input image of preliminary feature image with width w and height h, A series of convolutions using a two-dimensional Gaussian filter Smoothing the preliminary feature image; Scaling parameter for the standard deviation of the 2D Gaussian filter used for convolution.

[0011] In this scheme, the generated images have the same resolution but different scales, so feature maps of different sizes can be regarded as scale space, and the effective feature maps of different resolutions can be adjusted to the same resolution for subsequent splicing.

[0012] Preferably, according to the preliminary feature image, based on the scale sequence feature fusion module and the triple feature encoder, the feature image to be detected is obtained, and the specific steps include: Inputting the preliminary feature image into the scale sequence feature fusion module to obtain the first feature image; Perform convolution processing on the preliminary feature image to obtain a convolution image; Input the convolution image into a triple feature encoder for splicing to obtain a second feature image; wherein the number of channels of the triple feature encoder is determined according to the size information of the preliminary feature image; Based on the attention sequence fusion mechanism, the first feature image and the second feature image are fused to obtain the feature image to be detected.

[0013] In this scheme, the scale sequence feature fusion module is used to better combine the high-dimensional information of the deep feature map with the detailed information of the shallow feature map. The image size changes during the downsampling process, but the scale-invariant features do not change, and the structural features of the image are preserved. A triple feature encoder is also used to classify large, medium, and small features, which can capture the local fine details of small targets. Finally, the attention sequence fusion mechanism is used to fuse the feature information output by the two modules to further improve the accuracy of subsequent target detection.

[0014] Preferably, the number of channels of the triple feature encoder is determined according to the size information of the preliminary feature image, and the specific steps include: Preset the size parameter standard of the preliminary feature image; Comparing the size information of the preliminary feature image with the size parameter standard, classifying the preliminary feature image to obtain a classification result; the classification result includes a large-size feature image, a medium-size feature image, and a small-size feature image; According to the classification results, the number of channels of the triple feature encoder is adjusted; for large-size feature images, the number of channels is adjusted to 1, and for small-size feature images, convolution is used to adjust the number of channels.

[0015] In this scheme, the preliminary features are classified and the number of channels adapted is adjusted to ensure the resolution of the feature image.

[0016] Preferably, the specific steps of determining the final detection result of the electric equipment according to the preliminary detection information and the multiple temperature information include: Preset equipment defect judgment criteria; According to the multiple temperature information, the maximum temperature difference corresponding to the preliminary defect position is obtained; The final inspection results of the power equipment are determined based on the maximum temperature difference and the equipment defect judgment criteria.

[0017] In this scheme, it is possible to quickly detect whether the electric power equipment has defects.

[0018] In a second aspect, an embodiment of the present application provides an electric power equipment detection device based on image fusion, comprising: An image acquisition module, used to acquire visible light images, infrared images and multiple temperature information in infrared images of electric equipment; A first feature point extraction module, used to extract first feature points of the visible light image and the infrared image based on a first neural network; wherein the first feature points are used to describe geometric texture information of the visible light image and the infrared image; An image registration module, configured to register the infrared image according to the first feature point, based on the registration model and the visible light image, to obtain a registered image; A second feature point extraction module, used to extract second feature points of the visible light image and the registered image based on a second neural network; wherein the second feature points are used to describe semantic information of the visible light image and the infrared image; An image fusion module, used for fusing the registered image with the visible light image according to the second feature point based on a fusion model to obtain a fused image; The image feature enhancement module is used to build a feature fusion framework including a feature extraction network, a scale sequence feature fusion module and a triple feature encoder. Based on the feature fusion framework, the feature information of the fused image is enhanced to obtain the feature image to be detected, which specifically includes: based on the feature extraction network, the preliminary feature images of different levels of the fused image are extracted, and some convolutions of the feature extraction network are modified to depth-by-depth over-parameterized convolutions. According to the preliminary feature image, the feature image to be detected is obtained based on the scale sequence feature fusion module and the triple feature encoder; A detection module is used to input the feature image to be detected into the detection module for detection, and obtain preliminary detection information of the power equipment; wherein the preliminary detection information includes preliminary defect location, bounding box, category and confidence; The judgment module is used to determine the final detection result of the power equipment based on the preliminary detection information and multiple temperature information.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0021] It can be understood that the beneficial effects of the technical solutions provided in the second, third and fourth aspects can be found in the relevant description of the first aspect, and will not be repeated here.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 is a schematic flow chart of a method for detecting electric power equipment based on image fusion provided in some embodiments of the present application; Figure 2 is a schematic diagram of an infrared image and a visible light image shown in some embodiments of the present application; Figure 3 is a schematic diagram of registration images shown in some embodiments of the present application; Figure 4 is a schematic diagram of a fused image shown in some embodiments of the present application; Figure 5 is a block diagram of an electric power equipment detection device based on image fusion shown in some embodiments of the present application; Figure 6 It is a block diagram of an electronic device shown in some embodiments of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0026] In the following, in conjunction with the accompanying drawings, a method for detecting electric power equipment based on image fusion provided by an embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0027] Figure 1 This is a flow chart of a method for detecting power equipment based on image fusion according to the first embodiment of the present application. Figure 1 , the method comprising: Step S101: acquiring a visible light image, an infrared image, and multiple temperature information in the infrared image of the power equipment; Specifically, an infrared image of the power equipment is obtained by an infrared imaging instrument, and at the same time, several temperature information in the image, such as the highest temperature, the lowest temperature, and the temperature scale, is obtained. At the same time, a visible light image of the corresponding power equipment is obtained by a visible light camera.

[0028] Among them, in the infrared image and the visible light image, the size of the infrared image should be smaller than or equal to the size of the visible light image, and the infrared image can be contained in the visible light image. When detecting the target power equipment, the target power equipment is considered to be completely in the infrared image. For particularly large equipment, only the target power equipment can be included in the infrared image.

[0029] Specifically, an infrared thermal imaging instrument is used to perform infrared thermal imaging on the target power equipment, and a visible light camera is used to photograph the target power equipment. The infrared image of the power equipment requires that the power equipment to be tested is in the middle of the infrared image. The visible light image and the infrared image are in the same position to obtain corresponding pictures of the same device. The number of the highest temperature values ​​in each target device to be tested is equal to the highest pixel value of the temperature scale.

[0030] As a possible implementation method, taking reactors, resistors, grading rings, and insulating pipes as target power equipment as examples, infrared imaging is used to perform infrared imaging on the above power equipment, and a visible light camera is used to photograph the target power equipment.

[0031] Please continue reading Figure 2 The resolution of the infrared image on the left is 1024×768, and the resolution of the visible light image on the right is 1280×960. During the analysis, the target power equipment is in the middle of the infrared thermal imaging instrument, and infrared imaging of the target power equipment is performed at the same time; then the visible light camera is used to sample the same target area, and the infrared image is completely contained in the visible light image.

[0032] Step S102: extracting first feature points of the visible light image and the infrared image based on the first neural network; wherein the first feature points are used to describe geometric texture information of the visible light image and the infrared image; It should be noted that the main goal of extracting the first feature point is to find similar parts in two or more images, which are usually based on feature points, lines, edges, etc., for subsequent image matching and superposition. The feature extraction process needs to ensure that the extracted features are repeatable, distinguishable, efficient, and local.

[0033] Specifically, only unlabeled data is used to train the first neural network, and the generalization properties of these features make them robust to transformations.

[0034] Step S103: registering the infrared image according to the first feature point, based on the registration model and the visible light image to obtain a registered image; According to the first feature point, the infrared image of the power equipment is mapped onto the visible light image of the power equipment by using an image registration technology to obtain a mapped image, wherein the mapped image can be aligned with the visible light image; Specifically, the first feature points of the infrared image and the visible light image are extracted through the trained convolutional network, and then image mapping is performed based on the first feature points. Image mapping is a series of spatial transformation processes such as distortion and deformation of the infrared image to make it more suitable for the visible light image of the power equipment and align with the visible light image.

[0035] Furthermore, pixel interpolation processing is performed on the new pixel positions of the mapped image to obtain a registered image.

[0036] Among them, the total number of pixels of the mapped image is usually smaller than that of the visible light image, and the area outside the mapped image is filled with black pixels to achieve the same resolution size for the two, thereby obtaining a registered image with the same pixel value size as the visible light image.

[0037] Preferably, the resolution of the mapped image is 1280×960. The areas other than the deformed areas of the original infrared image are filled with black pixels. The RGB value of the black pixels is (0, 0, 0), which will not affect other areas during fusion. For the registration image, please refer to Figure 3 .

[0038] Step S104: extracting second feature points of the visible light image and the registered image based on the second neural network; wherein the second feature points are used to describe semantic information of the visible light image and the infrared image; It should be noted that the main goal of extracting the second feature point is to extract useful information from multiple images so as to synthesize a new image according to certain fusion rules. This new image usually describes the target more clearly and accurately and is more suitable for subsequent image processing tasks.

[0039] Specifically, the deep learning method is used to extract the second feature points of the visible light image and the registered image through the second neural network, wherein the brightness information and gradient information of the infrared image need to be considered to retain more texture details of the visible light image.

[0040] Step S105: According to the second feature point, based on the fusion model, the registered image is fused with the visible light image to obtain a fused image; Specifically, the visible light image and the mapping image are put into the image fusion model for fusion to eliminate the problem of uneven matching of some image backgrounds during registration, while enhancing the visibility of the foreground of the target image and reducing the impact of the image background on detection; please refer to the fusion image Figure 4 .

[0041] Step S106: constructing a feature fusion framework including a feature extraction network, a scale sequence feature fusion module and a triple feature encoder, and based on the feature fusion framework, strengthening the feature information of the fused image to obtain a feature image to be detected, which specifically includes: extracting preliminary feature images of different levels of the fused image based on the feature extraction network, modifying some convolutions of the feature extraction network to depth-by-depth over-parameterized convolutions, and obtaining the feature image to be detected based on the preliminary feature image and the scale sequence feature fusion module and the triple feature encoder; First, the fused image is input into a feature extraction network modified from a partial convolution (Conv) to a depth-wise over-parameterized convolution (DO-Conv). The fused image features are extracted through a feature extraction network with fewer parameters to obtain a preliminary feature image. Among them, in order to achieve end-to-end detection speed, the convolution structure in the feature extraction network is improved, and DO-Conv is used to replace the Conv in the original convolution structure. This convolution layer uses deep over-parameterization during training to explore more feature representation capabilities, and then these additional parameters are simplified back to the form of standard convolution layers through fusion operations to ensure that the execution efficiency of the final model remains unchanged.

[0042] It should be noted that in the process of modifying the feature extraction network to DST-yolo by improving the convmodule to do-convmodule, Do-conv itself does not have the effect of reducing the number of parameters and improving accuracy. The advantage of Do-conv is that it improves the model performance through over-parameterization in the training phase, and in the inference phase, it maintains the computational efficiency and parameter quantity unchanged through parameter merging. Since the subsequent improved feature fusion framework model can improve the accuracy of the model, the do-convmodule cannot directly improve the model accuracy, but through the residual structure of the original feature extraction network, the weight of some convolution modules is reduced, and finally the number of parameters of the feature extraction network is reduced, so as to achieve the purpose of increasing the speed.

[0043] Please refer to Table 1. Although PoolFormer-yolo guarantees speed, its map50 is the lowest among all models. Even though DST-yolo improves on the ST-yolo model, it only reduces map50 by 0.07 points, and the detection speed of each image is increased by about 22.6%: Table 1 Comparison of improved parameters and running time of feature extraction network Secondly, the preliminary feature image is input into the Scale Sequence Feature Fusion (SSFF) module which combines the global or high-level semantic information from images at multiple scales.

[0044] Scale means the details of the image. A blurred image may lose details but can retain its structural features.

[0045] Specifically, the scaled image as input to SSFF can be obtained by the following method: in, A two-dimensional input image representing a preliminary feature image of width w and height h. is a series of convolutions using a two-dimensional Gaussian filter Generated by smoothing the preliminary feature image. is the standard deviation scaling parameter of the 2D Gaussian filter used for convolution.

[0046] Preferably, a Triple Feature Encoder (TFE) module that can capture the local fine details of small target objects can be added at the same time. Before feature encoding, the number of feature channels needs to be adjusted to be consistent with the main scale features. After processing the large-size feature map, its number of channels is adjusted to 1, and then down-sampled using a hybrid structure of maximum pooling and average pooling, which helps to retain the effectiveness and diversity of high-resolution features and power equipment images. For small-size feature maps, convolution modules are also used to adjust the number of channels, and then up-sampled using the nearest neighbor interpolation method. This helps to maintain the richness of local features of low-resolution images and prevent the loss of small target feature information. Finally, three feature maps of the same size, large, medium, and small, are convolved once and then spliced ​​in the channel dimension, as shown below.

[0047] in Represents the feature map output by the TFE module. Represent feature maps of large, medium and small sizes, respectively. Depend on 's splicing. With The same resolution and the number of channels is Three times.

[0048] By modifying the above modules, it can serve as the feature fusion framework in the original target detection model; Furthermore, the attention sequence fusion mechanism is adopted to fuse the processing results of the SSFF module and the TFE module to obtain the feature image to be detected.

[0049] Preferably, YoloV8 is used as the basic framework, and an attentional scale sequence fusion mechanism (ASF) is added to fuse feature maps of different scales, so as to obtain better image features and extract richer and more accurate feature information.

[0050] In a possible implementation, this embodiment improves the feature fusion framework by introducing the SSFF module and the TFE module to form the ST-YOLO architecture. Given that the equipment layout in the substation environment is dense and the size differences are significant, it is necessary to pay attention to the detection of the features of the shallow feature map to improve the recall rate and precision of the model.

[0051] It should be noted that traditional feature fusion methods usually connect feature maps after upsampling, which often makes the subsequent detection process overly dependent on smaller feature maps, which is not ideal when dealing with densely populated scenes such as substations. To solve this problem, this embodiment proposes a new strategy: first normalize the feature maps at each level, and then perform the upsampling step. This process adjustment enables the model to capture the information of small objects more effectively.

[0052] For example, some models may ignore the existence of small targets such as pressure equalizing rings; however, after optimizing the feature fusion module, the detection of small targets has become more accurate and reliable even in complex backgrounds.

[0053] In this way, the improvement of this embodiment not only enhances the recognition capability of small targets, but also achieves more accurate target detection in the entire substation monitoring system.

[0054] Please refer to Table 2. ST-YOLO's recall rate, map50 and map50-90 parameters are the best among the four models. Among them, map50 is an important criterion for model detection. The accuracy based on the intersection of the true box and the detection box is 0.5, indicating that it has a higher accuracy rate under the premise of more accurate target selection: Table 2 Comparison of improved parameters of feature fusion framework Step S107: inputting the feature image to be detected into the detection module for detection to obtain preliminary detection information of the power equipment; wherein the preliminary detection information includes preliminary defect location, bounding box, category and confidence level; It should be noted that the detection module includes a detection head and a classification head. The detection head is mainly used in target detection tasks, and its purpose is to identify objects in the image and determine their locations. The classification head is mainly used in image classification tasks, and its purpose is to classify the input image into one of the predefined categories.

[0055] The preliminary detection information also includes the coordinate information of each power equipment.

[0056] Step S108: Determine the final detection result of the power equipment according to the preliminary detection information and the multiple temperature information.

[0057] Specifically, it is determined whether the target power equipment has defects based on the detected image information of the power equipment and the temperature difference information obtained by the infrared image.

[0058] Among them, the preset judgment standard can be "DL / T 664-2008 Application Specifications for Infrared Diagnosis of Energized Equipment".

[0059] The power equipment detection method based on image fusion provided in the above embodiment uses a fused image of an infrared image and a visible light image for target detection, and compares the temperature difference on the image with the standard, so as to quickly establish an equipment value model, accurately and intelligently compare equipment thermal defects, and identify equipment with hidden dangers.

[0060] This method can improve the detection speed while improving the accuracy and recall rate of the detection target. This method is quick and convenient to use after the model training is completed, but it is necessary to provide relevant power equipment data sets in advance to train the model. In addition, since this embodiment uses image registration technology, the size of the infrared image and the visible light image can be inconsistent, and the shooting range can also be inconsistent. It can still effectively detect the defects of the power equipment when using different equipment.

[0061] It should be noted that the power equipment detection method based on image fusion provided in the embodiment of the present application can be executed by a power equipment detection device based on image fusion, or a control module in the power equipment detection device based on image fusion for executing the loading of the power equipment detection method based on image fusion. In the embodiment of the present application, the method of the power detection device based on image fusion provided in the embodiment of the present application is explained by taking the execution of the loading of the power equipment detection method based on image fusion by the power equipment detection device based on image fusion as an example.

[0062] Figure 5 is a schematic diagram of a power detection device based on image fusion according to the second embodiment of the present application, please refer to Figure 5 , the power equipment detection device 200 based on image fusion includes: Image acquisition module 201: used to acquire visible light images, infrared images and multiple temperature information in infrared images of power equipment; The first feature point extraction module 202 is used to extract first feature points of the visible light image and the infrared image based on the first neural network; wherein the first feature points are used to describe geometric texture information of the visible light image and the infrared image; Image registration module 203: used to register the infrared image according to the first feature point, based on the registration model and the visible light image to obtain a registered image; specifically includes: According to the first feature point, the infrared image is spatially transformed to obtain a mapping image; wherein the mapping image can be aligned with the visible light image; Perform pixel interpolation processing on the new pixel points of the mapped image to obtain the registered image.

[0063] The second feature point extraction module 204 is used to extract the second feature points of the visible light image and the registered image based on the second neural network; wherein the second feature points are used to describe the semantic information of the visible light image and the infrared image; Image fusion module 205: used to fuse the registered image with the visible light image according to the second feature point based on the fusion model to obtain a fused image; Image feature enhancement module 206: used to build a feature fusion framework including a feature extraction network, a scale sequence feature fusion module and a triple feature encoder, based on the feature fusion framework, strengthen the feature information of the fused image, and obtain the feature image to be detected, which specifically includes: based on the feature extraction network, extracting preliminary feature images of different levels of the fused image, modifying some convolutions of the feature extraction network to depth-by-depth over-parameterized convolutions, and obtaining the feature image to be detected based on the preliminary feature image and the scale sequence feature fusion module and the triple feature encoder; Specifically, it includes: inputting the preliminary feature image into the scale sequence feature fusion module to obtain the first feature image; Perform convolution processing on the preliminary feature image to obtain a convolution image; Input the convolution image into a triple feature encoder for splicing to obtain a second feature image; wherein the number of channels of the triple feature encoder is determined according to the size information of the preliminary feature image; Specifically, firstly, a size parameter standard of a preliminary feature image is preset; then, the size information of the preliminary feature image is compared with the size parameter standard, and the preliminary feature image is classified to obtain a classification result; the classification result includes a large-size feature image, a medium-size feature image, and a small-size feature image; According to the classification results, the number of channels of the triple feature encoder is adjusted; for large-size feature images, the number of channels is adjusted to 1; for small-size feature images, convolution is used to adjust the number of channels.

[0064] Finally, based on the attention sequence fusion mechanism, the first feature image and the second feature image are fused to obtain the feature image to be detected.

[0065] Detection module 207: used to input the feature image to be detected into the detection module for detection, and obtain preliminary detection information of the power equipment; wherein the preliminary detection information includes preliminary defect location, boundary box, category and confidence level; The judgment module 208 is used to determine the final detection result of the power equipment according to the preliminary detection information and the multiple temperature information; specifically includes: Preset equipment defect judgment criteria; According to the multiple temperature information, the maximum temperature difference corresponding to the preliminary defect position is obtained; The final inspection results of the power equipment are determined based on the maximum temperature difference and the equipment defect judgment criteria.

[0066] The power equipment detection device based on image fusion in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0067] The power equipment detection device based on image fusion in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0068] The power equipment detection device based on image fusion provided in the embodiment of the present application can achieve Figures 1 to 4 In order to avoid repetition, the various processes implemented by the power equipment detection device based on image fusion in the method embodiment will not be described here.

[0069] The present embodiment provides a power equipment detection device based on image fusion, which first obtains visible light, infrared images and temperature information of the power equipment, and then uses the first neural network to extract the first feature points of the two images. Then, according to these feature points, the infrared image is registered by the registration model and the visible light image to generate a registered image to ensure the accuracy of subsequent image fusion. Then, the second neural network is used to extract the second feature points of the image, and based on these feature points, the registered image is fused with the visible light image through the fusion model to generate a fused image to eliminate the problem of uneven matching of the background of some images during registration, while enhancing the visibility of the foreground of the target image and reducing the impact of the image background on the detection. The feature information of the fused image is then enhanced through the feature fusion framework to form a feature image to be detected. Finally, the feature image to be detected is input into the detection module to obtain preliminary detection information, and combined with the temperature information, the final detection result of the power equipment is determined. This application uses image fusion and target detection technology to quickly, accurately and intelligently detect thermal defects of power equipment.

[0070] Optionally, see Figure 6 The embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the computer program 303 is executed by the processor 301, the various processes of the above-mentioned embodiment of the power equipment detection method based on image fusion are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0071] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the power equipment detection method based on image fusion is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0072] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0073] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the power equipment detection method based on image fusion, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0074] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0075] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0077] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for detecting electric power equipment based on image fusion, characterized in that: include: Acquire a visible light image, an infrared image, and a plurality of temperature information in the infrared image of the electric equipment; Based on a first neural network, extracting first feature points of the visible light image and the infrared image; wherein the first feature points are used to describe geometric texture information of the visible light image and the infrared image; According to the first feature point, based on a registration model and the visible light image, registering the infrared image to obtain a registered image; Based on a second neural network, extracting second feature points of the visible light image and the registered image; wherein the second feature points are used to describe semantic information of the visible light image and the infrared image; According to the second feature point, based on a fusion model, fusing the registered image with the visible light image to obtain a fused image; A feature fusion framework including a feature extraction network, a scale sequence feature fusion module and a triple feature encoder is constructed. Based on the feature fusion framework, the feature information of the fused image is enhanced to obtain a feature image to be detected, which specifically includes: based on the feature extraction network, preliminary feature images of different levels of the fused image are extracted, and some convolutions of the feature extraction network are modified to depth-by-depth over-parameterized convolutions. According to the preliminary feature image, based on the scale sequence feature fusion module and the triple feature encoder, the feature image to be detected is obtained; Inputting the feature image to be detected into the detection module for detection to obtain preliminary detection information of the power equipment; wherein the preliminary detection information includes preliminary defect location, bounding box, category and confidence level; A final detection result of the electric equipment is determined according to the preliminary detection information and the multiple temperature information.

2. The power equipment detection method based on image fusion according to claim 1 is characterized in that: The specific steps of registering the infrared image according to the first feature point based on the registration model and the visible light image to obtain the registered image include: According to the first feature point, spatially transform the infrared image to obtain a mapping image; wherein the mapping image can be aligned with the visible light image; Perform pixel interpolation processing on the new pixel points of the mapping image to obtain a registered image.

3. The power equipment detection method based on image fusion according to claim 1 is characterized in that: The image preprocessing function used by the scale sequence feature fusion module is: in, A two-dimensional input image representing the preliminary feature image of width w and height h, A series of convolutions using a two-dimensional Gaussian filter Performing smoothing processing on the preliminary feature image to generate; Scaling parameter for the standard deviation of the 2D Gaussian filter used for convolution.

4. The power equipment detection method based on image fusion according to claim 1 is characterized in that: The specific steps of obtaining the feature image to be detected according to the preliminary feature image based on the scale sequence feature fusion module and the triple feature encoder include: Inputting the preliminary feature image into the scale sequence feature fusion module to obtain a first feature image; Performing convolution processing on the preliminary feature image to obtain a convolution image; Inputting the convolution image into a triple feature encoder for splicing to obtain a second feature image; wherein the number of channels of the triple feature encoder is determined according to the size information of the preliminary feature image; The first feature image and the second feature image are spliced ​​to obtain a feature image to be detected.

5. The power equipment detection method based on image fusion according to claim 4 is characterized in that: The number of channels of the triple feature encoder is determined according to the size information of the preliminary feature image, and the specific steps include: Presetting a size parameter standard of the preliminary feature image; Comparing the size information of the preliminary feature image with the size parameter standard, classifying the preliminary feature image to obtain a classification result; the classification result includes a large-size feature image, a medium-size feature image, and a small-size feature image; According to the classification result, the number of channels of the triple feature encoder is adjusted; for a large-size feature image, the number of channels is adjusted to 1; for a small-size feature image, the number of channels is adjusted by convolution.

6. The power equipment detection method based on image fusion according to claim 1 is characterized in that: The specific steps of determining the final detection result of the electric power equipment according to the preliminary detection information and the plurality of temperature information include: Preset equipment defect judgment criteria; According to the multiple temperature information, obtaining the maximum temperature difference corresponding to the preliminary defect position; According to the maximum temperature difference and based on the equipment defect judgment standard, the final detection result of the power equipment is determined.

7. An image fusion-based power equipment detection device, used to execute the detection method according to any one of claims 1 to 6, characterized in that: include: An image acquisition module, used to acquire a visible light image, an infrared image and a plurality of temperature information in the infrared image of the electric equipment; A first feature point extraction module, used to extract first feature points of the visible light image and the infrared image based on a first neural network; wherein the first feature points are used to describe geometric texture information of the visible light image and the infrared image; An image registration module, configured to register the infrared image according to the first feature point, based on a registration model and the visible light image, to obtain a registered image; A second feature point extraction module, used to extract second feature points of the visible light image and the registered image based on a second neural network; wherein the second feature points are used to describe semantic information of the visible light image and the infrared image; An image fusion module, configured to fuse the registered image with the visible light image according to the second feature point based on a fusion model to obtain a fused image; An image feature enhancement module is used to build a feature fusion framework including a feature extraction network, a scale sequence feature fusion module and a triple feature encoder, and based on the feature fusion framework, the feature information of the fused image is enhanced to obtain a feature image to be detected, which specifically includes: based on the feature extraction network, preliminary feature images of different levels of the fused image are extracted, some convolutions of the feature extraction network are modified to depth-by-depth over-parameterized convolutions, and according to the preliminary feature images, the feature image to be detected is obtained based on the scale sequence feature fusion module and the triple feature encoder; A detection module, used for inputting the feature image to be detected into the detection module for detection, and obtaining preliminary detection information of the electric power equipment; wherein the preliminary detection information includes preliminary defect location, bounding box, category and confidence; The judgment module is used to determine the final detection result of the electric equipment according to the preliminary detection information and the multiple temperature information.

8. An electronic device, characterized in that: include: A memory, a processor, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, is the step of the power equipment detection method based on image fusion as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the power equipment detection method based on image fusion as described in any one of claims 1-6 are implemented.

Images