Substation Equipment Image Fusion Method, Device, and Computer Product

Abstract

The present invention discloses a method, device, and computer product for image fusion of substation equipment. It relates to the field of image fusion. The method includes: obtaining a target visible light image and a target infrared image of the substation equipment; performing first fusion processing on the contour features respectively corresponding to the target visible light image and the target infrared image to obtain an initial fusion image; performing edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image and an infrared edge image; determining a first weight mapping coefficient corresponding to the feature points in the initial fusion image and a second weight mapping coefficient corresponding to the feature points in the target infrared image; and performing second fusion processing on the feature points in the infrared edge image and the fusion edge image to obtain a target fusion image. The present invention solves the technical problem that the clarity of the fused image obtained by the substation equipment image fusion method in the related technology is poor.

Description

Technical Field

[0001] The present invention relates to the field of image fusion, and in particular, to a method and device for fusing substation equipment images and a computer product. Background Art

[0002] With the application of inspection robots and video monitoring devices in substations, the substation can be patrolled by a video monitoring device with visible light and infrared thermal imaging functions to assist in completing part of the inspection work of substation equipment.

[0003] In the related art, video monitoring devices with visible light and infrared thermal imaging functions are all used to register and fuse the visible light image and infrared image of substation equipment, and then analyze potential hazards of target objects (such as substation equipment) in the fused image to achieve the effect of patrol.

[0004] However, due to the lack of extraction of the edges of target objects in the collected visible light image and infrared image, the edge information of the target objects in the image is not completely retained during the fusion process, resulting in low clarity of the fused image, and potential hazards existing during the subsequent substation inspection cannot be detected in time.

[0005] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0006] Embodiments of the present invention provide a method and device for fusing substation equipment images and a computer product, so as to at least solve the technical problem that the clarity of the fused image obtained by the substation equipment image fusion method in the related art is poor.

[0007] According to one aspect of the embodiments of the present invention, a method for fusing substation equipment images is provided, including: obtaining a target visible light image and a target infrared image of substation equipment; performing first fusion processing on the contour features respectively corresponding to the target visible light image and the target infrared image to obtain an initial fused image; performing edge extraction processing on the target objects included in the initial fused image and the target infrared image respectively to obtain a fused edge image corresponding to the initial fused image and an infrared edge image corresponding to the target infrared image; determining first weight mapping coefficients respectively corresponding to the feature points included in the initial fused image, and second weight mapping coefficients respectively corresponding to the feature points included in the target infrared image; and performing second fusion processing on the feature points respectively included in the infrared edge image and the fused edge image based on the first weight mapping coefficients and the second weight mapping coefficients to obtain a target fused image.

[0008] According to another aspect of the embodiments of the present invention, there is also provided an image fusion device for substation equipment, including: an acquisition module, configured to acquire a target visible light image and a target infrared image of the substation equipment; an edge processing module, configured to perform a first fusion process on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fusion image; perform edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image; a weight mapping coefficient determination module, configured to determine first weight mapping coefficients corresponding to the feature points included in the initial fusion image respectively, and second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively; a fusion module, configured to perform a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image.

[0009] According to another aspect of the embodiments of the present invention, there is also provided a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the image fusion methods for substation equipment are implemented.

[0010] In the embodiments of the present invention, by acquiring a target visible light image and a target infrared image of the substation equipment; performing a first fusion process on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fusion image; performing edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image; determining first weight mapping coefficients corresponding to the feature points included in the initial fusion image respectively, and second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively; performing a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain the target fusion image. The purpose of performing a secondary fusion of the infrared edge image and the fusion edge image by combining the weight mapping coefficient on the basis of obtaining the initial fusion image through contour feature extraction to obtain a clearer target fusion image is achieved, thereby realizing the technical effect of improving the clarity of the fused image corresponding to the substation equipment, and further solving the technical problem of poor clarity of the fused image obtained by the substation equipment image fusion method in the related art. Description of the Drawings

[0011] The accompanying drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0012] Figure 1 is a flowchart of a method for image fusion of substation equipment according to an embodiment of the present invention;

[0013] Figure 2 is a registration flowchart of an optional method for image fusion of substation equipment according to an embodiment of the present invention;

[0014] Figure 3 is a fusion flowchart of an optional method for image fusion of substation equipment according to an embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of an optional non - subsampled contourlet transform decomposition according to an embodiment of the present invention;

[0016] Figure 5 is a schematic diagram of an optional convolutional network according to an embodiment of the present invention;

[0017] Figure 6 is a comparison diagram of optional image processing results according to an embodiment of the present invention;

[0018] Figure 7 is a schematic diagram of an image fusion device for substation equipment according to an embodiment of the present invention. Detailed implementation manners

[0019] In order to enable those skilled in the art to better understand the present invention solution, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] First, for the convenience of understanding the embodiments of the present invention, some terms or nouns involved in the present invention will be explained below:

[0022] The reflection component refers to the component generated when light or a wave is reflected at the interface with a medium. In optics, when light travels from one medium to another, a part of the light is reflected back into the original medium, and this part of the light is the reflection component. The intensity of the reflection component depends on the angle of the incident light, the refractive index of the medium, and the properties of the medium.

[0023] The Fast Library for Approximate Nearest Neighbors (FLANN) is an algorithm used to find the nearest neighbor points in a large dataset. It uses a technique called Approximate Nearest Neighbors (ANN) to speed up the nearest neighbor search. The FLANN matcher can be used for image matching and 3D point cloud matching.

[0024] The KD-TREE is a data structure used to organize points in a K-dimensional space. It is a binary tree that divides the space into regions and stores points in an efficient way so that queries for nearest neighbors or range searches can be performed efficiently. This method is constructed by recursively dividing the space along the axis with the maximum point distribution and repeating this process for each subspace until all points are contained in separate leaf nodes, enabling efficient search and retrieval of points based on spatial relationships.

[0025] The Non-Separable Contourlet Transform is an extended form of the discrete contourlet transform, which is used to decompose a signal into sub-bands of multiple scales and directions and is applicable to signals and images with complex contour and texture features. The NSCT decomposition can extract multi-scale and multi-direction features of the signal, so image compression, image enhancement, and target detection can be performed.

[0026] The Laplace transform is a commonly used mathematical tool in signal processing and image processing for analyzing and processing signals and images. It is mainly used to detect features such as edges and textures in images and plays an important role in image enhancement and image analysis. The basic form of the Laplace transform is the sum of second-order derivatives, and by performing the Laplace transform on the original signal or image, features such as edges and textures in the image can be highlighted. It can better reflect the structure and features in the image and plays an important role in fields such as image recognition and target detection.

[0027] According to an embodiment of the present invention, a method embodiment for image fusion of substation equipment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0028] Figure 1 is a flowchart of the method for image fusion of substation equipment according to an embodiment of the present invention, as Figure 1 shown, the method includes the following steps:

[0029] Step S102, obtaining a target visible light image and a target infrared image of the substation equipment;

[0030] Optionally, the infrared image is formed by using the infrared radiation emitted by an object. The infrared image is not obtained by shooting, but is synthesized by the upper computer through the reflected wave after infrared radiation. Infrared radiation is an electromagnetic wave that cannot be seen by the human eye, and its wavelength is longer than that of visible light, usually between 0.7 micrometers (μm) and 1 millimeter (mm). When the temperature of the target object (such as the target equipment) is higher than absolute zero, a reflected wave of infrared radiation will be reflected on the target object. The reflected wave is received by the infrared sensor and sent to the upper computer, converted into an electrical signal, and through signal processing and image processing technologies, the infrared radiation is converted into a visible infrared image.

[0031] The imaging of visible light images is achieved by focusing light through an optical lens or lens onto a photosensitive element. Through processes such as refraction and scattering of light, the photosensitive element records that an electrical signal is generated under light irradiation. The electrical signal presents the details and colors of the image through signal processing technology, and finally a visible light image is formed.

[0032] In an optional embodiment, obtaining a target visible light image and a target infrared image of a power transformation device includes: obtaining an initial infrared image and an initial visible light image; respectively preprocessing the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image, where the preprocessing includes at least one of the following: filtering processing, grayscale processing, image enhancement processing; respectively performing feature point extraction processing on the first infrared image and the first visible light image to obtain a plurality of infrared feature points corresponding to the first infrared image and a plurality of visible light feature points corresponding to the first visible light image, where the plurality of infrared feature points and the plurality of visible light feature points correspond one by one; matching the plurality of infrared feature points and the plurality of visible light feature points to obtain a registration result; performing correction processing on the first infrared image and the first visible light image based on the registration result to obtain a target visible light image and a target infrared image, where the target visible light image and the target infrared image have the same size and pixels.

[0033] Optionally, the initial visible light image can be obtained by photographing the power transformation device with an image acquisition device, and the initial infrared image can be obtained according to the electromagnetic wave synthesis technology. Since the above two images are obtained by different methods, there will be phenomena such as different imaging sizes and different pixels. Therefore, before fusion, it is necessary to adjust the above two images to a state where the size and pixels are exactly the same. Specifically, after respectively performing filtering processing, grayscale processing, and image enhancement processing on the initial infrared image and the initial visible light image in sequence to obtain the corresponding first infrared image and first visible light image, find the infrared feature points corresponding to the first infrared image and the visible light feature points corresponding to the first visible light image, and by matching the infrared feature points with the visible light feature points, the matched infrared feature points and visible light feature points are corresponding to the same position, thereby realizing the adjustment of the first infrared image and the first visible light image. For example, taking the first infrared image as a comparison image, using the correspondence between the visible light feature points and the infrared feature points, operations such as stretching, cropping, and pixel point adjustment of the first visible light image are performed to realize the adjustment of the first visible light image to a state where the size and pixels are the same as those of the first infrared image. It should be noted that the first visible light image can also be used as a comparison image to adjust the first infrared image to a state where the size and pixels are the same as those of the first visible light image. The specific operation method is the same as the case where the first infrared image is used as a comparison image, and will not be elaborated here.

[0034] Optionally, filtering, grayscale processing, and image enhancement processing are sequentially performed on the initial infrared image and the initial visible light image, which can effectively remove noise or other unnecessary components in the above two images, thereby facilitating the subsequent extraction of infrared feature points and visible light feature points. Filtering is used to remove noise or unnecessary components in the signal while retaining useful information in the signal. The filtering effect is achieved by modifying the spectrum of the signal in the above two images using frequency domain filtering, resulting in enhanced quality of the above two images. Grayscale processing is used to convert a color image into a black-and-white image, achieving image compression, feature extraction, and image enhancement. In grayscale processing, the color value of each pixel in the initial infrared image and the initial visible light image is converted into the corresponding grayscale value, converting the color image into a grayscale image. The grayscale value represents the brightness or grayscale level of the pixel, and the grayscale value ranges from 0 to 255, where 0 represents black and 255 represents white. Image enhancement processing is used to improve the image using digital image processing techniques to enhance the quality and visual effect of the image. It includes contrast enhancement, sharpening enhancement, noise removal, color enhancement, and image restoration. Among them, contrast enhancement makes the details in the above two images clearer and more prominent by adjusting the brightness and contrast of the above two images; sharpening enhancement makes the above two images look clearer and sharper by enhancing the edges and details of the above two images; noise removal uses filters and noise reduction algorithms to remove noise in the above two images, making the above two images clearer and smoother; color enhancement makes the colors in the above two images more vivid and realistic by adjusting the color balance and saturation of the above two images; image restoration restores the original appearance of the above two images by repairing missing or damaged parts in the above two images.

[0035] In an optional embodiment, preprocessing is respectively performed on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image, including: performing Gaussian blur processing on the initial visible light image to obtain multiple second visible light images, where the scales of the multiple second visible light images are different; determining the reflection components respectively corresponding to the multiple second visible light images based on the incident light angle of the environmental light source corresponding to the shooting environment of the initial visible light image, where the reflection component is used to indicate the light source component after the environmental light source irradiates the target object in the corresponding second visible light image and the target object reflects the environmental light source; performing enhancement processing on the reflection components respectively corresponding to the multiple second visible light images to obtain multiple third visible light images; performing third fusion processing on the multiple third visible light images to obtain a fourth visible light image; and performing preprocessing on the initial infrared image and the fourth visible light image respectively to obtain a first infrared image and a first visible light image.

[0036] Optionally, an initial visible light image is obtained, wherein the visible light includes two sources, the first being the light source itself, i.e., the external ambient light source, which can be a lighting lamp or sunlight, etc., and the second being the object reflected light source, i.e., when the external ambient light source is irradiated on the target object, the ambient light source is reflected through the surface of the target object, resulting in a reflected light source. Since the initial visible light image after shooting only needs the imaging obtained by the reflected light source of the target object, it is necessary to minimize the influence of the ambient light source on the imaging, wherein, when minimizing the influence of the ambient light source on the imaging, the image processing theory algorithm Retinex can be used to decompose the initial visible light image into second visible light images of different scales, extract the reflection component in the second visible light image, and increase the reflection component of the reflected light source to the maximum, thereby increasing the influence of the reflected light source on the imaging, thereby achieving the effect of reducing the influence of the ambient light source on the imaging, and finally the third visible light image is fused, i.e., the images of different scales with the reflection component as the maximum influence are fused, and the fourth visible light image obtained is the image with the reduced influence of the ambient light source on the imaging.

[0037] In an optional embodiment, multiple infrared feature points and multiple visible light feature points are matched to obtain a registration result, including: calculating the error distances between the multiple infrared feature points and the corresponding visible light feature points; marking the infrared feature points and the corresponding visible light feature points whose error distances are less than a distance threshold as matching points, and marking the infrared feature points and the corresponding visible light feature points whose error distances are greater than the distance threshold as mismatching points; deleting the infrared feature points and the corresponding visible light feature points marked as mismatching points among the multiple infrared feature points and the multiple visible light feature points to obtain a registration result.

[0038] Optionally, in the process of obtaining infrared feature points and visible light feature points, although the first visible light image and the initial infrared image have been preprocessed before extracting the infrared feature points and the visible light feature points, there will inevitably be noise or occlusion, resulting in the extracted infrared feature points and the visible light feature points not being able to completely match each other. Therefore, it is necessary to eliminate the unmatched infrared feature points and visible light feature points so that the first infrared image and the first visible light image can be adjusted subsequently according to the registration results.

[0039] Specifically, using the Random Sample Consensus (RANSAC) algorithm, a set of randomly selected noise points are determined as outliers, the visible light feature points are used as inliers, and the inliers and outliers are fitted to form a feature point model, which is equivalent to a sample model. The infrared feature points are input into the feature point model as a subset, and the error distances between the infrared feature points and the outliers and inliers in the fitted feature point model are calculated respectively to obtain the inlier error distance and the outlier error distance. By comparing the inlier error distance and the outlier error distance, when the inlier error distance is less than or equal to the outlier error distance, the inlier error distance is compared with a threshold. The points with an inlier error distance less than the threshold are marked as matching points, otherwise they are marked as mismatching points. The mismatching points are deleted to obtain a set of matching points, which is the registration result. In this matching result, each infrared feature point can find a visible light feature point with a similar position. On this basis, the first infrared image and the first visible light image are corrected based on the registration result, and the images are corrected and registered according to the matching points. The obtained target visible light image and target infrared image have a higher degree of matching.

[0040] Step S104: Perform a first fusion process on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fused image.

[0041] Optionally, there is at least one target object in the target infrared image and the target visible light image. In most cases, there are multiple target objects, that is, there are multiple substation equipment. The multiple target objects are identified and extracted to form the contour extraction of the target objects in the target infrared image and the target visible light image. Among them, the contour can be to place all the multiple target objects in one contour to form an overall contour feature, or to extract the contours of the multiple target objects separately, that is, there will be multiple contour features.

[0042] In an alternative embodiment, a first fusion process is performed on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fusion image, including: performing a first decomposition process on the frequency subbands in the target infrared image to obtain an infrared low-frequency subband and an infrared high-frequency subband, where the frequency subbands carry regional contour information, the infrared low-frequency subband is the frequency subband in the target infrared image with a frequency less than or equal to a first frequency threshold, and the infrared high-frequency subband is the frequency subband in the target infrared image with a frequency greater than the first frequency threshold; performing a second decomposition process on the frequency subbands in the target visible light image to obtain a visible light low-frequency subband and a visible light high-frequency subband, where the visible light low-frequency subband is the frequency subband in the target visible light image with a frequency less than or equal to a second frequency threshold, and the visible light high-frequency subband is the frequency subband in the target visible light image with a frequency greater than the second frequency threshold; fusing the infrared low-frequency subband with the visible light low-frequency subband to obtain a target low-frequency subband, and fusing the infrared high-frequency subband with the visible light high-frequency subband to obtain a target high-frequency subband; performing an inverse decomposition transform on the target high-frequency subband and the target low-frequency subband to obtain the initial fusion image.

[0043] Optionally, the Non-Separable Contourlet Transform (NSCT) algorithm is used to perform the first decomposition process on the target infrared image to obtain an infrared high-frequency subband and an infrared low-frequency subband, and perform the second decomposition on the target visible light image to obtain a visible light high-frequency subband and a visible light low-frequency subband. For an image, the frequency subbands contain information of different spatial frequencies, that is, feature information such as contours and textures. Decomposing the frequency subbands means decomposing the contours of the target object in the image in different spaces. After decomposition, the infrared high-frequency subband is fused with the visible light high-frequency subband, and the infrared low-frequency subband is fused with the visible light low-frequency subband. The above method can fuse the contour features of the infrared target object and the contour features of the visible light target object in the high-frequency spatial range, and better retain the integrity of the contour features in the high-frequency spatial range by not mixing the contour features in the low-frequency space. The target high-frequency subband is obtained, and the same operation is performed on the low-frequency subband to obtain the target low-frequency subband. The target high-frequency subband only carries the contour features in the high-frequency spatial range, and the target low-frequency subband only carries the contour features in the low-frequency spatial range. It is necessary to fuse the two to obtain the final contour information features in all spaces, and perform an inverse decomposition transform on the frequency subbands in all the above spaces to finally convert them into the initial fusion image.

[0044] Step S106: Perform edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image;

[0045] Optionally, the purpose of the above contour feature extraction and fusion is to identify and recognize the target object, that is, to find the target object and extract the edges of the found target object, that is, to extract the details of the outer edge of the target object. For example, the target object is not a square power transformation device, but has a curved and folded shape. The process of extracting the curved and folded shape is the process of extracting the details of the outer edge of the target object. By extracting the edges of the target object, it is equivalent to enhancing the edges of the target object, achieving a better effect of clear edges of the target object, and thus making the image clearer.

[0046] In an optional embodiment, edge extraction processing is respectively performed on the target objects included in the initial fusion image and the target infrared image to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image, including: respectively performing third decomposition processing on the initial fusion image and the target infrared image to obtain a plurality of fusion images and a plurality of infrared images, where the scales of the plurality of fusion images are different, and the scales of the plurality of infrared images are different; respectively performing upsampling processing on the plurality of fusion images and the plurality of infrared images to obtain upsampled images respectively corresponding to the plurality of fusion images and upsampled images respectively corresponding to the plurality of infrared images; performing ascending order sorting processing on the upsampled images respectively corresponding to the plurality of fusion images according to the scale to obtain a first sorting result; performing ascending order sorting processing on the upsampled images respectively corresponding to the plurality of infrared images according to the scale to obtain a second sorting result; sequentially performing addition processing on the upsampled images respectively corresponding to the plurality of fusion images according to the first sorting result to obtain a fusion edge image; sequentially performing addition processing on the upsampled images respectively corresponding to the plurality of infrared images according to the second sorting result to obtain an infrared edge image.

[0047] Optionally, perform Laplace transform on the initial fusion image and the target infrared image to achieve Laplace pyramid decomposition of the initial fusion image and the target infrared image, obtaining a plurality of fusion images of different scales and a plurality of infrared images of different scales, where the plurality of different scales are sorted in descending order according to the scale size, that is, the smaller scale is above and the larger scale is below. Perform upsampling operation on the fusion image of each scale, where upsampling is used to enhance the sampling rate of image features, that is, sampling more feature points within a fixed area, thereby improving the resolution and clarity of the image. According to the scale sorting, from top to bottom, sequentially perform addition processing on the upsampled images respectively corresponding to the plurality of fusion images to obtain a fusion edge image after extracting edge features. The same processing is performed on the target infrared image to obtain an infrared edge image. This method effectively retains more edge features in the target object, enhances the edges of the target object, that is, realizes the segmentation between the target object and the image background in the image, making the boundary between the target object and the image background clear and obvious, and thus making the fusion edge image and the infrared edge image clearer.

[0048] Step S108, determine the first weight mapping coefficients corresponding to the feature points included in the initial fusion image and the second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively;

[0049] Optionally, the first weight mapping coefficient is used to indicate the importance degree of the feature points in the initial fusion image, that is, the influence degree on the subsequent fusion process, and the second weight mapping coefficient is used to indicate the importance degree of the feature points in the target infrared image, that is, the influence degree on the subsequent fusion process. In this embodiment, the weight mapping coefficient is represented in the form of a weight vector, that is, the obtained first weight mapping coefficient and the second weight mapping coefficient are vector values.

[0050] In an optional embodiment, determining the first weight mapping coefficients corresponding to the feature points included in the initial fusion image and the second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively includes: obtaining a convolutional network, where the convolutional network includes two branch networks, and each of the two branch networks includes three convolutional layers and one max pooling layer; performing convolutional processing on the initial fusion image and the target infrared image by using the convolutional network to obtain the first weight mapping coefficients corresponding to the feature points included in the initial fusion image and the second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively, where one of the two branch networks takes the initial fusion image as the input, and the other branch network of the two branch networks takes the target infrared image as the input.

[0051] Optionally, use a Convolutional Neural Network (CNN) to perform convolutional operations to obtain the first weight mapping coefficient and the second weight mapping coefficient. The purpose of the convolutional operation is to extract the features in the input initial fusion image and the target infrared image: the first convolutional layer can only extract some low-level features, such as features like edges, lines, and corners, and more layers of the network can iteratively extract more complex features from the low-level features, such as features like internal surface shapes, concave and flat states, etc. The CNN network includes two branch networks, and each of the branch networks includes three convolutional layers and one max pooling layer. Specifically, the kernel size of the convolutional layer is 3×3, the stride is set to 1, the kernel size of the max pooling layer is set to 2×2, and the stride is set to 2. After the preliminary fusion image and the target infrared image pass through the convolutional layer and the max pooling layer respectively, each branch network can obtain 256 feature maps. Connect the 256 feature maps together and fully connect them to a predetermined 256-dimensional feature vector respectively, and output a two-dimensional vector that is fully connected to the 256-dimensional feature vector, which is the first weight mapping coefficient and the second weight mapping coefficient.

[0052] Step S110: Perform a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image.

[0053] Optionally, the first weight mapping coefficient reflects the influence degree of the initial fusion image on the target fusion image. Since the fusion edge image is obtained by extracting the edge features of the initial fusion image, the feature points in the fusion edge image, except for the edge feature points of the target object, are exactly the same as the feature points in the initial fusion image. Therefore, the first weight mapping coefficient also reflects the influence degree of the feature points in the fusion edge image, except for the edge feature points of the target object, on the target fusion image. It can be seen from this that the second weight mapping coefficient also reflects the influence degree of the feature points in the infrared edge image, except for the edge feature points of the target object, on the target fusion image. Performing a second fusion process on the infrared edge image and the fusion edge image based on the first weight mapping coefficient and the second weight mapping coefficient, that is, adding the feature points other than the edge features to the already extracted edge features, that is, first extracting the edge of the target object and then filling the inside of the target object edge to form a complete target fusion image, effectively ensuring that in the target fusion image, the edge of the target object is clear, and at the same time, the inside of the target object and the background outside the target object are also clear, thus ensuring the clarity of the entire target fusion image.

[0054] In an optional embodiment, performing a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image includes: calculating the similarity between the first weight mapping coefficient and the second weight mapping coefficient; in the case where the similarity is greater than or equal to a preset similarity threshold, fusing the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image; or in the case where the similarity is less than the preset similarity threshold, fusing the feature points included in the infrared edge image and the fusion edge image respectively based on the maximum value of the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image.

[0055] Optionally, in the process of fusing the fused edge image and the infrared edge image based on the first weight mapping coefficient and the second weight mapping coefficient, since the imaging of the infrared image is obtained through the reflection of electromagnetic waves, and the electromagnetic waves carry some energy, the influence value of the second weight mapping coefficient corresponding to the infrared edge image will be larger than the actual value, although the amount of increase is relatively weak, but only the larger value of the first weight mapping coefficient and the second weight mapping coefficient is selected, that is, the feature points in the infrared edge image and the fused edge image that have a greater impact on the target fused image are compared as the feature points of the target fused image, and the fusion coefficient cannot be selected comprehensively. By calculating the similarity, when the similarity between the first weight mapping coefficient and the second weight mapping coefficient is less than the preset similarity threshold, it means that the energy carried in the electromagnetic wave has little influence on the subsequent process, and the maximum value of the first weight mapping coefficient and the second weight mapping coefficient is directly selected as the fusion coefficient, and the infrared edge image and the fused edge image are fused. However, when the similarity between the first weight mapping coefficient and the second weight mapping coefficient is greater than the preset similarity threshold, it means that the energy carried in the electromagnetic wave has a greater influence on the subsequent process, and the first weight mapping coefficient and the second weight mapping coefficient are weighted averaged to obtain the fusion coefficient, and then the infrared edge image and the fused edge image are fused to finally obtain the target fused image.

[0056] Optionally, a method for identifying faults of substation equipment may also be provided, the method using a target fusion image obtained based on any of the above-mentioned substation equipment image fusion methods, the method comprising: identifying the target fusion image, detecting whether there is a target feature in the target fusion image, and obtaining a detection result, wherein the target feature is used to indicate whether there is a predetermined fault in the target fusion image, and the target feature may be a crack, wear, etc.; determining whether there is a fault in the substation equipment based on the detection result. Specifically, when the detection result indicates that there is a target feature in the target fusion image, it is determined that there is a fault in the substation equipment. Since the target fusion image obtained based on the above method has a higher clarity, the fault detection result obtained on this basis is more accurate and reliable, and can more timely discover the safety hazards existing in the operation of the substation equipment.

[0057] Through the above steps S102 to S110, the purpose of extracting the edge feature values of the target object can be achieved, thereby achieving the technical effect of making the fused image clearer, and further solving the technical problem of poor clarity of the fused image obtained by the substation image fusion method in the related art.

[0058] Based on the above embodiments and optional embodiments, the present invention proposes an optional implementation mode: Figure 2 is a registration flow chart of an optional substation equipment image fusion method according to an embodiment of the present invention, Figure 3It is a fusion flow chart of an optional image fusion method for substation equipment according to an embodiment of the present invention. The method includes:

[0059] S1: Obtain an initial visible light image and an initial infrared image, and process the initial visible light image using the multi-scale image processing theory algorithm, the Retinex algorithm.

[0060] S2: Perform Gaussian filtering, grayscale conversion, and Canny edge extraction on the initial infrared light image and the processed initial visible light image respectively to obtain a first visible light image and a first infrared image.

[0061] S3: Use the feature extraction algorithm ASIFT to simulate affine images and perform feature point extraction and matching on the first visible light image and the first infrared image.

[0062] S31: By setting different rotation angles and tilt angles θ, perform convolution calculations on the first visible light image and the first infrared image in the following manner:

[0063]

[0064] where: I is the first visible light image and the first infrared image, t is the number of rotations, is the image after t rotations by the angle;

[0065] S32: Select affine images with the same direction at the same points as the initial visible light image and the initial infrared image, and similar scales in the tilt direction and perform feature extraction and matching of the simulated images using the ASIFT-like algorithm.

[0066] S33: Use the Fast Library for Approximate Nearest Neighbors matcher FLANN to match infrared feature points and visible light feature points.

[0067] S4: Use the RANSAC algorithm to detect mismatched points, and add the Random Sample Consensus (RANSAC) algorithm to eliminate mismatched points.

[0068] Specifically, use a KD-Tree to place infrared feature points and visible light feature points in the n-dimensional space R respectively nDivide it into multiple regions, and arbitrarily set a point in the KD-TREE as the target point. Among them, the first target point in the KD-TREE corresponding to the infrared feature point should have the same position as the second target point in the KD-TREE corresponding to the visible light feature point. Calculate the first distance between multiple infrared feature points and the first target point in the KD-TREE, and calculate the second distance between multiple visible light feature points and the second target point. Identify the error between the first distance and the second distance. If it is within the threshold range, mark it as a matching point; if the error between the first distance and the second distance is not within the threshold range, mark it as a mismatching point.

[0069] Through the recursive search of the KD-TREE from top to bottom, compare the values of the first distance and the second distance based on a certain specific dimension, and distinguish the regions where the infrared feature points and the visible light feature points are located; loop through the comparison of the first distance and the second distance in multiple regions until the search is completed.

[0070] S5. Decompose the initial infrared image and the initial visible light image through non-separable contourlet transform (NSCT) to obtain the infrared high-frequency subband, infrared low-frequency subband, visible light high-frequency subband, and visible light low-frequency subband, as Figure 4 shown, Figure 4 which is an optional non-downsampled contourlet transform decomposition schematic diagram according to an embodiment of the present invention.

[0071] Specifically, use the non-downsampled pyramid filter bank (NSPFB) to perform multi-scale decomposition on the initial visible light image and the initial infrared image respectively to obtain low-frequency components and high-frequency components; use NSDFB to perform multi-directional decomposition on the high-frequency components to obtain infrared high-frequency subbands and visible light high-frequency subbands with different scales and different directions; repeat the above operations on the low-frequency components to obtain multiple visible light low-frequency subbands and infrared low-frequency subbands corresponding to the initial visible light image and the initial infrared image respectively.

[0072] S6. Fuse the infrared low-frequency subband and the visible light low-frequency subband in a guided filtering manner to obtain the target low-frequency subband F L ; fuse the infrared high-frequency subband and the visible light high-frequency subband in a guided filtering manner to obtain the target high-frequency subband F H ;

[0073] S7. Obtain the initial fused image F1 by performing the inverse NSCT transformation on the target low-frequency subband F L and the target high-frequency subband F H ;

[0074] S8. Feed the initial infrared image and the initial fused image into two branches of the convolutional network respectively to generate the weight mapping coefficient w. Here, the weight mapping coefficient w is a vector value, and the weight mapping coefficient w includes the first weight mapping coefficient corresponding to the feature points in the initial fused image and the second weight mapping coefficient corresponding to the feature points in the target infrared image.

[0075] Specifically, the convolutional network has two identical branches, and each branch consists of three convolutional layers and one max pooling layer. The purpose of the convolutional operation is to extract different features of the input: the first convolutional layer can only extract some low-level features, such as edges, lines, and corners, etc. More layers of the network can iteratively extract more complex features from the low-level features. In the convolutional network of this embodiment, the kernel size and stride of each convolutional layer are set to 3×3 and 1 respectively. The kernel size and stride of the max pooling layer are set to 2×2 and 2 respectively. After the two branches go through the convolutional and pooling operations, each branch can obtain 256 feature maps. Connect the 256 feature maps, and then fully connect them with a 256-dimensional feature vector to output a two-dimensional vector that is fully connected with the 256-dimensional feature vector, as Figure 5 shown, Figure 5 is a schematic diagram of an optional convolutional network according to an embodiment of the present invention.

[0076] S9. Perform Laplace transform decomposition on the initial infrared image and the initial fused image respectively and extract edge features to obtain a fused edge image and an infrared edge image;

[0077] S10. Based on the weight mapping coefficient w, fuse the fused edge image and the infrared edge image. Since the imaging mechanisms of infrared and visible light images are different, only selecting the larger value of the local region energy cannot comprehensively select the fusion coefficient. Here, the local region energy corresponds to the weight mapping coefficient, and the larger value of the local region energy is the larger value between the first weight mapping coefficient and the second weight mapping coefficient.

[0078] Calculate the similarity value R. When R≥TH, adopt the "weighted average" fusion mode of the weight mapping coefficient w; when R<TH, then adopt the method of local energy comparison, and select the feature points corresponding to the larger local energy value in the fused edge image and the infrared edge image for fusion, where TH is a preset threshold.

[0079] It should be noted that the feature points with larger local energy in the initial fused image are the feature points with larger local energy in the fused edge image. The fused edge image is obtained from the initial fused image, and only the edges of the target object are extracted, and the remaining local energy remains the same.

[0080] Specifically, the first weight mapping coefficient is equal to the local region energy in the fused edge image F1 The relationship between the two is expressed by the following formula:

[0081]

[0082] Among them, (x, y) is the coordinate position of the feature point in the fused edge image, x + m is the position where the abscissa x of the feature point is offset by m distances, y + n is the position where the ordinate y of the feature point is offset by n distances, and L{F1} l is the fused edge image obtained by performing Laplace transform on the initial fused image.

[0083] The second weight mapping coefficient corresponding to the feature point in the target infrared image is equal to the local region energy in the target infrared image

[0084]

[0085] Among them, (x, y) is the coordinate position of the feature point in the infrared edge image I, x + m is the position where the x - coordinate of the feature point is shifted by m distances, y + n is the position where the y - coordinate of the feature point is offset by n distances, and L{I} l is the infrared edge image obtained by performing Laplace transform on the target infrared image.

[0086] Calculate the similarity R in the following way:

[0087]

[0088] Determine the fused sub - band coefficient L{F} in the following way l (x,y):

[0089]

[0090] Among them: G(x, y) is the Gaussian coefficient, and the specific expression of the Gaussian coefficient is:

[0091]

[0092] σ is called the scale parameter of the Gaussian coefficient.

[0093] S11, based on the Laplacian pyramid fusion technology, uses the fused sub - band coefficient L{F} l (x,y) to fuse the infrared edge image and the fused edge image to obtain the target fused image F.

[0094] Through the above steps S1 to S11, problems such as the inability to accurately register visible light images and infrared images and the unclear fused images after data fusion in the existing defect and fault diagnosis process of substation equipment are solved. Image registration is achieved by using the feature extraction algorithm (MSR-ASIFT). The edge features of the image are enhanced by combining the non-subsampled contourlet transform decomposition algorithm (NSCT) and the convolutional neural network (CNN) for information complementarity, so as to achieve better fusion of infrared images and visible light images of substation equipment and obtain a clearer target fused image. Among them, Figure 6 is an optional comparison diagram of image processing results according to an embodiment of the present invention. Other related methods include discrete wavelet transform processing, brain functional imaging technology, application server technology, convolutional neural network, super-resolution image reconstruction technology, and discrete wavelet transform + convolutional network processing. It can be seen that compared with the fused images obtained in the related art, the target fused image obtained according to the embodiment of the present invention has higher clarity and better image fusion effect.

[0095] In this embodiment, a substation equipment image fusion device is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the terms "module" and "device" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0096] According to an embodiment of the present invention, a device embodiment for implementing the above substation equipment image fusion method is also provided. Figure 7 is a schematic structural diagram of a substation equipment image fusion device according to an embodiment of the present invention. As Figure 7 shown, the above substation equipment image fusion device includes: an acquisition module 700, an edge processing module 702, a weight mapping coefficient determination module 704, and a fusion module 706, where:

[0097] The acquisition module 700 is used to acquire the target visible light image and the target infrared image of the substation equipment;

[0098] The edge processing module 702 is connected to the acquisition module 700 and is used to perform first fusion processing on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fused image; perform edge extraction processing on the target objects included in the initial fused image and the target infrared image respectively to obtain a fused edge image corresponding to the initial fused image and an infrared edge image corresponding to the target infrared image;

[0099] A weight mapping coefficient determination module 704, connected to the edge processing module 702, is configured to determine first weight mapping coefficients corresponding to the feature points included in the initial fusion image and second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively;

[0100] A fusion module 706, connected to the weight mapping coefficient determination module 704, is configured to perform a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image.

[0101] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following manner: the above-mentioned various modules can be located in the same processor; or, the above-mentioned various modules are located in different processors in any combination.

[0102] It should be noted here that the above-mentioned acquisition module 700, edge processing module 702, weight mapping coefficient determination module 704, and fusion module 706 correspond to steps S102 to S110 in the embodiment. The examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the content disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules, as part of the device, can run in a computer terminal.

[0103] It should be noted that the optional or preferred implementation manners of this embodiment can be referred to the relevant descriptions in the embodiment, and will not be elaborated here.

[0104] The above-mentioned substation equipment image fusion device may further include a processor and a memory. The above-mentioned acquisition module 700, edge processing module 702, weight mapping coefficient determination module 704, and fusion module 706, etc. are all stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to implement corresponding functions.

[0105] The processor contains a kernel, and the kernel retrieves the corresponding program modules from the memory. One or more kernels can be set. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one storage chip.

[0106] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the above-mentioned non-volatile storage medium includes a stored program, wherein when the above-mentioned program runs, it controls the device where the non-volatile storage medium is located to execute any one of the above-mentioned substation equipment image fusion methods.

[0107] Optionally, in this embodiment, the non-volatile storage medium may be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group. The non-volatile storage medium includes a stored program.

[0108] Optionally, when the program runs, it controls the device where the non-volatile storage medium is located to perform the following functions: obtaining a target visible light image and a target infrared image of a power transformation device; performing a first fusion process on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fusion image; performing edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image; determining first weight mapping coefficients corresponding to the feature points included in the initial fusion image respectively, and second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively; based on the first weight mapping coefficients and the second weight mapping coefficients, performing a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively to obtain a target fusion image.

[0109] According to an embodiment of the present application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, where the program, when running, executes any one of the above power transformation device image fusion methods.

[0110] According to an embodiment of the present application, an embodiment of a computer program product is also provided. Optionally, in this embodiment, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the program of any one of the above power transformation device image fusion method steps.

[0111] Optionally, when the above computer program product is executed on a data processing device, it is adapted to execute a program initialized with the following method steps: obtaining a target visible light image and a target infrared image of a power transformation device; performing a first fusion process on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fusion image; performing edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image; determining first weight mapping coefficients corresponding to the feature points included in the initial fusion image respectively, and second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively; based on the first weight mapping coefficients and the second weight mapping coefficients, performing a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively to obtain a target fusion image.

[0112] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored on the memory and executable on the processor. When the processor executes the program, the following steps are implemented: obtaining a target visible light image and a target infrared image of a power transformation device; performing first fusion processing on the contour features respectively corresponding to the target visible light image and the target infrared image to obtain an initial fusion image; performing edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image; determining first weight mapping coefficients respectively corresponding to the feature points included in the initial fusion image, and second weight mapping coefficients respectively corresponding to the feature points included in the target infrared image; and performing second fusion processing on the feature points respectively included in the infrared edge image and the fusion edge image based on the first weight mapping coefficients and the second weight mapping coefficients to obtain a target fusion image.

[0113] The order of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments.

[0114] In the above embodiments of the present invention, the descriptions of the respective embodiments have their own focuses. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0115] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the above module division can be a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces, and the indirect coupling or communication connection of modules or modules can be in an electrical or other form.

[0116] The modules described above as separate components may or may not be physically separated. The components displayed as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, in each embodiment of the present invention, the functional modules can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0118] If the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present invention. The aforementioned non-volatile storage medium includes: various media such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0119] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for image fusion of substation equipment, characterized in that, Including: Obtaining a target visible light image and a target infrared image of a power transformation device; Performing first fusion processing on the contour features respectively corresponding to the target visible light image and the target infrared image to obtain an initial fusion image; performing edge extraction processing on the target objects included in the initial fusion image and the target infrared image respectively to obtain a fusion edge image corresponding to the initial fusion image and an infrared edge image corresponding to the target infrared image; Determining first weight mapping coefficients respectively corresponding to the feature points included in the initial fusion image, and second weight mapping coefficients respectively corresponding to the feature points included in the target infrared image; Performing second fusion processing on the feature points respectively included in the infrared edge image and the fusion edge image based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image; Wherein, the determining the first weight mapping coefficients respectively corresponding to the feature points included in the initial fusion image, and the second weight mapping coefficients respectively corresponding to the feature points included in the target infrared image includes: obtaining a convolutional network, wherein the convolutional network includes two branch networks, and the two branch networks respectively include three convolutional layers and one max pooling layer; using the convolutional network to perform convolutional processing on the initial fusion image and the target infrared image to obtain the first weight mapping coefficients respectively corresponding to the feature points included in the initial fusion image, and the second weight mapping coefficients respectively corresponding to the feature points included in the target infrared image, wherein one of the two branch networks takes the initial fusion image as an input, and the other branch network of the two branch networks takes the target infrared image as an input; The performing second fusion processing on the feature points respectively included in the infrared edge image and the fusion edge image based on the first weight mapping coefficient and the second weight mapping coefficient to obtain the target fusion image includes: calculating the similarity between the first weight mapping coefficient and the second weight mapping coefficient; in the case where the similarity is greater than or equal to a preset similarity threshold, performing fusion on the feature points respectively included in the infrared edge image and the fusion edge image based on the first weight mapping coefficient and the second weight mapping coefficient to obtain the target fusion image; or in the case where the similarity is less than the preset similarity threshold, performing fusion on the feature points respectively included in the infrared edge image and the fusion edge image based on the maximum value of the first weight mapping coefficient and the second weight mapping coefficient to obtain the target fusion image.

2. The method according to claim 1, characterized in that, The obtaining the target visible light image and the target infrared image of the power transformation device includes: Obtaining an initial infrared image and an initial visible light image; Performing preprocessing on the initial infrared image and the initial visible light image respectively to obtain a first infrared image and a first visible light image, wherein the preprocessing includes at least one of the following: filtering processing, grayscale processing, image enhancement processing; Feature point extraction processing is respectively performed on the first infrared image and the first visible light image to obtain a plurality of infrared feature points corresponding to the first infrared image and a plurality of visible light feature points corresponding to the first visible light image, wherein the plurality of infrared feature points and the plurality of visible light feature points correspond one by one; The plurality of infrared feature points and the plurality of visible light feature points are matched to obtain a registration result; Based on the registration result, correction processing is performed on the first infrared image and the first visible light image to obtain the target visible light image and the target infrared image, wherein the target visible light image and the target infrared image have the same size and pixels.

3. The method according to claim 2, characterized in that, The preprocessing of the initial infrared image and the initial visible light image respectively to obtain the first infrared image and the first visible light image includes: Performing Gaussian blur processing on the initial visible light image to obtain a plurality of second visible light images, wherein the scales of the plurality of second visible light images are different; Based on the incident light angle of the environmental light source corresponding to the shooting environment of the initial visible light image, determining the reflection components respectively corresponding to the plurality of second visible light images, wherein the reflection component is used to indicate the light source component after the environmental light source irradiates the target object in the corresponding second visible light image and the target object reflects the environmental light source; Performing enhancement processing on the reflection components respectively corresponding to the plurality of second visible light images to obtain a plurality of third visible light images; Performing third fusion processing on the plurality of third visible light images to obtain a fourth visible light image; Performing the preprocessing on the initial infrared image and the fourth visible light image respectively to obtain the first infrared image and the first visible light image.

4. The method according to claim 2, wherein The matching of the plurality of infrared feature points and the plurality of visible light feature points to obtain a registration result includes: Calculating the error distances between the plurality of infrared feature points and the corresponding visible light feature points respectively; Marking the infrared feature points and the corresponding visible light feature points with an error distance less than the distance threshold as matching points, and marking the infrared feature points and the corresponding visible light feature points with an error distance greater than the distance threshold as mismatched points; Deleting the infrared feature points marked as the mismatched points and the corresponding visible light feature points among the plurality of infrared feature points and the plurality of visible light feature points to obtain the registration result.

5. The method according to claim 1, wherein The first fusion processing of the contour features respectively corresponding to the target visible light image and the target infrared image to obtain an initial fusion image includes: Performing first decomposition processing on the frequency subbands in the target infrared image to obtain an infrared low-frequency subband and an infrared high-frequency subband, wherein the frequency subbands carry regional contour information, the infrared low-frequency subband is the frequency subband in the target infrared image with a frequency less than or equal to the first frequency threshold, and the infrared high-frequency subband is the frequency subband in the target infrared image with a frequency greater than the first frequency threshold; Perform a second decomposition process on the frequency sub-bands in the target visible light image to obtain a visible light low-frequency sub-band and a visible light high-frequency sub-band, where the visible light low-frequency sub-band is the frequency sub-band in the target visible light image with a frequency less than or equal to a second frequency threshold, and the visible light high-frequency sub-band is the frequency sub-band in the target visible light image with a frequency greater than the second frequency threshold; Fuse the infrared low-frequency sub-band with the visible light low-frequency sub-band to obtain a target low-frequency sub-band, and fuse the infrared high-frequency sub-band with the visible light high-frequency sub-band to obtain a target high-frequency sub-band; Perform an inverse decomposition transform on the target high-frequency sub-band and the target low-frequency sub-band to obtain the initial fusion image.

6. The method according to claim 1, wherein The edge extraction process for the target objects included in the initial fusion image and the target infrared image respectively to obtain the fusion edge image corresponding to the initial fusion image and the infrared edge image corresponding to the target infrared image includes: Perform a third decomposition process on the initial fusion image and the target infrared image respectively to obtain a plurality of fusion images and a plurality of infrared images, where the scales of the plurality of fusion images are different, and the scales of the plurality of infrared images are different; Perform an upsampling process on the plurality of fusion images and the plurality of infrared images respectively to obtain the upsampled images corresponding to the plurality of fusion images respectively and the upsampled images corresponding to the plurality of infrared images respectively; Perform an ascending order sorting process on the upsampled images corresponding to the plurality of fusion images respectively according to the scale to obtain a first sorting result; perform the ascending order sorting process on the upsampled images corresponding to the plurality of infrared images respectively according to the scale to obtain a second sorting result; Perform an addition process on the upsampled images corresponding to the plurality of fusion images respectively in sequence according to the first sorting result to obtain the fusion edge image; Perform an addition process on the upsampled images corresponding to the plurality of infrared images respectively in sequence according to the second sorting result to obtain the infrared edge image.

7. An image fusion device for substation equipment, characterized in that, Including: An acquisition module for acquiring a target visible light image and a target infrared image of a substation equipment; An edge processing module for performing a first fusion process on the contour features corresponding to the target visible light image and the target infrared image respectively to obtain an initial fusion image; performing an edge extraction process on the target objects included in the initial fusion image and the target infrared image respectively to obtain the fusion edge image corresponding to the initial fusion image and the infrared edge image corresponding to the target infrared image; A weight mapping coefficient determination module for determining the first weight mapping coefficients corresponding to the feature points included in the initial fusion image respectively and the second weight mapping coefficients corresponding to the feature points included in the target infrared image respectively; A fusion module for performing a second fusion process on the feature points included in the infrared edge image and the fusion edge image respectively based on the first weight mapping coefficient and the second weight mapping coefficient to obtain a target fusion image; Among them, the weight mapping coefficient determination module is further configured to: obtain a convolutional network, where the convolutional network includes two branch networks, and the two branch networks respectively include three convolutional layers and a max pooling layer; use the convolutional network to perform convolutional processing on the initial fusion image and the target infrared image to obtain the first weight mapping coefficients corresponding to the feature points included in the initial fusion image and the second weight mapping coefficients corresponding to the feature points included in the target infrared image, where one of the two branch networks takes the initial fusion image as an input, and the other branch network of the two branch networks takes the target infrared image as an input; The fusion module is further configured to: calculate the similarity between the first weight mapping coefficient and the second weight mapping coefficient; in the case where the similarity is greater than or equal to a preset similarity threshold, fuse the feature points included in the infrared edge image and the fusion edge image based on the first weight mapping coefficient and the second weight mapping coefficient to obtain the target fusion image; or in the case where the similarity is less than the preset similarity threshold, fuse the feature points included in the infrared edge image and the fusion edge image based on the maximum value of the first weight mapping coefficient and the second weight mapping coefficient to obtain the target fusion image.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the substation equipment image fusion method according to any one of claims 1 to 6.

Images