Image fusion method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an image fusion method and device, equipment and a storage medium. The method comprises the following steps: acquiring a first low-frequency sub-image, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image; obtaining a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm; obtaining a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image by using a preset fusion weight algorithm; and performing infrared and visible light fusion processing according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fusion image. Through the method, the target fusion image fusing the infrared and visible light images integrates the advantages of the two images, and the accuracy and efficiency of substation inspection are improved.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to an image fusion method and device, equipment and storage medium. Background Art

[0002] Inspection work is crucial in the daily operation and maintenance of substations. Inspection involves collecting infrared images and visible light images of the inspection environment. Infrared images can reflect the temperature distribution of equipment by detecting the thermal radiation of objects, and have unique advantages in discovering potential problems such as overheating failures of equipment. However, infrared images often have low resolution and lack equipment details and texture information; visible light images have high resolution and can clearly present the appearance, shape and other detailed features of the equipment, but cannot directly display the temperature information of the equipment.

[0003] Therefore, when confirming the inspection results, it is often necessary to compare two images to complete the confirmation, which reduces the inspection efficiency. Summary of the invention

[0004] The main purpose of the present invention is to provide an image fusion method and device, equipment and storage medium, which can solve the low inspection efficiency in the prior art.

[0005] To achieve the above object, the present invention provides an image fusion method in a first aspect, the method comprising:

[0006] Acquire an image to be fused, wherein the image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image, and a second high-frequency sub-image, wherein the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; and the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection;

[0007] Using a preset fusion coefficient algorithm, a first low-frequency sub-image, and a second low-frequency sub-image, a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image are obtained, wherein the fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy and the regional gradient of the low-frequency sub-image and the fusion coefficient;

[0008] Using a preset fusion weight algorithm, a first high-frequency sub-image, and a second high-frequency sub-image, a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image are obtained, wherein the fusion weight algorithm is used to reflect the mapping relationship between the firing times of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image in the iteration process and the fusion weight;

[0009] The infrared and visible light images are fused according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fused image.

[0010] In a feasible implementation, the step of obtaining the image to be fused includes:

[0011] Acquire the infrared image and the visible light image;

[0012] De-noising the infrared image and the visible light image to obtain a de-noised infrared image and a visible light image;

[0013] Performing image enhancement processing on the denoised infrared image and visible light image respectively to obtain enhanced infrared image and visible light image;

[0014] Performing image registration processing using the enhanced infrared image and visible light image to obtain a registered infrared image and visible light image;

[0015] The non-subsampled shearlet transform is used to perform multi-scale decomposition processing on the registered infrared image and the visible light image respectively to obtain the image to be fused.

[0016] In a feasible implementation, the fusion processing of the infrared and visible light images is performed according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image, and the second high-frequency sub-image to obtain a target fused image, including:

[0017] Based on the first fusion coefficient and the second fusion coefficient, the first low-frequency sub-image and the second low-frequency sub-image at each decomposition scale are fused to obtain a low-frequency fused image of infrared and visible light at each decomposition scale;

[0018] Based on the first fusion weight and the second fusion weight, the first high-frequency sub-image and the second high-frequency sub-image at each decomposition scale are fused to obtain a high-frequency fused image of infrared and visible light at each decomposition scale;

[0019] The low-frequency fusion image and the high-frequency fusion image at each decomposition scale are used to perform image reconstruction processing by inverse non-subsampled shearlet transform to obtain a low-frequency reconstructed image and a high-frequency reconstructed image;

[0020] The low-frequency reconstructed image and the high-frequency reconstructed image are added together to obtain the target fused image.

[0021] In a feasible implementation, the using a preset fusion coefficient algorithm, the first low-frequency sub-image, and the second low-frequency sub-image to obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image includes:

[0022] According to a preset region division method, the first low-frequency sub-image and the second low-frequency sub-image are respectively divided into regions to obtain a first image block of the first low-frequency sub-image and a second image block of the second low-frequency sub-image in each region;

[0023] Calculating the square sum of pixel values ​​using the pixel values ​​of each pixel point in the first image block to obtain the first region energy of the first image block in each region; calculating the square sum of pixel values ​​using the pixel values ​​of each pixel point in the second image block to obtain the second region energy of the human image block in each region;

[0024] Calculating an average gradient value using the pixel values ​​of each pixel point in the first image block to obtain a first regional gradient of the first image block in each region; calculating an average gradient value using the pixel values ​​of each pixel point in the second image block to obtain a second regional gradient of the second image block in each region;

[0025] Based on the first region energy, the second region energy, the first region gradient, the second region gradient and the fusion coefficient algorithm of each region, a first fusion coefficient of each region of the first low-frequency sub-image and a second fusion coefficient of each region of the second low-frequency sub-image are obtained.

[0026] In a feasible implementation, the neurons correspond one-to-one to the pixel positions of the high-frequency sub-images, and the method of using a preset fusion weight algorithm, the first high-frequency sub-image, and the second high-frequency sub-image to obtain a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image includes:

[0027] Determining the firing times of each neuron during the iteration of the pulse coupled neural network;

[0028] According to each of the ignition times and a preset fusion weight algorithm, a first fusion weight of each pixel position of the first high-frequency sub-image and a second fusion weight of each pixel position of the second high-frequency sub-image are obtained.

[0029] In a feasible implementation, the denoising process is performed on the infrared image and the visible light image respectively to obtain the denoised infrared image and the visible light image, including:

[0030] For each pixel in the infrared image and the visible light image:

[0031] Obtaining a pixel region adjacent to the pixel based on an initial filtering window;

[0032] Determine the median, maximum and minimum values ​​of pixels in the adjacent pixel area;

[0033] If the original pixel value of the pixel point is between the maximum value and the minimum value, and the absolute value of the difference between the median value and the pixel value is less than a preset pixel threshold, the original pixel value of the pixel point is retained;

[0034] If the original pixel value of the pixel point is not between the maximum value and the minimum value, and / or the absolute value of the difference between the median value and the pixel value is greater than or equal to a preset pixel threshold, then increasing the filter window to obtain an updated filter window;

[0035] If the updated filtering window is smaller than or equal to the preset window size threshold, returning to the step of obtaining the adjacent pixel area of ​​the pixel point based on the initial filtering window;

[0036] If the updated filtering window is larger than a preset window size threshold, the original pixel value is replaced with the median value.

[0037] In a feasible implementation, performing image enhancement processing on the denoised infrared image and the visible light image respectively to obtain enhanced infrared image and visible light image includes:

[0038] Using a preset Gaussian kernel function to perform convolution operations on the denoised infrared image and the visible light image, respectively, to obtain illumination components of the infrared image and the visible light image;

[0039] Determine the reflectivity components of the infrared image and the visible light image respectively according to the illumination component and a preset reflectivity component algorithm;

[0040] Normalizing the reflectivity component using a preset linear stretching algorithm to obtain a normalized reflectivity component;

[0041] An enhanced infrared image and a visible light image are obtained based on the normalized reflectivity component.

[0042] In a feasible implementation, performing image registration processing using the enhanced infrared image and visible light image to obtain a registered infrared image and visible light image includes:

[0043] Based on a preset SIFT algorithm, feature point extraction is performed on the enhanced infrared image and visible light image respectively to obtain key points and key point descriptors of the infrared image and the visible light image respectively;

[0044] Using the key point descriptor and a preset similarity algorithm to perform key point matching between the infrared image and the visible light image, to obtain a matching point set between the infrared image and the visible light image;

[0045] Calculate the transformation matrix between the infrared image and the visible light image according to the matching point set;

[0046] The infrared image or the visible light image is transformed using the transformation matrix to obtain a registered infrared image and a registered visible light image.

[0047] To achieve the above object, the second aspect of the present invention provides an image fusion device, the device comprising:

[0048] Image acquisition module: used to acquire the image to be fused, the image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection;

[0049] A coefficient determination module is used to obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, a first low-frequency sub-image and a second low-frequency sub-image, wherein the fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy and the regional gradient of the low-frequency sub-image and the fusion coefficient;

[0050] A weight determination module: used to obtain a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image by using a preset fusion weight algorithm, a first high-frequency sub-image and a second high-frequency sub-image, wherein the fusion weight algorithm is used to reflect the mapping relationship between the firing times of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image in the iteration process and the fusion weight;

[0051] Image fusion module: used to fuse the infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fused image.

[0052] To achieve the above-mentioned purpose, the third aspect of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps shown in the first aspect and any feasible implementation method.

[0053] To achieve the above-mentioned purpose, the fourth aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps shown in the first aspect and any feasible implementation method.

[0054] The embodiments of the present invention have the following beneficial effects:

[0055] The present invention provides an image fusion method, which comprises: obtaining an image to be fused, wherein the image to be fused comprises a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, wherein the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection; using a preset fusion coefficient algorithm, the first low-frequency sub-image and the second low-frequency sub-image, obtaining a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image, wherein the fusion coefficient algorithm is used to reflect the The mapping relationship between regional energy, regional gradient and fusion coefficient; using the preset fusion weight algorithm, the first high-frequency sub-image and the second high-frequency sub-image, the first fusion weight of the first high-frequency sub-image and the second fusion weight of the second high-frequency sub-image are obtained. The fusion weight algorithm is used to reflect the mapping relationship between the number of firings of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image and the fusion weight in the iteration process; infrared and visible light are fused according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fusion image. Through the above method, the infrared image and the visible light image can be fused into a target fusion image, so that the target fusion image has both high-resolution visibility and temperature information, and the advantages of the two images are combined to improve the accuracy and efficiency of substation inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] in:

[0058] Figure 1 is a flow chart of an image fusion method in an embodiment of the present invention;

[0059] Figure 2is another flow chart of an image fusion method in an embodiment of the present invention;

[0060] Figure 3 is a structural block diagram of an image fusion device in an embodiment of the present invention;

[0061] Figure 4 4 is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] See also Figure 1 , Figure 1 1 is a flow chart of an image fusion method in an embodiment of the present invention. The method can be applied to both a terminal and a server. The terminal and the server are connected via a network. The terminal can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. This embodiment is described by applying to a terminal as an example. Figure 1 The method shown includes the following steps:

[0064] 101. Obtain an image to be fused;

[0065] The image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, wherein the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection;

[0066] It should be noted that the fusion of infrared images and visible light images can combine the advantages of both and provide more comprehensive and accurate information for status monitoring and fault diagnosis of substation equipment. In order to improve the efficiency of substation inspection, this application fuses the infrared and visible light images collected during the substation inspection process so that the target fused image has both clear and visible details and temperature information.

[0067] Among them, the infrared and visible light images to be fused are obtained by simultaneously shooting the same object with infrared and visible light. In the present application, the images to be fused are obtained by multi-scale decomposition of visible light images collected during substation inspection and multi-scale decomposition of infrared images collected during substation inspection, and the multi-scale decomposition includes but is not limited to non-subsampled shearlet transform and the like. After non-subsampled shearlet transform, multiple sub-images at multiple decomposition scales can be obtained for each image, specifically including a first low-frequency sub-image, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, wherein the number of the first low-frequency sub-image, the first high-frequency sub-image, the second low-frequency sub-image and the second high-frequency sub-image is related to the number of decomposition scales, the first low-frequency sub-image and the second high-frequency sub-image are the results of multi-scale decomposition of the infrared image, and the second low-frequency sub-image and the second high-frequency sub-image are the results of multi-scale decomposition of the visible light image.

[0068] 102. Obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, the first low-frequency sub-image and the second low-frequency sub-image;

[0069] Furthermore, after obtaining the above-mentioned image to be fused, the fusion coefficient of the low-frequency sub-image can be determined by using a preset fusion coefficient algorithm, wherein the fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy, regional gradient and fusion coefficient of the low-frequency sub-image, that is, the fusion coefficient can be obtained from the regional energy and regional gradient of the low-frequency sub-image, and the fusion coefficient of the visible light low-frequency sub-image and the fusion coefficient of the infrared low-frequency sub-image are obtained respectively, and the fusion coefficient is used to reflect the importance of the two types of low-frequency sub-images when the low-frequency images are fused. For example, the fusion coefficient of the high importance is higher, and the fusion coefficient of the low importance is lower.

[0070] 103. Obtain a first fusion weight of the first high frequency sub-image and a second fusion weight of the second high frequency sub-image by using a preset fusion weight algorithm, the first high frequency sub-image and the second high frequency sub-image.

[0071] Furthermore, after obtaining the above-mentioned image to be fused, the fusion weight of the high-frequency sub-image can be determined by using a preset fusion weight algorithm, wherein the fusion weight algorithm is used to reflect the mapping relationship between the number of firings of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image and the fusion weight during the iteration process; that is, the fusion weight can be obtained from the number of firings, and the fusion coefficient of the high-frequency sub-image of visible light and the fusion coefficient of the high-frequency sub-image of infrared are obtained respectively, and the fusion weight is used to reflect the importance of the two types of high-frequency sub-images when the high-frequency images are fused. For example, the fusion coefficient of the high-importance sub-image is higher, and the fusion coefficient of the low-importance sub-image is lower.

[0072] 104. Perform fusion processing on the infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image, and the second high-frequency sub-image to obtain a target fused image.

[0073] After obtaining the fusion coefficient and the fusion weight, the image fusion processing combining the fusion coefficient and the fusion weight can be realized, and the fusion processing of the infrared and visible light images can be performed to obtain the target fusion image, so that the target fusion image has both details and temperature information.

[0074] The present invention provides an image fusion method, which comprises: obtaining an image to be fused, wherein the image to be fused comprises a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, wherein the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection; using a preset fusion coefficient algorithm, the first low-frequency sub-image and the second low-frequency sub-image, obtaining a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image, wherein the fusion coefficient algorithm is used to reflect the The mapping relationship between regional energy, regional gradient and fusion coefficient; using the preset fusion weight algorithm, the first high-frequency sub-image and the second high-frequency sub-image, the first fusion weight of the first high-frequency sub-image and the second fusion weight of the second high-frequency sub-image are obtained. The fusion weight algorithm is used to reflect the mapping relationship between the number of firings of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image and the fusion weight in the iteration process; infrared and visible light are fused according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fusion image. Through the above method, the infrared image and the visible light image can be fused into a target fusion image, so that the target fusion image has both high-resolution visibility and temperature information, and the advantages of the two images are combined to improve the accuracy and efficiency of substation inspection.

[0075] It should be noted that the collected infrared and visible light images can be preprocessed to improve the image rendering ability and reduce noise. The preprocessing includes but is not limited to denoising, enhancement and registration. The preprocessed images are used for multi-scale decomposition to obtain the image to be fused, which has higher accuracy. For details of the preprocessing process, please refer to Figure 2 .

[0076] See also Figure 2 , Figure 2 is another flow chart of an image fusion method in an embodiment of the present invention. Figure 2The methods shown include:

[0077] 201. Acquire the infrared image and the visible light image;

[0078] It is understandable that image acquisition is achieved using an image acquisition device, which acquires a set of infrared images and visible light images. The image acquisition device includes an infrared camera and a visible light camera, and the installation position should ensure that the equipment to be inspected can be fully covered, and the shooting angles of the two cameras should be kept consistent or close as much as possible to facilitate subsequent image registration work.

[0079] 202. Perform denoising processing on the infrared image and the visible light image respectively to obtain a denoised infrared image and a visible light image;

[0080] It can be understood that the preprocessing at least includes denoising, wherein the denoising includes but is not limited to adaptive median filtering. The denoising is performed by filtering using adaptive median filtering, and reference can be made to the following steps A01 to A07, i.e., step 202, including steps A01 to A06:

[0081] For each pixel in the infrared image and the visible light image:

[0082] A01. Obtaining a pixel region adjacent to the pixel based on an initial filtering window;

[0083] A02. Determine the median, maximum and minimum values ​​of pixels in the adjacent pixel area;

[0084] A03. If the original pixel value of the pixel point is between the maximum value and the minimum value, and the absolute value of the difference between the median value and the pixel value is less than a preset pixel threshold, retain the original pixel value of the pixel point;

[0085] A04. If the original pixel value of the pixel point is not between the maximum value and the minimum value, and / or the absolute value of the difference between the median value and the pixel value is greater than or equal to a preset pixel threshold, then increasing the filter window to obtain an updated filter window;

[0086] A05. If the updated filtering window is smaller than or equal to the preset window size threshold, return to the step of obtaining the adjacent pixel area of ​​the pixel point based on the initial filtering window;

[0087] A06. If the updated filtering window is larger than a preset window size threshold, the original pixel value is replaced with the median value.

[0088] It should be noted that the initial filter window size is preset, such as 3×3 pixels. For each pixel in the image, the median and maximum and minimum values ​​of the grayscale values ​​of the pixels in its neighborhood are calculated through steps A01 to A06. If the grayscale value of the pixel is between the maximum and minimum values, and the absolute value of the difference between the neighborhood median and the grayscale value of the pixel is less than the preset threshold value T1, the pixel is considered not to be noise, and the grayscale value of the pixel remains unchanged; otherwise, the filter window size is increased and the above steps are repeated.

[0089] If the filter window size is increased to a preset window size threshold (such as 7×7 pixels) and still does not meet the condition, the grayscale value of the pixel is replaced with the median value in the neighborhood. The above operation is performed on all pixels in the image in turn to complete the adaptive median filtering process.

[0090] 203. Perform image enhancement processing on the denoised infrared image and visible light image respectively to obtain enhanced infrared image and visible light image;

[0091] After obtaining the denoised image, image enhancement processing can be performed. Specifically, step 203 includes the following steps B01 to B04:

[0092] B01. Using a preset Gaussian kernel function, perform convolution operations on the denoised infrared image and the visible light image to obtain illumination components of the infrared image and the visible light image respectively;

[0093] B02. determining the reflectivity components of the infrared image and the visible light image respectively according to the illumination component and a preset reflectivity component algorithm;

[0094] B03, using a preset linear stretching algorithm to normalize the reflectivity component to obtain a normalized reflectivity component;

[0095] B04. Obtaining an enhanced infrared image and a visible light image based on the normalized reflectivity component.

[0096] Exemplarily, the illumination component is first estimated based on Gaussian filtering, specifically including: selecting a suitable Gaussian kernel function. The formula of the Gaussian kernel function is:

[0097]

[0098] Where σ is the standard deviation of the Gaussian kernel, which determines the scale of the filter (the degree of blur). For substation inspection images, σ can be selected between 3 and 10 according to the resolution and noise of the image.

[0099] The original image S(x,y) is convolved with the Gaussian kernel function to obtain the estimated value L(x,y) of the illumination component. The original image here is the denoised infrared and visible light image. The formula for the convolution operation is:

[0100]

[0101] Where n and m depend on the size of the Gaussian kernel. Generally, the Gaussian kernel size can be determined according to σ. For example, when σ=5, the Gaussian kernel size can be (6σ+1)×(6σ+1), that is, 31×31 pixels.

[0102] Further, the reflectivity component is calculated and the output range is adjusted, specifically including: according to the Retinex theory, through the formula Calculate the reflectance component. In actual calculations, since the estimated value of the illumination component L(x, y) may be 0 at some pixels, special processing is required. For example, a very small threshold can be set. When L(x, y) is less than the threshold, it is set as the threshold to avoid division by zero errors.

[0103] The value range of the reflectivity component R(x,y) obtained may not meet the display requirements and needs to be adjusted. It is usually normalized to the range of [0,255] for easy display and subsequent processing. The linear stretching method can be used, the formula is:

[0104]

[0105] Where R min and R max are the minimum and maximum values ​​in the reflectivity component R(x,y), respectively.

[0106] 204. Perform image registration processing using the enhanced infrared image and visible light image to obtain a registered infrared image and visible light image;

[0107] Furthermore, the enhanced image needs to be registered to obtain an aligned image. Specifically, step 204 includes the following steps C01 to C04:

[0108] C01. Performing feature point extraction processing on the enhanced infrared image and visible light image based on a preset SIFT algorithm to obtain key points and key point descriptors of the infrared image and the visible light image respectively;

[0109] C02, using the key point descriptor and a preset similarity algorithm to perform key point matching between the infrared image and the visible light image to obtain a matching point set between the infrared image and the visible light image;

[0110] C03, calculating the transformation matrix between the infrared image and the visible light image according to the matching point set;

[0111] C04. Using the transformation matrix to transform the infrared image or the visible light image, to obtain a registered infrared image and a registered visible light image.

[0112] It should be noted that the registration process is mainly divided into several steps: 1) feature point extraction based on SIFT algorithm 2) feature point matching 3) image transformation and alignment.

[0113] Among them, the SIFT algorithm (Scale-invariant feature transform) is a feature detection and description algorithm used in image processing, which has rotation, scale and illumination invariance. Feature point extraction based on the SIFT algorithm is mainly divided into several steps: 1) constructing a Gaussian pyramid 2) key point detection 3) determining the direction of the key point 4) generating a key point descriptor:

[0114] For example, constructing a Gaussian pyramid specifically includes: performing Gaussian blurring of different scales on the original image to obtain a series of images of different scales, and then subtracting images of adjacent scales to obtain a Gaussian difference pyramid (DoG), where the original image is the enhanced infrared and visible light image.

[0115] Key point detection: Search for local extreme points in the DoG pyramid, that is, compare each pixel with its 8 adjacent pixels and the 9×2 pixels corresponding to the upper and lower adjacent scales. If the pixel is a local extreme point, it is determined as a key point.

[0116] Determine the direction of the key point: With the key point as the center, calculate the gradient direction histogram of the image in its neighborhood, and take the peak direction of the histogram as the main direction of the key point. At the same time, some auxiliary directions can also be determined to enhance the description ability of the key point.

[0117] Generate key point descriptors: In the neighborhood of the key point, select an area of ​​a certain size and divide it into multiple sub-areas. Calculate the gradient amplitude and direction of the image in each sub-area, and then count this information as a feature vector as the descriptor of the key point.

[0118] The feature point matching specifically includes: for the feature points extracted from the infrared image and the visible light image, the Euclidean distance or other similarity measure between their descriptors is calculated. For each feature point, the feature point with the closest distance to its descriptor is found in the other image as the matching point.

[0119] A distance threshold is set. When the ratio of the closest distance to the second closest distance is less than the threshold, the match is considered reliable and the matching point is retained; otherwise, it is considered as a wrong matching point and is removed.

[0120] Image transformation and alignment specifically include: calculating the transformation matrix between the infrared image and the visible light image based on the correct matching point set obtained by feature point matching. Generally, the geometric deformation of the substation inspection image is small, and the affine transformation matrix is ​​used.

[0121] The affine transformation matrix contains transformation parameters such as rotation, translation, and scaling, which can be calculated using the following formula:

[0122] [x'] = [ab][x] + [tx];

[0123] [y']=[cd][y]+[ty];

[0124] Among them, (x, y) is the pixel coordinate in the original image, (x', y') is the pixel coordinate in the transformed image, [ab; cd] is the rotation and scaling matrix, and [tx ty] is the translation vector.

[0125] The infrared image or visible light image is transformed by the above formula to align the two images in space. The pixel values ​​of the transformed image can be filled by bilinear interpolation or nearest neighbor interpolation to obtain a smooth aligned image, that is, the registered infrared image and visible light image.

[0126] 205. Perform multi-scale decomposition processing on the registered infrared image and the visible light image respectively by using non-subsampled shearlet transform to obtain the image to be fused;

[0127] The image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, wherein the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection;

[0128] It should be noted that step 205 and Figure 1 Part of the contents of step 101 in the method shown is similar, and will not be described here to avoid repetition. For details, please refer to Figure 1 The content of step 101 in the method shown.

[0129] For example, the preprocessed image enters the multi-scale decomposition module, and the infrared image and the visible light image are decomposed into sub-images of multiple scales and directions using non-subsampled shearlet transform, including:

[0130] 1) Laplacian pyramid decomposition (LP), including:

[0131] Step 1: Low-pass filtering and downsampling

[0132] First, the original image is low-pass filtered. Commonly used low-pass filters include Gaussian low-pass filters. The original image here is the infrared and visible light images after registration processing. The formula of Gaussian low-pass filter is Where (u,v) is the frequency domain coordinate, It represents the distance to the origin of the frequency domain, and D0 is the cutoff frequency, which controls the smoothness of the filter.

[0133] The filtered image is downsampled, that is, a pixel value is taken every other row and column to obtain a low-resolution image of the next layer. Downsampling can be achieved by a simple index selection operation. For example, in a two-dimensional matrix, for an M×N image, after downsampling, of the image.

[0134] Step 2: Difference calculation and previous layer construction

[0135] Calculate the original image S and the image S1 obtained by low-pass filtering and upsampling after downsampling (upsampling is the inverse operation of downsampling, which enlarges the image size by inserting zero values ​​between pixel values) up The difference between the two images is obtained by taking the difference image D1. The difference image D1 reflects the high-frequency information of the image at the current scale.

[0136] Step 3: Repeat the operation to build a multi-layer pyramid

[0137] The low-resolution image S1 obtained in the previous layer is used as input, and steps 1 and 2 are repeated to perform multiple rounds of low-pass filtering, downsampling, and difference calculation to construct a multi-layer Laplacian pyramid. For example, a 3-5-layer pyramid can be constructed, and the specific number of layers is determined according to the size and complexity of the image and the requirements of subsequent processing. Generally speaking, for substation inspection images with higher resolution, 4-5 layers can be selected to better capture information at different scales.

[0138] 2) Directional localization-shearlet filter bank application, including:

[0139] Step 1: Calculate shear wave direction parameters

[0140] According to the required number of directional decompositions n (e.g. n = 8 means that the image is decomposed into 8 directions at each scale), the directional parameters of the shear wave are calculated. For a two-dimensional image, the directional parameters of the shear wave are related to the angle in polar coordinates. This can be obtained by the formula Calculate the angle values ​​in n directions.

[0141] Step 2: Construct a shearlet filter bank

[0142] Based on the calculated directional parameters, the corresponding shearlet filter bank is constructed. Shearlet filters can be obtained by rotating and translating the basic functions. Common construction methods include using the theory of Fourier transform or wavelet transform to design the frequency response characteristics of the filter.

[0143] Each filter in the filter bank has a specific directional selectivity and can perform filtering operations on the image in different directions to extract the feature information of the image in that direction.

[0144] Step 3: Directional filtering operation

[0145] For each layer of low-resolution image obtained by Laplacian pyramid decomposition, a shearlet filter bank is applied for directional filtering. The image is convolved with the filter in each direction to obtain sub-band images in different directions.

[0146] Through the above detailed multi-scale decomposition steps, the infrared and visible light images of substation inspection can be effectively decomposed into sub-images with rich feature information, which provides a good foundation for subsequent high-quality image fusion and helps to improve the accuracy and reliability of substation equipment monitoring and fault diagnosis.

[0147] 206. Obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, the first low-frequency sub-image and the second low-frequency sub-image;

[0148] The fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy, regional gradient and fusion coefficient of the low-frequency sub-image;

[0149] It should be noted that step 206 and Figure 1 The contents of step 102 in the method shown are similar, and are not described here to avoid repetition. For details, please refer to Figure 1 The content of step 102 in the method shown.

[0150] In a feasible implementation, for the low-frequency sub-image, its regional energy and gradient are calculated, and the fusion coefficient is determined according to the set threshold and weight calculation formula. Step 206 includes the following steps D01-D04:

[0151] D01. Divide the first low-frequency sub-image and the second low-frequency sub-image into regions according to a preset region division method to obtain a first image block of the first low-frequency sub-image and a second image block of the second low-frequency sub-image in each region;

[0152] D02, using the pixel values ​​of each pixel point in the first image block to perform a square sum calculation of the pixel values, to obtain a first region energy of the first image block in each region; using the pixel values ​​of each pixel point in the second image block to perform a square sum calculation of the pixel values, to obtain a second region energy of the human image block in each region;

[0153] D03, using the pixel values ​​of each pixel point in the first image block to calculate the average gradient value, to obtain the first regional gradient of the first image block in each region; using the pixel values ​​of each pixel point in the second image block to calculate the average gradient value, to obtain the second regional gradient of the second image block in each region;

[0154] D04. Based on the first region energy, the second region energy, the first region gradient, the second region gradient and the fusion coefficient algorithm of each region, obtain a first fusion coefficient of each region of the first low-frequency sub-image and a second fusion coefficient of each region of the second low-frequency sub-image.

[0155] Specifically, determining the region division method includes: dividing the infrared and visible light low-frequency sub-images obtained after preprocessing and multi-scale decomposition into non-overlapping regions of the same size, and the selection of the region size needs to comprehensively consider factors such as image resolution, detail richness, and computational efficiency. The preset region division method can be the above-mentioned division into non-overlapping regions of the same size, and the image blocks of each region are obtained through step D01.

[0156] Furthermore, the regional energy and gradient are calculated, including:

[0157] 1) Regional energy calculation: For each region, calculate its energy value. The energy value can be obtained by calculating the square sum of the grayscale values ​​of the pixels in the region. Suppose the grayscale value of the pixels in a region is I(x1, y1), where (x1, y1) represents the coordinates of the pixel in the region, then the energy E of the region is calculated as:

[0158]

[0159] Calculate the energy E of the corresponding areas of the infrared low-frequency sub-image and the visible light low-frequency sub-image respectively IR and E VIS .

[0160] 2) Regional gradient calculation, specifically including: using a gradient operator (such as the Sobel operator) to calculate the gradient value of the region. The Sobel operator includes horizontal and vertical operators, which are used to calculate the grayscale change rate of pixels in the horizontal and vertical directions respectively:

[0161] Horizontal Sobel operator

[0162] Vertical Sobel operator

[0163] For each pixel J(x1,y1) in the region, its gradient value is calculated by convolution with the Sobel operator. The gradient magnitude of the pixel (x1,y1) is Where * indicates the convolution operation.

[0164] The average gradient value of the calculated region is the average value of the gradient amplitude of all pixels in the region. The average gradient G of the corresponding regions of the infrared low-frequency sub-image and the visible light low-frequency sub-image are obtained respectively. IR and G VIS .

[0165] Finally, the fusion coefficient is determined, including:

[0166] Set the energy and gradient weight factors: According to the actual application requirements and experimental results, determine the energy weight factor α and the gradient weight factor β, and α+β=1. If you want to focus more on the energy information of the image, you can

[0167] To appropriately increase the value of α; if you pay more attention to gradient information (i.e., the edges and details of the image), you can increase the value of β.

[0168] Calculate the fusion coefficient: For each region, calculate the fusion coefficient F according to the following formula i (i represents the area number):

[0169]

[0170] Among them, E IR,i 、E VIS，i are the energy values ​​of the ith region of the infrared and visible light low-frequency sub-images, respectively, G IR,i , G VIS,i are the average gradient values ​​of the i-th region of the infrared and visible light low-frequency sub-images, respectively.

[0171] 207. Obtain a first fusion weight of the first high frequency sub-image and a second fusion weight of the second high frequency sub-image by using a preset fusion weight algorithm, the first high frequency sub-image and the second high frequency sub-image;

[0172] The fusion weight algorithm is used to reflect the mapping relationship between the firing times of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image and the fusion weight during the iteration process;

[0173] It should be noted that step 207 and Figure 1 The contents of step 103 in the method shown are similar, and are not described here to avoid repetition. For details, please refer to Figure 1 The content of step 103 in the method shown.

[0174] In a feasible implementation, for the high-frequency sub-image, based on the PCNN algorithm, the fusion weight is calculated according to the number of firing times of neurons, and the neurons correspond one-to-one to each pixel position of the high-frequency sub-image. Step 207 includes steps E01-E02:

[0175] E01, determining the firing times of each neuron in the iterative process of the pulse coupled neural network;

[0176] E02. Obtaining a first fusion weight for each pixel position of the first high-frequency sub-image and a second fusion weight for each pixel position of the second high-frequency sub-image according to each of the ignition times and a preset fusion weight algorithm.

[0177] Specifically, the preparation of fusion rules based on the improved pulse coupled neural network (PCNN) includes two aspects: one is the initialization of PCNN model parameters, and the other is the calculation of PCNN iteration and ignition times:

[0178] Among them, the PCNN model parameter initialization specifically includes:

[0179] Determine the connection strength W of neurons: The connection strength matrix W reflects the degree of mutual influence between neurons. It can be initialized according to the characteristics of the image and experience, and is generally set to a two-dimensional matrix of the same size as the image, with element values ​​in the range of [0,1]. For example, for an M×N image, W can be initialized to a random matrix with a mean of 0.5 and a standard deviation of 0.1.

[0180] Set the threshold θ: The threshold θ determines the firing condition of the neuron. The initial threshold can be set to a larger value and then gradually adjusted during the iteration process. A common method is to set the initial threshold to half of the maximum grayscale value of the image, that is, Where I is the input high-frequency sub-image (infrared or visible light).

[0181] Determine the time constant τ: The time constant τ controls the dynamic behavior of the neuron. It affects the pulse emission frequency and decay rate of the neuron. Generally, it can be set to a smaller value based on experience, such as τ = 10 (the specific value can be adjusted according to the actual effect).

[0182] Image input corresponds to neurons: The pixel values ​​of the high-frequency sub-image are used as input signals and assigned to the corresponding PCNN neurons. Assume that the high-frequency sub-image is I H , the PCNN neuron array has a one-to-one correspondence with the image pixels, that is, the pixel value I at position (x, y) in the image H (x,y) serves as the input signal to the neuron at position (x,y).

[0183] The iteration process specifically includes:

[0184] Neuron internal activity item calculation: For each neuron, calculate its internal activity item U i,j The internal activity term is determined by the input signal I i,j and connect input L i,j Jointly determined, the calculation formula is U i,j [n]=I i,j +L i,j [n-1], where (i, j) represents the coordinate position of the neuron, n represents the number of iterations, and L i,j The initial value of is 0.

[0185] Connection input calculation: Connection input L i,j is the weighted sum of the outputs of surrounding neurons, calculated as Where N(i,j) represents the neighborhood range of neuron (i,j), W i,j,k,l is the connection weight between neuron (i, j) and its neighboring neurons (k, l), Y k,l [n-1] is the output of the neighborhood neuron (k, l) at the n-1th iteration (the value is 0 or 1, indicating whether it is fired).

[0186] Threshold adjustment: According to the time constant τ and the threshold θ of the previous iteration i,j [n-1], update threshold θ i,j [n], the calculation formula is Wherein Vθ is a constant, which indicates the increase of the threshold value, and generally takes a small value, such as Vθ=0.1.

[0187] Neuron firing judgment: Compare the internal activity items U of neurons i,j [N] and threshold θ i,j [n], if U i,j [n]>θ i,j [n], then the neuron fires and outputs Y i,j [n]=1; otherwise, Y i,j [n]=0.

[0188] Firing times statistics: After a certain number of iterations (e.g. T iterations, T can be determined based on the experimental results, generally between 10-20 times), count the firing times C of each neuron i,j The number of firings reflects the activity level of the image feature at the location of the neuron.

[0189] Furthermore, performing high-frequency sub-image fusion through steps E01-E02 includes:

[0190] Determine the fusion weight: Calculate the fusion weight of the corresponding position pixel of the infrared and visible high-frequency sub-images according to the number of ignitions. Suppose the number of ignitions of the pixel (x, y) in the infrared high-frequency sub-image is C IR (x, y), the number of ignitions of pixel (x, y) in the visible light high-frequency sub-image is C VIS (x,y), then the fusion weight W IR (x,y) and W VIS (x,y) are:

[0191]

[0192]

[0193] 208. Perform fusion processing on the infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fused image.

[0194] It should be noted that step 208 and Figure 1 The contents of step 104 in the method shown are similar, and are not described here to avoid repetition. Figure 1 The content of step 104 in the method shown.

[0195] In a feasible implementation, step 208 includes steps F01-F04:

[0196] F01. Based on the first fusion coefficient and the second fusion coefficient, the first low-frequency sub-image and the second low-frequency sub-image at each decomposition scale are fused to obtain a low-frequency fused image of infrared and visible light at each decomposition scale;

[0197] Exemplarily, the low-frequency sub-image fusion includes:

[0198] Element-level fusion: According to the calculated fusion coefficient, the infrared and visible low-frequency sub-images are fused at the element level. Assume that the infrared low-frequency sub-image is The visible light low-frequency sub-image is The fused low-frequency sub-image is The fusion formula is:

[0199]

[0200] Where (x2, y2) represents the coordinates of the pixel in the low-frequency sub-image, and F(x2, y2) is the fusion coefficient of the corresponding area.

[0201] F02. Based on the first fusion weight and the second fusion weight, the first high-frequency sub-image and the second high-frequency sub-image at each decomposition scale are fused to obtain a high-frequency fused image of infrared and visible light at each decomposition scale;

[0202] Exemplary high-frequency sub-image fusion operation: using the calculated fusion weight, the infrared and visible high-frequency sub-images are fused. Suppose the infrared high-frequency sub-image is The visible light high frequency sub-image is The fused high-frequency sub-image is The fusion formula is:

[0203]

[0204] F03, using the low-frequency fusion image and the high-frequency fusion image at each decomposition scale to perform image reconstruction processing by inverse non-subsampled shearlet transform to obtain a low-frequency reconstructed image and a high-frequency reconstructed image;

[0205] The sub-image reconstruction includes: reconstructing the fused low-frequency sub-image and high-frequency sub-image to obtain a preliminary fused image. The reconstruction process can be performed according to the inverse process of multi-scale decomposition. For example, in the case of multi-scale decomposition using non-subsampled shearlet transform (NSST), the fused sub-image is restored to the size and spatial domain of the original image through an inverse NSST operation.

[0206] For example, low frequency sub-image reconstruction may include:

[0207] Inverse pyramid reconstruction: Starting from the lowest resolution fused low-frequency sub-image, it is gradually upsampled and fused with the residual information of the next higher low-frequency sub-image to restore the overall contour and low-frequency information of the image.

[0208] Upsampling operation: For each layer of low-frequency sub-images, use a suitable upsampling method (such as bilinear interpolation or nearest neighbor interpolation) to increase its resolution to the size of the previous layer.

[0209] Residual fusion: Add the residual information of the upsampled image to the low-frequency sub-image of the previous layer. In the multi-scale decomposition process, the residual information is obtained by calculating the difference between the low-resolution images of adjacent layers. During reconstruction, this residual information is gradually added back to the upsampled image to restore the low-frequency details and overall trend of the image. Repeat this process until the low-frequency reconstructed image of the highest resolution layer is obtained.

[0210] High frequency sub-image reconstruction may include:

[0211] Inverse filtering: For the fused high-frequency sub-images at each scale, inverse filtering is performed in the opposite order and manner of the directional filtering in the multi-scale decomposition process to restore the high-frequency details and edge information of the image.

[0212] Determine the inverse filter: Based on the shearlet filter bank used in NSST, construct the corresponding inverse filter. The inverse filter should be designed to restore the high-frequency information of the original image in that direction as much as possible.

[0213] Inverse filtering operation: For each high-frequency sub-image in each direction, perform a convolution operation with the corresponding inverse filter. The convolution process is similar to forward filtering, but the direction and order of the filter must be taken into account during calculation. By performing inverse filtering on the high-frequency sub-images in all directions and adding the results, a high-frequency reconstructed image at that scale is obtained.

[0214] Inter-scale fusion: The high-frequency reconstructed images at different scales are fused in descending order. In the fusion process, a simple weighted average or other fusion strategies designed according to image characteristics and application requirements can be used.

[0215] F04. Add the low-frequency reconstructed image and the high-frequency reconstructed image to obtain the target fused image.

[0216] It can be understood that the reconstructed low-frequency image and the reconstructed high-frequency image are added to obtain a preliminary reconstructed fusion image. This step combines the overall contour information of the low frequency and the detailed edge information of the high frequency to form an image with complete frequency characteristics.

[0217] The present invention provides an image fusion method, which has the following advantages: 1) Improve inspection accuracy: The fused image contains both the temperature information of the infrared image and the detailed features of the visible light image, which can more accurately detect the operating status of the equipment and timely discover potential fault hazards, such as the specific location and shape of overheated equipment, thereby improving the accuracy of inspection. 2) Enhance fault diagnosis capability: The rich image information provides a more comprehensive basis for fault diagnosis. By analyzing the fused image, the type and degree of equipment failure can be more accurately judged, which is helpful to formulate more effective maintenance strategies, reduce equipment downtime, and improve the operational reliability of the substation. 3) Improve image quality: The multi-scale decomposition and fusion rules are adopted to effectively retain the details and edge information of the image, so that the fused image has higher clarity and contrast, which is more convenient for staff to observe and analyze. 4) Strong adaptability: It is suitable for infrared and visible light image acquisition equipment of different types and brands, as well as various complex substation environments, and has strong versatility and adaptability.

[0218] See also Figure 3 , Figure 3is a structural block diagram of an image fusion device in an embodiment of the present invention, such as Figure 3 The device shown comprises:

[0219] Image acquisition module 301: used to acquire the image to be fused, the image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection;

[0220] The coefficient determination module 302 is used to obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, a first low-frequency sub-image and a second low-frequency sub-image, wherein the fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy and the regional gradient of the low-frequency sub-image and the fusion coefficient;

[0221] The weight determination module 303 is used to obtain a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image by using a preset fusion weight algorithm, a first high-frequency sub-image and a second high-frequency sub-image, wherein the fusion weight algorithm is used to reflect the mapping relationship between the firing times of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image in the iteration process and the fusion weight;

[0222] Image fusion module 304: used to fuse the infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fused image.

[0223] It should be noted that Figure 3 The functions of each module in the device shown are Figure 1 The contents of each step in the method shown are similar. To avoid repetition, they are not described here. For details, please refer to Figure 1 The content of each step in the method shown.

[0224] The present invention provides an image fusion device, which includes: an image acquisition module: used to acquire images to be fused, the images to be fused include a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection; a coefficient determination module: used to obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, the first low-frequency sub-image and the second low-frequency sub-image, the fusion coefficient algorithm is used to reflect the region of the low-frequency sub-image The mapping relationship between domain energy, regional gradient and fusion coefficient; weight determination module: used to obtain the first fusion weight of the first high-frequency sub-image and the second fusion weight of the second high-frequency sub-image by using the preset fusion weight algorithm, the first high-frequency sub-image and the second high-frequency sub-image, and the fusion weight algorithm is used to reflect the mapping relationship between the number of firings of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image in the iteration process and the fusion weight; image fusion module: used to perform fusion processing of infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fusion image. . Through the above device, the infrared image and the visible light image can be fused into a target fusion image, so that the target fusion image has both high-resolution visibility and temperature information, and the advantages of the two images are combined to improve the accuracy and efficiency of substation inspection.

[0225] Figure 4 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0226] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: Figure 1 or Figure 2 The steps of the method are shown.

[0227] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor performs the following steps: Figure 1 or Figure 2 The steps of the method are shown.

[0228] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0229] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image fusion method, characterized in that: The method comprises: Acquire an image to be fused, wherein the image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image, and a second high-frequency sub-image, wherein the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; and the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection; Using a preset fusion coefficient algorithm, a first low-frequency sub-image, and a second low-frequency sub-image, a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image are obtained, wherein the fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy and the regional gradient of the low-frequency sub-image and the fusion coefficient; Using a preset fusion weight algorithm, a first high-frequency sub-image, and a second high-frequency sub-image, a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image are obtained, wherein the fusion weight algorithm is used to reflect the mapping relationship between the firing times of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image in the iteration process and the fusion weight; The infrared and visible light images are fused according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fused image.

2. The method according to claim 1, characterized in that: The step of obtaining the image to be fused includes: Acquire the infrared image and the visible light image; De-noising the infrared image and the visible light image to obtain a de-noised infrared image and a visible light image; Performing image enhancement processing on the denoised infrared image and visible light image respectively to obtain enhanced infrared image and visible light image; Performing image registration processing using the enhanced infrared image and visible light image to obtain a registered infrared image and visible light image; The non-subsampled shearlet transform is used to perform multi-scale decomposition processing on the registered infrared image and the visible light image respectively to obtain the image to be fused.

3. The method according to claim 2, characterized in that: The step of fusing the infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image, and the second high-frequency sub-image to obtain a target fused image includes: Based on the first fusion coefficient and the second fusion coefficient, the first low-frequency sub-image and the second low-frequency sub-image at each decomposition scale are fused to obtain a low-frequency fused image of infrared and visible light at each decomposition scale; Based on the first fusion weight and the second fusion weight, the first high-frequency sub-image and the second high-frequency sub-image at each decomposition scale are fused to obtain a high-frequency fused image of infrared and visible light at each decomposition scale; The low-frequency fusion image and the high-frequency fusion image at each decomposition scale are used to perform image reconstruction processing by inverse non-subsampled shearlet transform to obtain a low-frequency reconstructed image and a high-frequency reconstructed image; The low-frequency reconstructed image and the high-frequency reconstructed image are added together to obtain the target fused image.

4. The method according to claim 1, characterized in that: The method of obtaining a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, the first low-frequency sub-image and the second low-frequency sub-image includes: According to a preset region division method, the first low-frequency sub-image and the second low-frequency sub-image are respectively divided into regions to obtain a first image block of the first low-frequency sub-image and a second image block of the second low-frequency sub-image in each region; Calculating the square sum of pixel values ​​using the pixel values ​​of each pixel point in the first image block to obtain the first region energy of the first image block in each region; calculating the square sum of pixel values ​​using the pixel values ​​of each pixel point in the second image block to obtain the second region energy of the human image block in each region; Calculating an average gradient value using the pixel values ​​of each pixel point in the first image block to obtain a first regional gradient of the first image block in each region; calculating an average gradient value using the pixel values ​​of each pixel point in the second image block to obtain a second regional gradient of the second image block in each region; Based on the first region energy, the second region energy, the first region gradient, the second region gradient and the fusion coefficient algorithm of each region, a first fusion coefficient of each region of the first low-frequency sub-image and a second fusion coefficient of each region of the second low-frequency sub-image are obtained.

5. The method according to claim 1, characterized in that: The neurons correspond one-to-one to the pixel positions of the high-frequency sub-images, and the first fusion weight of the first high-frequency sub-image and the second high-frequency sub-image are obtained by using a preset fusion weight algorithm, the first high-frequency sub-image and the second high-frequency sub-image, including: Determining the firing times of each neuron during the iteration of the pulse coupled neural network; According to each of the ignition times and a preset fusion weight algorithm, a first fusion weight of each pixel position of the first high-frequency sub-image and a second fusion weight of each pixel position of the second high-frequency sub-image are obtained.

6. The method according to claim 2, characterized in that: The denoising process is performed on the infrared image and the visible light image respectively to obtain the denoised infrared image and the visible light image, including: For each pixel in the infrared image and the visible light image: Obtaining a pixel region adjacent to the pixel based on an initial filtering window; Determine the median, maximum and minimum values ​​of pixels in the adjacent pixel area; If the original pixel value of the pixel point is between the maximum value and the minimum value, and the absolute value of the difference between the median value and the pixel value is less than a preset pixel threshold, the original pixel value of the pixel point is retained; If the original pixel value of the pixel point is not between the maximum value and the minimum value, and / or the absolute value of the difference between the median value and the pixel value is greater than or equal to a preset pixel threshold, then increasing the filter window to obtain an updated filter window; If the updated filtering window is smaller than or equal to the preset window size threshold, returning to the step of obtaining the adjacent pixel area of ​​the pixel point based on the initial filtering window; If the updated filtering window is larger than a preset window size threshold, the original pixel value is replaced with the median value.

7. The method according to claim 2, characterized in that: The performing image enhancement processing on the denoised infrared image and the visible light image respectively to obtain enhanced infrared image and visible light image comprises: Using a preset Gaussian kernel function to perform convolution operations on the denoised infrared image and the visible light image, respectively, to obtain illumination components of the infrared image and the visible light image; Determine the reflectivity components of the infrared image and the visible light image respectively according to the illumination component and a preset reflectivity component algorithm; Normalizing the reflectivity component using a preset linear stretching algorithm to obtain a normalized reflectivity component; An enhanced infrared image and a visible light image are obtained based on the normalized reflectivity component.

8. The method according to claim 2, characterized in that: The method of performing image registration processing using the enhanced infrared image and visible light image to obtain the registered infrared image and visible light image includes: Based on a preset SIFT algorithm, feature point extraction is performed on the enhanced infrared image and visible light image respectively to obtain key points and key point descriptors of the infrared image and the visible light image respectively; Using the key point descriptor and a preset similarity algorithm to perform key point matching between the infrared image and the visible light image, to obtain a matching point set between the infrared image and the visible light image; Calculate the transformation matrix between the infrared image and the visible light image according to the matching point set; The infrared image or the visible light image is transformed using the transformation matrix to obtain a registered infrared image and a registered visible light image.

9. An image fusion device, characterized in that: The device comprises: Image acquisition module: used to acquire the image to be fused, the image to be fused includes a plurality of first low-frequency sub-images, a first high-frequency sub-image, a second low-frequency sub-image and a second high-frequency sub-image, the first low-frequency sub-image and the first high-frequency sub-image are obtained by multi-scale decomposition of infrared images collected during substation inspection; the second low-frequency sub-image and the second high-frequency sub-image are obtained by multi-scale decomposition of visible light images collected during substation inspection; A coefficient determination module is used to obtain a first fusion coefficient of the first low-frequency sub-image and a second fusion coefficient of the second low-frequency sub-image by using a preset fusion coefficient algorithm, a first low-frequency sub-image and a second low-frequency sub-image, wherein the fusion coefficient algorithm is used to reflect the mapping relationship between the regional energy and the regional gradient of the low-frequency sub-image and the fusion coefficient; A weight determination module: used to obtain a first fusion weight of the first high-frequency sub-image and a second fusion weight of the second high-frequency sub-image by using a preset fusion weight algorithm, a first high-frequency sub-image and a second high-frequency sub-image, wherein the fusion weight algorithm is used to reflect the mapping relationship between the firing times of each neuron of the pulse coupled neural network constructed based on the high-frequency sub-image in the iteration process and the fusion weight; Image fusion module: used to fuse the infrared and visible light images according to the first fusion coefficient, the second fusion coefficient, the first fusion weight, the second fusion weight, the first low-frequency sub-image, the second low-frequency sub-image, the first high-frequency sub-image and the second high-frequency sub-image to obtain a target fused image.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

11. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

Images