Infrared and visible light image fusion method and system based on feature block segmentation and separation

Abstract

This invention discloses a method and system for fusing infrared and visible light images based on feature block segmentation. The method includes: designing two image enhancement methods according to the characteristics of infrared and visible light imaging, and using the SIFT algorithm to quickly separate primary and secondary feature blocks; designing a fusion network that can adaptively adjust the network output based on the amount of feature information, and performing multi-task training on the fusion network for primary and secondary features, enabling the network to serve both tasks; and performing a stitching operation on the fused primary and secondary feature block image blocks, eliminating stitching seams through Boehringer Ink fusion to obtain a fused infrared and visible light image with prominent salient features. This invention exhibits a clear characteristic of significant feature regions in the fusion results of primary and secondary feature image blocks, improving the efficiency and accuracy of subsequent processing algorithms, and making the results more readable than state-of-the-art fusion algorithms.

Description

Technical Field

[0001] This invention relates to the fields of computer and software technology, and specifically to a method and system for infrared and visible light image fusion based on feature block segmentation and separation. Background Technology

[0002] Infrared images can overcome the effects of harsh working environments, capturing the thermal radiation characteristics of the target so that the imaging image does not need to consider the intensity of reflected light from the environment. However, infrared images can only describe the target outline, and the imaging results lack texture details. Therefore, visible light images, which contain rich details but are limited by ambient light, are an excellent complement to infrared images. The fused image overcomes the limitations of a single sensor and combines the advantages of both. In high-level vision processing tasks, such as segmentation, tracking, and detection, image fusion can improve the performance of subsequent algorithms.

[0003] To achieve higher-quality fusion results, researchers have combined methods from multiple domains to improve fusion quality, including traditional methods and deep learning-based approaches. In simple applications, traditional pixel-level image fusion algorithms, such as multi-scale transformation, sparse representation, saliency maps, and other hybrid methods, are often used. However, the limitation of traditional methods, which require manually specifying fusion rules, makes them unsuitable for handling the diverse fusion objects in high-level vision tasks. Therefore, researchers have addressed this issue by constructing end-to-end deep learning network models. While various methods have achieved excellent fusion results, existing fusion algorithms tend to prioritize overall visual quality and human readability. Image fusion, as a preprocessing module for high-level vision tasks, aims for high interpretability in subsequent processing algorithms. Fusion algorithms need to be designed with the needs of high-level vision tasks in mind. These tasks require the fusion network to highlight primary feature regions and weaken secondary background feature regions to identify more salient objects. This necessitates targeted fusion strategies for primary and secondary feature regions. However, existing fusion algorithms typically employ the same fusion strategy, leading to ambiguity in network convergence during training.

[0004] To address this problem, various fusion algorithms for segmenting salient regions have been proposed. Some researchers extract the main feature regions by cross-comparing all pixels of the original image. However, since the fusion network does not design separate main feature region extraction schemes for infrared and visible light images, its identical strategy affects the salient target extraction effect. Therefore, a fusion method for infrared and visible light images based on salient target extraction and low-light region enhancement has been proposed. Researchers designed separate extraction strategies for infrared and visible light images according to their different visual characteristics. The intensity saliency is defined by comparing the intensity of each pixel in the original image with the background to extract salient targets. However, due to the need for manual customization of the fusion rules and the significant interference of noisy original images with the salient target extraction effect, it is not suitable for high-level vision tasks. To solve this problem, a fusion task design method tailored to the characteristics of high-level vision tasks has been proposed. Researchers proposed a fusion network model, TarDAL, using object-aware dual adversarial learning, which distinguishes between thermal targets in infrared images and background texture details in visible light images using two object-aware discriminators. Its lightweight network characteristics allow TarDAL to be deployed well in high-level vision tasks. However, TarDAL relies on a single source for extracting salient regions, depending on the imaging quality of infrared images. This makes it difficult to accurately extract salient regions of targets in complex and varied scenes. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an infrared and visible light image fusion method and system based on feature block segmentation and separation. This method focuses the image fusion resources on the main features, designs the network based on the needs of high-level vision tasks, and retains the main features of the source image to the greatest extent possible, which promotes the effect of subsequent image processing algorithms.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] The infrared and visible light image fusion method based on feature block segmentation and separation proposed in this invention includes:

[0008] S1. Based on the characteristics of infrared and visible light imaging, two image enhancement methods are designed, and the SIFT algorithm is used to quickly separate the main feature blocks and secondary feature blocks.

[0009] S2. Spatial and texture detail information from the primary and secondary feature blocks are fused in parallel in the fusion network to obtain primary and secondary feature image fusion blocks, and the two types of image fusion blocks are trained in a multi-task manner.

[0010] Fusion networks can adaptively adjust their structure based on the potential contribution of feature images to better serve the fusion task of primary and secondary feature blocks.

[0011] S3. Using Boson fusion, the main feature image fusion block and the secondary feature image fusion block after training are stitched together and the stitching seams are eliminated to obtain an infrared and visible light fusion image with prominent features.

[0012] Furthermore, in step S1, the specific steps for obtaining the primary and secondary feature blocks are as follows:

[0013] S101. The specific content of image enhancement for infrared images is as follows:

[0014] Infrared image imaging features are determined by the intensity of the target's thermal radiation, resulting in a clearly defined target outline in the image. To extract more targets, edge detection algorithms are used for image enhancement of the infrared image; a Gaussian filter is used to filter potential noise in the infrared image. The specific steps are as follows:

[0015]

[0016] Where G(x,y) represents the Gaussian function, and σ represents the standard deviation of the Gaussian filter.

[0017] The Sobel operator is used to calculate the gradient magnitude and direction of each pixel in the infrared image in order to find possible edge locations. The specific formula is as follows:

[0018]

[0019] Among them, G x and G y These represent the gradient values ​​in the horizontal and vertical directions, respectively, and I represents the input image.

[0020] By suppressing non-maximum values, the location of local gradient maxima in infrared images is found and considered as potential edges. The specific suppression formula is as follows:

[0021]

[0022] Here, Δx and Δy represent the gradient directions, with values ​​of 0 and ±1, respectively.

[0023] Since potential edges may contain noise, the gradient magnitude is divided into two thresholds to determine the desired edges, as shown in the following formula:

[0024] If G(xy)≥Th the point is a strong edge point

[0025] If Tl≤G(xy)<Th the point is a weak edge point

[0026] If G(xy)<Tl the point is a nonedge point

[0027] Among them, T h =0.2G max T l =0.1G max , G max This represents the maximum value of the gradient magnitude.

[0028] The specific content of image enhancement for visible light images is as follows:

[0029] Visible light image imaging depends on the quality of ambient light. Low light and overexposure environments significantly affect the quality of details. Therefore, image enhancement is achieved by converting the image to the frequency domain and using a bandpass filter to filter low and high frequencies. A two-dimensional Fourier transform is used to convert the visible light image to the frequency domain; the zero-frequency component in the frequency domain is shifted to the center of the spectrum, and the amplitude spectrum is calculated. The specific formula is as follows:

[0030] f mp =20·log 10 (|f|)

[0031] Where f represents the image transformed to the frequency domain, f mp This represents the amplitude spectrum.

[0032] High-pass filtering is applied to the amplitude spectrum to preserve high-frequency information. The specific formula is as follows:

[0033] f hp =f mp ·H(u,v)

[0034] Where H(u,v) represents the transfer function of the high-pass filter, f hp This represents an image that retains high-frequency information.

[0035] The filtered image is subjected to an inverse Fourier transform, followed by thresholding to obtain the final visible light image preprocessing. The specific formula for thresholding is as follows:

[0036]

[0037] Where FFT() represents the Fast Fourier Transform function, I vis This indicates that a visible light image will be output.

[0038] S102. Use the SIFT algorithm to annotate feature points in the preprocessed infrared and visible light images.

[0039] S103. Based on the distribution of feature points on the labeled image, the image is segmented into primary feature blocks and secondary feature blocks.

[0040] Furthermore, in step S103, the image segmentation process based on the distribution of feature points is as follows:

[0041] Regions with densely distributed feature points are considered as primary feature regions, and clustering is performed on these feature points. An adjustable moving window is used to traverse the image; when the number of feature points within the window is greater than or equal to a threshold, that feature point is segmented as the primary feature block Q. i The formula for calculating the threshold is:

[0042]

[0043] Where η is the threshold, and h and w are the height and width of the image.

[0044] Repeat the process multiple times until the window can no longer contain any more feature points; at this point, the remaining area of ​​the segmented original image is the secondary feature block.

[0045] Furthermore, in step S2, the specific steps for training the primary feature image fusion block and the secondary feature image fusion block are as follows:

[0046] S201. Multi-layer convolutional layers are used to extract features from both primary and secondary feature blocks, forcing the fused image to contain richer texture details to enhance the feature extraction capabilities of subsequent processing tasks. Multiple gradient operator modules are connected in parallel within the multi-layer convolutions to enhance edge and texture details in the image, and 1×1 regularized convolutional layers are used to eliminate channel dimension differences. Considering that the feature image after gradient calculation may lose some information during propagation, gradient calculation module output and convolutional layer output are added at the end of the convolutional layers to integrate depth and detail features. The output of each convolutional layer is represented as follows:

[0047]

[0048] Where RELU is the activation function, BN is the batch normalization function, Conv is the convolution function, Sobel is the operator, P is the output of the convolutional layer, and i is the output of the convolutional layer.

[0049] S202. After feature extraction, the infrared and visible light images each yield five layers of output feature maps. These feature maps are then concatenated along the channel dimension, increasing the dimensionality from 1D to 2D. The concatenated result is fed into the five-layer input of the feature fusion layer, and the output of the feature fusion layer is as follows:

[0050] S i =concat(P i_ir ,P i_vis dim=1)

[0051]

[0052] Among them, S i P represents a combined channel for infrared and visible light images. i_ir P represents the feature map of an infrared image. i_vis L represents the feature table of a visible light image. i 0 This represents the output of the i-th feature fusion layer.

[0053] S203. Feature maps from different scales exhibit significant differences in feature representation. Therefore, the outputs of adjacent feature fusion layers are cross-connected. The connection result is then passed through convolution, batch normalization, and activation layers before being fed into a multi-scale cross-fusion layer. At this point, the output consists of a primary feature image fusion block and a secondary feature image fusion block, represented as follows:

[0054]

[0055] Among them, L i 1 This represents the multi-scale cross-fusion output of the i-th layer.

[0056] S204. Calculate the similarity function between the feature maps of the infrared and visible light images and the infrared and visible light images to obtain the contribution of the feature maps to the future fusion result. The specific calculation formula is as follows:

[0057] pSSIM(P,I ir ,I vis =SSIM(P,I) ir )+SSIM(P,I vis )

[0058]

[0059]

[0060] Where P is the output of the feature extraction layer, and I ir For infrared images, I vis For visible light images, SSIM is the similarity function, and C... i CN represents the size of each layer's output channel, and CN represents the total number of channels.

[0061] A higher contribution indicates richer feature information contained in that layer. At the end of the fusion network, we allocate an appropriate number of channels based on the contribution of different layers.

[0062] S205. To evaluate the importance of network parameters and maintain training performance across different tasks, the Fisher information matrix is ​​used as a parameter importance evaluation term. The average value of the curvature (i.e., the second derivative) of the log-likelihood function with respect to the parameter is calculated. The smaller this value, the smaller the curvature, and the smaller the impact of changes in the model's parameters on changes in the probability density function, indicating that the parameter is of low importance. The specific formula is as follows:

[0063]

[0064]

[0065] in, Let θ be the multi-task balancing function, D be the training task, and θ be the network parameters for the current task. * For the network parameters of the previous task; μ i This is a parameter importance evaluation item.

[0066] S206. Design a loss function sensitive to image gradient transformation based on the characteristics of the main feature image fusion blocks. The main feature image fusion blocks are trained; a loss function using mean squared error is designed based on the characteristics of the secondary feature image fusion blocks. The secondary feature image fusion block is then trained using the following formula:

[0067]

[0068]

[0069]

[0070]

[0071] in, Let h be the gradient transformation metric function, w be the image height, and I be the image width. f To merge images.

[0072] Furthermore, in step S3, the specific details of obtaining a fused infrared and visible light image with prominent features are as follows:

[0073] Using the visible light image as the output image, all feature image fusion blocks are seamlessly stitched together sequentially. Given the output image and the feature image fusion blocks, and assuming a specified region is fused to position P in the output image, while ensuring that the pixel values ​​of the feature block image match the pixel gradients of the output image, a set of pixel gradients of the output image is traversed to minimize the pixel gradient difference between the two images near the stitching position, ultimately obtaining the infrared and visible light fused image. The specific formula is as follows:

[0074]

[0075]

[0076] Where x and y are the pixel positions of the image, I tgt It is the output image, I src It is a feature image fusion block, R src It refers to the designated area.

[0077] Furthermore, this invention also proposes an infrared and visible light image fusion system based on feature block segmentation and separation, including...

[0078] The source image segmentation module is used to design two image enhancement methods based on the characteristics of infrared and visible light imaging, and uses the SIFT algorithm to quickly separate them to obtain the main feature blocks and secondary feature blocks.

[0079] The image fusion module is used to fuse spatial information and texture detail information in the main feature block and the secondary feature block in parallel in the fusion network to obtain the main feature image fusion block and the secondary feature image fusion block, and to train the two types of image fusion blocks in a multi-task manner.

[0080] The image stitching module is used to stitch together the trained main feature image fusion blocks and secondary feature image fusion blocks using Boson fusion and eliminate stitching seams to obtain infrared and visible light fused images with prominent features.

[0081] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the infrared and visible light image fusion method based on feature block segmentation and separation described above.

[0082] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the infrared and visible light image fusion method based on feature block segmentation and separation.

[0083] The present invention adopts the above technical solution, and its significant technical effects compared with the prior art are as follows:

[0084] (1) High processing efficiency: The core of this invention lies in the separation of primary and secondary features and the prediction participation module, which makes the fusion result show obvious features and can improve the efficiency and accuracy of subsequent processing algorithms.

[0085] (2) Lightweight Network: This invention adjusts the output channel size by predicting the degree of information retention of the input feature image in the fusion result. The fusion network maintains a lightweight architecture and adopts a multi-task training method, training the fusion network in batches to reduce the network parameters by half.

[0086] (3) Excellent fusion results: Qualitative experiments and object detection task tests on various datasets demonstrate the excellent performance of this invention. The feature block separation fusion feature makes the results of this invention more readable than state-of-the-art fusion algorithms. Attached Figure Description

[0087] Figure 1 This is a flowchart illustrating the overall steps of the present invention.

[0088] Figure 2 This is a schematic diagram of the feature block segmentation of an image according to the present invention.

[0089] Figure 3 This is a schematic diagram of the fusion result of the present invention. Detailed Implementation

[0090] The technical solution of the present invention will now be clearly and completely described in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0091] This invention proposes a method for fusing infrared and visible light images based on feature block segmentation and separation, such as... Figure 1 As shown, it includes the following steps:

[0092] S1. Based on the imaging characteristics of infrared and visible light, two image enhancement methods are designed, and the SIFT algorithm is used to quickly separate the main feature blocks and secondary feature blocks, such as... Figure 2 As shown, the specific steps are as follows:

[0093] S101. The specific content of image enhancement for infrared images is as follows:

[0094] Edge detection algorithms are used to enhance infrared images; Gaussian filters are used to filter potential noise in infrared images. Figure 2 The second column of the second row shows the enhancement effect of the infrared image. The specific steps are as follows:

[0095]

[0096] Where G(x,y) represents the Gaussian function, and σ represents the standard deviation of the Gaussian filter.

[0097] The Sobel operator is used to calculate the gradient magnitude and direction of each pixel in the infrared image in order to find possible edge locations. The specific formula is as follows:

[0098]

[0099] Among them, G x and G y These represent the gradient values ​​in the horizontal and vertical directions, respectively, and I represents the input image.

[0100] By suppressing non-maximum values, the location of local gradient maxima in infrared images is found and considered as potential edges. The specific suppression formula is as follows:

[0101]

[0102] Here, Δx and Δy represent the gradient directions, with values ​​of 0 and ±1, respectively.

[0103] Since potential edges may contain noise, the gradient magnitude is divided into two thresholds to determine the desired edges, as shown in the following formula:

[0104] If G(xy)≥Th the point is a strong edge point

[0105] If Tl≤G(xy)<Th the point is a weak edge point

[0106] If G(xy)<Tl the point is a nonedge point

[0107] Among them, T h =0.2G max , T l =0.1G max G max This represents the maximum value of the gradient magnitude.

[0108] The specific content of image enhancement for visible light images is as follows:

[0109] Visible light image imaging depends on the quality of ambient light. Low light and overexposure environments significantly affect the quality of details. Therefore, image enhancement is achieved by converting the image to the frequency domain and using a bandpass filter to filter low and high frequency components. A two-dimensional Fourier transform is used to convert the visible light image to the frequency domain; the zero-frequency component in the frequency domain is shifted to the center of the spectrum, and the amplitude spectrum is calculated. Figure 2 The image in the second column of the first row shows the enhancement effect of the visible light image. The specific formula for the enhancement process is as follows:

[0110] f mp=20·log 10 (|f|)

[0111] Where f represents the image transformed to the frequency domain, f mp This represents the amplitude spectrum.

[0112] High-pass filtering is applied to the amplitude spectrum to preserve high-frequency information. The specific formula is as follows:

[0113] f hp =f mp ·H(u,v)

[0114] Where H(u,v) represents the transfer function of the high-pass filter, f hp This represents an image that retains high-frequency information.

[0115] The filtered image is subjected to an inverse Fourier transform, followed by thresholding to obtain the final visible light image preprocessing. The specific formula for thresholding is as follows:

[0116]

[0117] Where FFT() represents the Fast Fourier Transform function, I vis This indicates that a visible light image will be output.

[0118] S102. Use the SIFT algorithm to annotate feature points in the preprocessed infrared and visible light images. Figure 2 The third column of images shows the feature point annotation results for infrared and visible light images.

[0119] S103. Based on the distribution of feature points on the labeled image, segment the image into primary feature blocks and secondary feature blocks. Figure 2 The fourth column of the image shows the segmentation results of the infrared and visible light images. The rectangular regions enclosed within the image represent the primary feature blocks, while the remaining regions represent secondary feature blocks. The specific segmentation process is as follows:

[0120] Regions with densely distributed feature points are considered as primary feature regions, and clustering is performed on these feature points. An adjustable moving window is used to traverse the image; when the number of feature points within the window is greater than or equal to a threshold, that feature point is segmented as the primary feature block Q. i The formula for calculating the threshold is:

[0121]

[0122] Where η is the threshold, and h and w are the height and width of the image.

[0123] Repeat the process multiple times until the window can no longer contain any more feature points; at this point, the remaining area of ​​the segmented original image is the secondary feature block.

[0124] S2. Spatial and texture detail information from the primary and secondary feature blocks are fused in parallel within the fusion network to obtain primary and secondary feature image fusion blocks. These two types of image fusion blocks are then trained using a multi-task approach. The specific steps are as follows:

[0125] S201. Multi-layer convolutional layers are used to extract features from both primary and secondary feature blocks, forcing the fused image to contain richer texture details to enhance the feature extraction capabilities of subsequent processing tasks. Multiple gradient operator modules are connected in parallel within the multi-layer convolutions to enhance edge and texture details in the image, and 1×1 regularized convolutional layers are used to eliminate channel dimension differences. Considering that the feature image after gradient calculation may lose some information during propagation, gradient calculation module output and convolutional layer output are added at the end of the convolutional layers to integrate depth and detail features. The output of each convolutional layer is represented as follows:

[0126]

[0127] Where RELU is the activation function, BN is the batch normalization function, Conv is the convolution function, Sobel is the operator, P is the output of the convolutional layer, and i is the output of the convolutional layer.

[0128] S202. After feature extraction, the infrared and visible light images each yield five layers of output feature maps. These feature maps are then concatenated along the channel dimension, increasing the dimensionality from 1D to 2D. The concatenated result is fed into the five-layer input of the feature fusion layer, and the output of the feature fusion layer is as follows:

[0129] S i =concat(P i_ir ,P i_vis dim=1)

[0130]

[0131] Among them, S i P represents a combined channel for infrared and visible light images. i_ir P represents the feature map of an infrared image. i_vis L represents the feature table of a visible light image. i 0 This represents the output of the i-th feature fusion layer.

[0132] S203. Feature maps from different scales exhibit significant differences in feature representation. Therefore, the outputs of adjacent feature fusion layers are cross-connected. The connection result is then passed through convolution, batch normalization, and activation layers before being fed into a multi-scale cross-fusion layer to reduce dimensionality differences. The output of the multi-scale cross-fusion layer is represented as follows:

[0133]

[0134] Among them, L i 1 This represents the multi-scale cross-fusion output of the i-th layer.

[0135] S204. Calculate the similarity function between the feature maps of the infrared and visible light images and the infrared and visible light images to obtain the contribution of the feature maps to the future fusion result. The specific calculation formula is as follows:

[0136] pSSIM(P,I ir ,I vis =SSIM(P,I) ir )+SSIM(P,I vis )

[0137]

[0138]

[0139] Where P is the output of the feature extraction layer, and I ir For infrared images, I vis For visible light images, SSIM is the similarity function, and C... i CN represents the size of each layer's output channel, and CN represents the total number of channels.

[0140] S205. To evaluate the importance of network parameters and maintain training performance across different tasks, the Fisher information matrix is ​​used as a parameter importance evaluation term. This is achieved by calculating the average curvature (i.e., the second derivative) of the log-likelihood function with respect to the parameter. The smaller this value, the smaller the curvature, and the less impact changes in the model's parameters have on the probability density function, indicating low parameter importance. The specific formula is as follows:

[0141]

[0142]

[0143] in, Let θ be the multi-task balancing function, D be the training task, and θ be the network parameters for the current task. * For the network parameters of the previous task; μ i This is a parameter importance evaluation item.

[0144] S206. Design a loss function sensitive to image gradient transformation based on the characteristics of the main feature image fusion blocks. The main feature image fusion blocks are trained; a loss function using mean squared error is designed based on the characteristics of the secondary feature image fusion blocks. The secondary feature image fusion block is then trained using the following formula:

[0145]

[0146]

[0147]

[0148]

[0149] in, Let h be the gradient transformation metric function, w be the image height, and I be the image width. f To merge images.

[0150] S3. Using Boson fusion, the main feature image fusion block and the secondary feature image fusion block after training are stitched together and the stitching seams are eliminated to obtain an infrared and visible light fusion image with prominent features. Figure 3 It is the result of fusing two sets of infrared and visible light images, specifically:

[0151] Using the visible light image as the output image, all feature image fusion blocks are seamlessly stitched together sequentially. Given the output image and the feature image fusion blocks, and assuming a specified region is fused to position P in the output image, while ensuring that the pixel values ​​of the feature block image match the pixel gradients of the output image, a set of pixel gradients of the output image is traversed to minimize the pixel gradient difference between the two images near the stitching position, ultimately obtaining the infrared and visible light fused image. The specific formula is as follows:

[0152]

[0153]

[0154] Where x and y are the pixel positions of the image, I tgt It is the output image, I src It is a feature image fusion block, R src It refers to the designated area.

[0155] from Figure 3 It can be observed that after segmentation, block fusion, and stitching, the fusion result exhibits excellent performance in both regional detail and overall fusion quality. Furthermore, as a preprocessing module for high-level vision tasks, we use object detection algorithms to detect the saliency of the main targets. Figure 3 The last column shows the test results, which demonstrate that this invention can effectively help target detection algorithms detect targets such as cars and pedestrians.

[0156] This invention also proposes an infrared and visible light image fusion system based on feature block segmentation and separation, including a source image segmentation module, an image fusion module, an image stitching module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.

[0157] This invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0158] This invention also proposes a computer-readable storage medium storing a computer program. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0159] The above description is merely an exemplary embodiment of the present invention and does not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An infrared and visible light image fusion method based on feature block segmentation and separation, including: S1. Based on the characteristics of infrared and visible light imaging, two image enhancement methods are designed, and the SIFT algorithm is used to quickly separate the main feature blocks and secondary feature blocks; specifically: S101. The specific content of image enhancement for infrared images is as follows: Image enhancement of infrared images is performed using an edge detection algorithm; potential noise in infrared images is filtered out using a Gaussian filter. The specific steps are as follows: Where G(x,y) represents the Gaussian function, and σ represents the standard deviation of the Gaussian filter; The Sobel operator is used to calculate the gradient magnitude and direction of each pixel in an infrared image. The specific formula is as follows: Among them, G x and G y These represent the gradient values ​​in the horizontal and vertical directions, respectively, and I represents the input image; By suppressing non-maximum values, the location of local gradient maxima in infrared images is found and considered as potential edges. The specific suppression formula is as follows: Where Δx and Δy represent the gradient directions; The gradient magnitude is divided into two thresholds to determine the required edges. The specific formula is as follows: If G(xy)≥Th the point is a strong edge point If Tl≤G(xy) <Th the point is a weak edge point If G(xy) <Tl the point is a nonedge point Among them, T h =0.2G max T l =0.1G max G max This represents the maximum value of the gradient magnitude. The specific content of image enhancement for visible light images is as follows: Using a two-dimensional Fourier transform, the visible light image is converted to the frequency domain; the zero-frequency component in the frequency domain is shifted to the center of the spectrum, and the amplitude spectrum is calculated. The specific formula is as follows: f mp =20·log 10 (|f|) Where f represents the image transformed to the frequency domain, f mp Represents the amplitude spectrum; High-pass filtering is applied to the amplitude spectrum to preserve high-frequency information. The specific formula is as follows: f hp =f mp ·H(u,v) Where H(u,v) represents the transfer function of the high-pass filter, f hp Represents an image that retains high-frequency information; The filtered image is subjected to an inverse Fourier transform, followed by thresholding to obtain the final visible light image preprocessing. The specific formula for thresholding is as follows: Where FFT() represents the Fast Fourier Transform function, I vis This indicates that the output is a visible light image; S102. Use the SIFT algorithm to annotate feature points in the preprocessed infrared and visible light images respectively. S103. Based on the distribution of feature points on the annotated image, segment the image into primary feature blocks and secondary feature blocks; specifically: Regions with densely distributed feature points are considered as primary feature regions, and feature points within these regions are clustered. An adjustable moving window is used to traverse the image; when the number of feature points within the window is greater than or equal to a threshold, that feature point is segmented as the primary feature block Q. i The formula for calculating the threshold is: Where η is the threshold, and h and w are the height and width of the image; Repeat the process multiple times until the window can no longer contain any more feature points; at this point, the remaining area of ​​the segmented original image is the secondary feature block. S2. Spatial and texture detail information in the main and secondary feature blocks are fused in parallel in the fusion network to obtain the main feature image fusion block and the secondary feature image fusion block, and the two image fusion blocks are trained in a multi-task manner. S3. Using Boson fusion, the main feature image fusion block and the secondary feature image fusion block after training are stitched together and the stitching seams are eliminated to obtain an infrared and visible light fusion image with prominent features.

2. The infrared and visible light image fusion method based on feature block segmentation and separation according to claim 1, characterized in that, In step S2, the specific steps for training the primary feature image fusion block and the secondary feature image fusion block are as follows: S201. Use multiple convolutional layers to extract features from the main and secondary feature blocks. Multiple gradient operator modules are connected in parallel within the convolutional layers, and a 1×1 regularized convolutional layer is used to eliminate channel dimension differences. Gradient calculation module output and convolutional layer output are added at the end of the convolutional layers. The output of each convolutional layer is represented as follows: Where RELU is the activation function, BN is the batch normalization function, Conv is the convolution function, Sobel is the operator, P is the output of the convolutional layer, and i is the output of the convolutional layer. S202. After feature extraction, the infrared and visible light images each yield five layers of output feature maps. These feature maps are then concatenated along the channel dimension, increasing the dimensionality from 1D to 2D. The concatenated result is fed into the five-layer input of the feature fusion layer, and the output of the feature fusion layer is as follows: S i =concat(P i_ir ,P i_vis ,dim=1) Among them, S i P represents a combined channel for infrared and visible light images. i_ir P represents the feature map of an infrared image. i_vis L represents the feature map of a visible light image. i 0 This represents the output of the i-th feature fusion layer; S203. The output of the cross-connected adjacent feature fusion layer is passed through convolution, batch normalization, and activation layers to the multi-scale cross-fusion layer. At this point, the main feature image fusion block and the secondary feature image fusion block are output, and the output is represented as follows: Among them, L i 1 This represents the multi-scale cross-fusion output of the i-th layer; S204. Calculate the similarity function between the feature maps of the infrared and visible light images and the infrared and visible light images to obtain the contribution of the feature maps to the future fusion result. The specific calculation formula is as follows: pSSIM(P,I ir I vis )=SSIM(P,I ir )+SSIM(P,I vis ) Where P is the output of the feature extraction layer, and I ir For infrared images, I vis For visible light images, SSIM is the similarity function, and C... i CN represents the total number of channels, which is the size of the output channels allocated for each layer. S205. Using the Fisher information matrix as a parameter importance evaluation term, calculate the average curvature of the log-likelihood function with respect to the parameter. The specific formula is as follows: in, Let θ be the multi-task balancing function, D be the training task, and θ be the network parameters for the current task. * For the network parameters of the previous task, μ i For parameter importance evaluation items; S206. Design a loss function sensitive to image gradient transformation based on the characteristics of the main feature image fusion blocks. The main feature image fusion blocks are trained; a loss function using mean squared error is designed based on the characteristics of the secondary feature image fusion blocks. The secondary feature image fusion block is then trained using the following formula: in, Let h be the gradient transformation metric function, w be the image height, and I be the image width. f To merge images.

3. The infrared and visible light image fusion method based on feature block segmentation and separation according to claim 1, characterized in that, In step S3, the specific content of obtaining a fused infrared and visible light image with prominent features is as follows: Using the visible light image as the output image, all feature image fusion blocks are seamlessly stitched together sequentially. Given the output image and the feature image fusion blocks, and assuming a specified region is fused to position P in the output image, while ensuring that the pixel values ​​of the feature block image match the pixel gradients of the output image, a set of pixel gradients of the output image is traversed to minimize the pixel gradient difference between the two images near the stitching position, ultimately obtaining the infrared and visible light fused image. The specific formula is as follows: Where x and y are the pixel positions of the image, I tgt It is the output image, I src It is a feature image fusion block, R src It refers to the designated area.

4. An infrared and visible light image fusion system based on feature block segmentation and separation, characterized in that, This includes a source image segmentation module, used to design two image enhancement methods based on the characteristics of infrared and visible light imaging, and to use the SIFT algorithm for rapid separation to obtain primary and secondary feature blocks; specifically: Step 1, Image Enhancement of Infrared Images: Image enhancement of infrared images is performed using an edge detection algorithm; potential noise in infrared images is filtered out using a Gaussian filter. The specific steps are as follows: Where G(x,y) represents the Gaussian function, and σ represents the standard deviation of the Gaussian filter; The Sobel operator is used to calculate the gradient magnitude and direction of each pixel in an infrared image. The specific formula is as follows: Among them, G x and G y These represent the gradient values ​​in the horizontal and vertical directions, respectively, and I represents the input image; By suppressing non-maximum values, the location of local gradient maxima in infrared images is found and considered as potential edges. The specific suppression formula is as follows: Where Δx and Δy represent the gradient directions; The gradient magnitude is divided into two thresholds to determine the required edges. The specific formula is as follows: If G(xy)≥Th the point is a strong edge point If Tl≤G(xy) <Th the point is a weak edge point If G(xy) <Tl the point is anonedge point Among them, T h =0.2G max T l =0.1G max G max This represents the maximum value of the gradient magnitude. The specific content of image enhancement for visible light images is as follows: Using a two-dimensional Fourier transform, the visible light image is converted to the frequency domain; the zero-frequency component in the frequency domain is shifted to the center of the spectrum, and the amplitude spectrum is calculated. The specific formula is as follows: f mp =20·log 10 (|f|) Where f represents the image transformed to the frequency domain, f mp Represents the amplitude spectrum; High-pass filtering is applied to the amplitude spectrum to preserve high-frequency information. The specific formula is as follows: f hp =f mp ·H(u,v) Where H(u,v) represents the transfer function of the high-pass filter, f hp Represents an image that retains high-frequency information; The filtered image is subjected to an inverse Fourier transform, followed by thresholding to obtain the final visible light image preprocessing. The specific formula for thresholding is as follows: Where FFT() represents the Fast Fourier Transform function, I vis This indicates that the output is a visible light image; Step 2: Use the SIFT algorithm to annotate feature points in the preprocessed infrared and visible light images respectively; Step 3: Based on the distribution of feature points on the labeled image, segment the image into primary feature blocks and secondary feature blocks; specifically: Regions with densely distributed feature points are considered as primary feature regions, and feature points within these regions are clustered. An adjustable moving window is used to traverse the image; when the number of feature points within the window is greater than or equal to a threshold, that feature point is segmented as the primary feature block Q. i The formula for calculating the threshold is: Where η is the threshold, and h and w are the height and width of the image; Repeat the process multiple times until the window can no longer contain any more feature points; at this point, the remaining area of ​​the segmented original image is the secondary feature block. The image fusion module is used to fuse spatial information and texture detail information in the main feature block and the secondary feature block in parallel in the fusion network to obtain the main feature image fusion block and the secondary feature image fusion block, and to train the two types of image fusion blocks in a multi-task manner. The image stitching module is used to stitch together the trained main feature image fusion blocks and secondary feature image fusion blocks using Boson fusion and eliminate stitching seams to obtain infrared and visible light fused images with prominent features.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method of any one of claims 1 to 3.

Images