Infrared visible light image fusion method, electronic equipment and computer medium
By introducing channel and spatial attention mechanisms in U-Net network, the problems of information loss and edge blur in infrared visible light image fusion are solved, and high-quality image fusion under different lighting conditions are achieved, which improves the stability of the fusion effect. and clarity.
Patent Information
- Application Number
- CN202411095814.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
The existing infrared visible light image fusion technology has problems such as image information loss, blurred edges and unstable fusion effect. Especially when there is insufficient light or bad weather, it is difficult to obtain clear and high-quality target information.
An infrared visible light image fusion method is adopted to obtain visible light and infrared light images for channel stitching, use U-Net network for noise addition and denoising operations, and set the channel attention mechanism and spatial attention mechanism in the jump connection, dynamic Adjust the image feature weight to generate a fusion weighted feature image, and finally obtain a high-fidelity target fusion image.
It effectively avoids image information loss and edge blur, improves the stability and clarity of the fusion effect, and ensures high-quality target information obtained under different lighting conditions.
Smart Images

Figure CN119068298A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image fusion, and in particular to an infrared and visible light image fusion method, electronic equipment and computer medium. Background Art
[0002] In the case of insufficient light, such as at night, infrared cameras are usually used to collect infrared images. Infrared images mainly reflect the information of thermal radiation in the collected scene. However, the resolution and contrast ratio of infrared images are relatively low, and they cannot fully reflect the texture details of the target. In the case of sufficient light, visible light images are usually collected. However, the clarity of visible light images is related to weather conditions. When the weather is bad or the lighting conditions are poor, the quality of visible light images is poor and important information is easily lost.
[0003] In existing technical fields, such as security monitoring, computer vision, military detection, remote sensing, etc., it is necessary to fuse infrared images and visible light images to obtain clear, high-quality fused images of target information. However, the current fusion methods based on transform domain and fusion methods based on decomposition and reconstruction have problems such as image information loss, edge blur, and unstable fusion effect. Summary of the invention
[0004] The present invention proposes an infrared-visible light image fusion method, an electronic device and a computer medium to solve the problems of image information loss, edge blur, unstable fusion effect, etc. in the current fusion methods based on transform domain and fusion methods based on decomposition and reconstruction.
[0005] The technical solution of the present invention is as follows: an infrared-visible light image fusion method, comprising obtaining a visible light image and an infrared light image of a position to be photographed; performing channel splicing on the visible light image and the infrared light image to obtain a multi-channel original image; performing denoising operation on the multi-channel original image and then performing denoising operation to obtain a denoising prediction network, wherein the denoising prediction network is a U-Net network, the denoising prediction network includes N stages, and the jump connection of each stage is provided with a channel attention mechanism and a spatial attention mechanism in parallel, and N is an integer greater than or equal to 1; the multi-channel original image is input into the denoising prediction network so that the denoising prediction network The N stages all output N multi-channel intermediate images based on the multi-channel original image, and after the multi-channel intermediate image is output at the i-th stage, the multi-channel intermediate image is converted into a channel feature image based on the channel attention mechanism corresponding to the i-th stage, wherein i is an integer from 1 to N; the multi-channel intermediate image is converted into a spatial feature image based on the spatial attention mechanism corresponding to the i-th stage; the channel feature image, the spatial feature image and the multi-channel intermediate image of each stage are respectively spliced to obtain N fused weighted feature images; and the N fused weighted feature images are fused to obtain a target fused image.
[0006] Compared with the related art, the embodiments of the present application have at least the following advantages:
[0007] This application uses denoising and denoising operations on multi-channel original images to obtain a U-Net network based on channel and spatial attention mechanisms. In the jump connection of the U-Net network, the channel attention mechanism is used to obtain the channel feature information in the multi-channel original image and the spatial attention mechanism is used to obtain the spatial feature information in the multi-channel original image. It avoids the problem that only the spatial attention mechanism or the channel attention mechanism is used to process the multi-channel original image, resulting in incomplete feature information of the acquired image, loss of image information, unclear features, etc. Afterwards, the self-attention mechanism is used to process multiple fused weighted feature images, and the feature areas in the multiple fused weighted feature images are extracted to obtain a target fused image that can reflect the target information.
[0008] In some embodiments, the fusing of N fused weighted feature images to obtain a target fused image includes: extracting an initial weighted feature image from each of the fused weighted feature images; performing dimensionality conversion on each of the initial weighted feature images, and fusing the converted multiple initial weighted feature images into a fused initial feature image, wherein the dimension of the fused initial feature image is the same as the dimension of the multi-channel original image; extracting a region of interest from the fused initial feature image, and obtaining the target fused image based on the region of interest, wherein the region of interest is the region where the photographed object is located in the visible light image.
[0009] In some embodiments, the extracting the region of interest in the fused initial feature image and obtaining the target fused image based on the region of interest includes: extracting the region of interest in the fused initial feature image and using the region of interest as the fused target feature image; performing dimension conversion on the fused target feature image to obtain the target fused image, wherein the dimension of the target fused image is the same as that of the visible light image.
[0010] In some embodiments, the channel attention mechanism corresponding to the i-th stage converts the multi-channel intermediate image into a channel feature image, including: calculating the average pooling value of the multi-channel intermediate image according to the spatial dimension; converting the multi-channel intermediate image into a first channel image based on the average pooling value; calculating the maximum pooling value of the multi-channel intermediate image according to the spatial dimension; converting the multi-channel intermediate image into a second channel image based on the maximum pooling value; extracting the first eigenvector in the first channel image and the second eigenvector in the second channel image; calculating the channel attention weights of the first eigenvector and the second eigenvector; and obtaining the channel feature image based on the channel attention weights and the multi-channel intermediate image.
[0011] In some embodiments, the spatial attention mechanism corresponding to the i-th stage converts the multi-channel intermediate image into a spatial feature image, including: calculating the average pooling value of the multi-channel intermediate image according to the channel dimension; converting the multi-channel intermediate image into a first spatial image based on the average pooling value; calculating the maximum pooling value of the multi-channel intermediate image according to the channel dimension; converting the multi-channel intermediate image into a second spatial image based on the maximum pooling value; splicing the first spatial image and the second spatial image to obtain a spliced spatial image; calculating the spatial attention weight of the spliced spatial image; and obtaining the spatial feature image based on the spatial attention weight and the multi-channel intermediate image.
[0012] In some embodiments, after fusing N fused weighted feature images to obtain a target fused image, the method further includes: calculating a first loss value and a second loss value of the target fused image; calculating a target loss value based on the first loss value and the second loss value; and using the target loss value to update the model parameters of the denoising prediction network until the denoising prediction network training is completed.
[0013] In some embodiments, the method of calculating the first loss value of the target fused image is:
[0014]
[0015] Among them, F i is the first loss value, H is the height value of the image, W is the width value of the image, and I f is the target fused image, I r is the infrared light image, I v is the visible light image, ||…|| F is the matrix Frobenius norm, α is the regularization parameter; the method for calculating the second loss value of the target fused image is:
[0016] F s =1-SSIM(I f ,I v )
[0017] Among them, F s is the second loss value, and SSIM(…) is the structural similarity measure.
[0018] An embodiment of the present application also provides an infrared-visible light image fusion method, comprising: obtaining a visible light image and an infrared light image of a position to be photographed; performing channel stitching on the visible light image and the infrared light image to obtain a multi-channel original image; performing denoising operation on the multi-channel original image and then performing denoising operation to obtain a denoising prediction network, wherein the denoising prediction network is a U-Net network, and the denoising prediction network includes N stages, and the jump connection of each stage is a channel attention mechanism and a spatial attention mechanism set in series, and N is an integer greater than or equal to 1; the multi-channel original image is input into the denoising prediction network so that the The N stages of the denoising prediction network output N multi-channel intermediate images based on the multi-channel original image, and after the multi-channel intermediate image is output at the i-th stage, the multi-channel intermediate image is converted into a channel feature image based on the channel attention mechanism corresponding to the i-th stage, where i is an integer from 1 to N; the channel feature image is processed based on the spatial attention mechanism corresponding to the i-th stage to obtain a spatial feature image; the spatial feature image and the multi-channel intermediate image of each stage are spliced to obtain N fused weighted feature images; and the N fused weighted feature images are fused to obtain a target fused image.
[0019] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the infrared-visible light image fusion method as described above.
[0020] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned infrared-visible light image fusion method.
[0021] Compared with the prior art, the above-mentioned infrared visible light image fusion method, electronic device and computer medium. When using the channel attention mechanism and spatial attention to process the multi-channel intermediate image, the average pooling value of the multi-channel intermediate image is calculated to retain more detail information in the image, and the maximum pooling value of the multi-channel intermediate image is calculated to retain part of the information with higher feature significance in the image. After that, the channel attention weight and the spatial attention weight are calculated. The channel attention weight and the spatial attention weight are used to obtain a channel feature image and a spatial feature image that pay more attention to the key area. Finally, the channel feature image and the spatial feature image are fused to obtain a fused weighted feature image to focus on the key area in the fused weighted feature image, and obtain a target fused image with good accuracy and high fidelity. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a step flow chart of an infrared-visible light image fusion method according to one embodiment of the present application.
[0023] Figure 2 for Figure 1 Schematic diagram of the image fusion process using the infrared and visible light image fusion method.
[0024] Figure 3 for Figure 1 Schematic diagram of the process of the cascaded channel spatial attention mechanism in the infrared-visible light image fusion method.
[0025] Figure 4 This is another step flow chart of the infrared-visible light image fusion method according to one embodiment of the present application.
[0026] Figure 5 for Figure 4 Schematic diagram of the image fusion process using the infrared and visible light image fusion method.
[0027] Figure 6 for Figure 4 Schematic diagram of the process of channel attention mechanism in the infrared and visible light image fusion method in .
[0028] Figure 7 for Figure 4 Schematic diagram of the process of spatial attention mechanism in the infrared and visible light image fusion method in .
[0029] Figure 8 This is a flowchart of the steps of training a denoising prediction network according to one embodiment of the present application.
[0030] Fig. 9 It is a structural schematic diagram of an electronic device according to an embodiment of the present application.
[0031] Main component symbols
[0032] Electronic equipment 1000
[0033] Processor 1001
[0034] Memory 1002
[0035] Computer Programs 1003
[0036] The following specific implementation methods will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0037] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.
[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.
[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0040] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0041] In this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.
[0042] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0043] Example
[0044] like Figure 1 The figure is a flowchart of the steps of an embodiment of the infrared visible light image fusion method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.
[0045] See also Figure 1 As shown, the infrared-visible light image fusion method may include the following steps.
[0046] Step S101: obtaining a visible light image and an infrared light image of a position to be photographed, and performing channel stitching on the visible light image and the infrared light image to obtain a multi-channel original image.
[0047] In this embodiment, a visible light infrared monitoring ball camera can be used to monitor the position to be photographed in the acquisition area in real time. The visible light infrared monitoring ball camera can monitor the temperature distribution in the area in real time and collect infrared light images. The visible light infrared monitoring ball camera can also collect visible light images in the area. In other embodiments, a first camera without an infrared collection function can be used to collect visible light images in the area, and a second camera with an infrared collection function can be used to collect infrared images in the area. This application does not limit the collection method of visible light images and infrared images.
[0048] Furthermore, since the visible light image includes three channels, R, G, and B, the infrared light image includes one channel. Figure 2 As shown, the visible light image and the infrared light image are channel fused to obtain a multi-channel original image with four channels. In this embodiment, the visible light image with three channels and the infrared light image with one channel are directly channel spliced, without the need to separate or convert the channels of the visible light image and the infrared light image. At the same time, directly splicing the visible light image and the infrared light image can better retain and integrate the information of the visible light image and the infrared light image, avoid information loss and redundant calculation, and improve the overall fusion effect.
[0049] Step S102: performing a denoising operation on the multi-channel original image after adding noise, and then performing a denoising operation to obtain a denoising prediction network.
[0050] In some embodiments, Figure 2 As shown in the figure, the denoising prediction network is a U-Net network. The denoising prediction network consists of N stages, and the jump connection of each stage is set up with a channel attention mechanism and a spatial attention mechanism in series, where N is an integer greater than or equal to 1. That is, in the jump connection between the downsampling and the corresponding upsampling in each stage of the U-Net network, a channel attention mechanism and a spatial attention mechanism (denoted as Att) are set up. Figure 3 , which is a serial connection method of the channel attention mechanism and the spatial attention mechanism in the embodiment of the present application.
[0051] For details, please refer to Figure 2In this embodiment, the downsampling process includes five stages, and the corresponding upsampling process also includes five stages. A channel attention mechanism and a spatial attention mechanism are set in series in the jump connection between the downsampling and upsampling in each stage. Wherein, W represents the width value of the image, H represents the height value of the image, and 64, 128, 256, 521 and 1024 all represent the dimensions of the image. ⊕ represents convolution and fusion processing. Conv represents convolution processing, BN represents normalization processing, and LReLu and Tanh are both activation functions. Addition represents addition processing. Wherein, LReLu activation function and Tanh activation function are both prior art, and this application will not repeat them.
[0052] In some embodiments, a multi-channel original image is subjected to denoising and denoising operations based on a diffusion model. Specifically, in the forward diffusion process of the diffusion model, noise is gradually added to the multi-channel original image until it approaches pure noise. Specifically, Gaussian noise is sequentially added to the multi-channel original image through t time steps. When the time step is t, the image after adding noise can be expressed as:
[0053]
[0054] Among them, Z is the standard normal distribution, I t and I t-1 They represent the 4-channel noise images generated by adding t and t-1 times Gaussian noise, α t is the variance of the Gaussian noise added in time step t, and N indicates that it obeys the Gaussian distribution.
[0055] In each forward diffusion step, the diffusion model generates the image information with noise added for the next time step based on the current image information and noise information.
[0056] In the reverse diffusion process of the diffusion model, P(I t-1 |I t ), a two-dimensional Gaussian noise is randomly sampled at time t, and then denoised step by step, and finally a generated image with the same distribution as the real image is obtained. Therefore, by processing the multi-channel original image in the forward diffusion process and the reverse diffusion process, the denoising prediction network can be trained. The training goal of the diffusion model is to model the features of the four-channel image through diffusion.
[0057] Step S103: Input the multi-channel original image into the denoising prediction network, so that the N stages of the denoising prediction network output N multi-channel intermediate images based on the multi-channel original image, and after the i-th stage outputs the multi-channel intermediate image, the multi-channel intermediate image is converted into a channel feature image based on the channel attention mechanism corresponding to the i-th stage.
[0058] In some embodiments, the channel attention mechanism emphasizes important features by calculating the correlation between channels. In this embodiment, the denoising prediction network has 5 stages. That is, both downsampling and upsampling have 5 stages. Each stage corresponds to a multi-channel intermediate image. For example, the top stage of downsampling is recorded as the first stage, and a multi-channel intermediate image is obtained after the downsampling process of the first stage. Then, based on the channel attention mechanism corresponding to the first stage, the multi-channel intermediate image is converted into a channel feature image.
[0059] For details, please combine Figure 3 , including the following:
[0060] (1): According to the spatial dimension, the average pooling value of the multi-channel intermediate image is calculated, and based on the average pooling value, the multi-channel intermediate image is converted into the first channel image.
[0061] In some embodiments, the multi-channel intermediate image is pooled. Pooling is to integrate feature points in a small neighborhood to obtain new features. Pooling includes maximum pooling and average pooling. The average pooling operation calculates the average value of each channel of the feature map to obtain a one-dimensional vector with the same number of channels. Each element of this vector represents the average activation value on the corresponding channel, which can be regarded as a global statistical feature of the channel. The channel identifier g generated by global average pooling is also a vector, and each element is the average value on the corresponding channel. The image processed by average pooling can achieve the purpose of reducing dimensionality.
[0062] The advantage of average pooling is that it can extract the overall information of the feature map and average the information within the channel, which helps to reduce the model's sensitivity to specific locations. In addition, the global average pooling operation can also reduce the number of model parameters and reduce model complexity, thereby improving the generalization ability of the model.
[0063] (2): According to the spatial dimension, the maximum pooling value of the multi-channel intermediate image is calculated, and based on the maximum pooling value, the multi-channel intermediate image is converted into a second channel image.
[0064] In some embodiments, the maximum pooling operation extracts the maximum value of each channel of the feature map, thereby obtaining a one-dimensional vector with the same number of channels. Each element of this vector represents the maximum activation value on the corresponding channel, so it can be regarded as a global feature representation of the channel. The channel identifier mc generated by global maximum pooling is a vector, each element of which is the maximum value on the corresponding channel.
[0065] The advantage of maximum pooling is that it can highlight the most significant information in the feature map, thereby alleviating the overfitting phenomenon to a certain extent. In addition, since the global maximum pooling operation is performed independently on each channel, it has a certain spatial invariance, that is, it has a certain robustness to the spatial transformation of the input.
[0066] (3): Extract the first eigenvector in the first channel image and the second eigenvector in the second channel image.
[0067] In this embodiment, the denoising prediction network also includes a fully connected layer. The first channel image and the second channel image are input into the fully connected layer to obtain a first eigenvector and a second eigenvector. Full connection (also called a fully connected layer) is a neural network layer commonly used in deep learning. The main function is to connect all neurons in the previous layer to all neurons in the current layer, and each connection has a weight. The output of the fully connected layer is obtained by weighted summation of the input and weight, and processed by an activation function.
[0068] (4): Calculate the channel attention weights of the first eigenvector and the second eigenvector.
[0069] In this embodiment, the first eigenvector and the second eigenvector are processed using the sigmoid activation function to obtain the channel attention weight. The sigmoid activation function and the processing of the fully connected layer are both prior arts, and are not described in detail in this application.
[0070] It should be noted that using the channel attention mechanism to process multi-channel intermediate images and obtain channel feature images can pay more attention to channel feature information.
[0071] (5): Based on the channel attention weights and multi-channel intermediate images, the channel feature image is obtained.
[0072] In this embodiment, the channel attention weights are multiplied by the multi-channel intermediate images to obtain the channel feature images.
[0073] Step S104: Convert the channel feature image into a spatial feature image based on the spatial attention mechanism corresponding to the i-th stage.
[0074] In some embodiments, please combine Figure 3 , the spatial attention mechanism is used to process the channel feature image output by the channel attention mechanism to obtain the spatial feature image. The spatial attention mechanism emphasizes important areas by calculating the correlation between pixels.
[0075] Specifically include the following:
[0076] (1): According to the channel dimension, the average pooling value of the channel feature image is calculated to obtain the first spatial image.
[0077] (2): According to the channel dimension, the maximum pooling value of the channel feature image is calculated to obtain the second spatial image.
[0078] (3): Stitch the first spatial image and the second spatial image to obtain a stitched spatial image.
[0079] (4): Calculate the spatial attention weights of the spliced spatial image.
[0080] In this embodiment, the spliced spatial image is first convolved to obtain a convolved spatial image, and then the convolved spatial image is processed using a sigmoid activation function to obtain a spatial attention weight.
[0081] (5): Based on the spatial attention weights and multi-channel intermediate images, a spatial feature image is obtained.
[0082] In this embodiment, the spatial attention weight is multiplied by the multi-channel intermediate image to obtain a spatial feature image.
[0083] It should be noted that using the spatial attention mechanism to process the channel feature image and obtain the spatial feature image can pay more attention to the spatial feature information.
[0084] Step S105: splicing the spatial feature images and multi-channel intermediate images of each stage to obtain N fused weighted feature images.
[0085] For example, in the upsampling process in the top stage, the 64-channel image is fused with the spatial feature image obtained by the channel attention mechanism and the spatial attention mechanism in the skip connection of this stage to obtain a fused weighted feature image. The number of fused weighted feature images is the same as the number of downsampling stages.
[0086] In this embodiment, since the downsampling process includes five stages, five fused weighted feature images are obtained.
[0087] It should be noted that, in other embodiments, the spatial attention mechanism may be used to process the multi-channel intermediate image to obtain a spatial feature image. The channel attention mechanism is then used to process the spatial feature image to obtain a channel feature image. Finally, in the upsampling process of the denoising prediction network, the channel feature image and the multi-channel intermediate image of each stage are spliced respectively to obtain N fused weighted feature images.
[0088] Step S106: Fusing N fused weighted feature images to obtain a target fused image.
[0089] In this embodiment, after convolution and fusion processing is performed on five fused weighted feature images, an initial fused feature image with 128 channels is obtained.
[0090] Furthermore, based on the initial fusion feature image, in order to obtain the target fusion image, the following processing needs to be performed. The specific steps include: performing multiple convolutions, batch normalization, activation functions and addition processing on the initial fusion feature fusion image to obtain an intermediate fusion feature image. Then, the intermediate fusion feature image is subjected to convolution processing and activation function processing to obtain the target fusion image (i.e. Figure 2 The activation function in this step is the Tanh function. The Tanh function is a prior art and will not be described in detail in this application.
[0091] In this embodiment, the initial fused feature image is first convolved, and then the image after convolution is batch normalized and activated, and finally, the image is added. This step is performed at least twice to improve the performance and generalization ability of the denoising prediction network, and can effectively remove the noise data in the image and obtain the feature information in the fused image.
[0092] Compared with the prior art, the embodiments of the present application have at least the following advantages: the present application is based on the U-Net network of the channel attention mechanism and the spatial attention mechanism, and uses the channel attention mechanism and the spatial attention mechanism to dynamically adjust the weights of different channels and positions in the image, so that the image processed by the channel attention mechanism and the spatial attention mechanism can pay more attention to key information, thereby extracting higher-level feature information to obtain a target fusion image with high clarity and high fidelity.
[0093] like Figure 4 As shown, it is a flowchart of another embodiment of the infrared visible light image fusion method of the present application. In the above embodiment, the image processed by the channel attention mechanism and the spatial attention mechanism in series may have channel information or spatial information lost, resulting in the loss of feature information in the target fused image. In order to solve this problem, the above embodiment is improved, and the specific steps are as follows:
[0094] Step S201: obtaining a visible light image and an infrared light image of a position to be photographed, and performing channel stitching on the visible light image and the infrared light image to obtain a multi-channel original image.
[0095] This step is the same as step 101 and will not be described in detail in this application.
[0096] Step S202: performing denoising and denoising operations on the multi-channel original image to obtain a denoising prediction network.
[0097] In some embodiments, the denoising prediction network is a U-Net network based on a channel attention mechanism and a spatial attention mechanism. The denoising prediction network includes N stages, and the jump connections of each stage are provided with a channel attention mechanism and a spatial attention mechanism in parallel, where N is an integer greater than or equal to 1.
[0098] For details, please refer to Figure 5 , the denoising prediction network is a U-Net network based on the channel attention mechanism and the spatial attention mechanism. In the jump connection between the downsampling and the corresponding upsampling in each stage of the U-Net network, the channel attention mechanism and the spatial attention mechanism are set. And the channel attention mechanism and the spatial attention mechanism are connected in parallel. Please combine Figure 6 and Figure 7 , respectively, are structural diagrams of the channel attention mechanism and the spatial attention mechanism in the embodiments of the present application.
[0099] Specifically, downsampling includes five stages, and the corresponding upsampling also includes five stages. In each stage of the jump connection between downsampling and upsampling, a parallel channel attention mechanism and a spatial attention mechanism are set. Among them, W represents the width value of the image, H represents the height value of the image, and 64, 128, 256, 521 and 1024 represent the dimensions of the image. CAM represents the channel attention mechanism, SAM represents the spatial attention mechanism, DSConv represents the separable deep convolutional layer, FC represents the fully connected layer, Reshape represents the reshaping layer, Conv represents the convolution processing, ⊙ represents the multiplication, and Softmax and Tanh are both activation functions.
[0100] Step S203: input the multi-channel original image into the denoising prediction network, so that the N stages of the denoising prediction network output N multi-channel intermediate images based on the multi-channel original image, and after the multi-channel intermediate image is output at the i-th stage, the multi-channel intermediate image is converted into a channel feature image based on the channel attention mechanism corresponding to the i-th stage, where i is an integer from 1 to N.
[0101] This step is the same as step 103 and will not be described in detail in this application.
[0102] Step S204: Convert the multi-channel intermediate image into a spatial feature image based on the spatial attention mechanism corresponding to the i-th stage.
[0103] In this embodiment, the denoising prediction network has 5 stages. That is, both downsampling and upsampling have 5 stages. Each stage corresponds to a multi-channel intermediate image. For example, the top stage of downsampling is recorded as the first stage, and a multi-channel intermediate image is obtained after the downsampling process of the first stage. Then, the multi-channel intermediate image is converted into a spatial feature image based on the spatial attention mechanism corresponding to the first stage.
[0104] Please combine Figure 7 , the specific steps are as follows:
[0105] (1): According to the channel dimension, the average pooling value of the multi-channel intermediate image is calculated to obtain the third space image.
[0106] (2): According to the channel dimension, the maximum pooling value of the multi-channel intermediate image is calculated to obtain the fourth spatial image.
[0107] (3): Stitching the third spatial image and the fourth spatial image to obtain a stitched spatial image.
[0108] (4): Calculate the spatial attention weights of the spliced spatial image.
[0109] In this embodiment, the spliced spatial image is first convolved to obtain a convolved spatial image, and then the convolved spatial image is processed using a sigmoid activation function to obtain a spatial attention weight.
[0110] (5): Based on the spatial attention weights and multi-channel intermediate images, a spatial feature image is obtained.
[0111] Step S205: splicing the channel feature images, spatial feature images and multi-channel intermediate images of each stage respectively to obtain N fused weighted feature images.
[0112] For example, in the upsampling process in the top stage, the 576-channel image is fused with the channel feature image obtained by the channel attention mechanism in the skip connection of this stage and the spatial feature image obtained by the spatial attention mechanism to obtain a fused weighted feature image. The number of fused weighted feature images is the same as the number of downsampling stages.
[0113] In this embodiment, since the downsampling process includes five stages, five fused weighted feature images are obtained.
[0114] Step S206: Fusing N fused weighted feature images to obtain a target fused image.
[0115] (1): Extract the initial weighted feature image from multiple fused weighted feature images.
[0116] In this embodiment, a depthwise separable convolution layer is used to extract the initial weighted feature image from multiple fused weighted feature images. Depthwise separable convolution (DSC) consists of depthwise convolution and pointwise convolution, where depthwise convolution is used to extract spatial features and pointwise convolution is used to extract channel features. Depthwise separable convolution groups convolutions in the feature dimension, performs independent depthwise convolution on each channel, and aggregates all channels using a 1x1 convolution (pointwise convolution) before output.
[0117] (2): Perform dimension transformation on each initial weighted feature image, and fuse the converted multiple initial weighted feature images into a fused initial feature image, where the dimension of the fused initial feature image is the same as the dimension of the multi-channel original image.
[0118] In this embodiment, the reshape dense layer and the Reshape layer in Keras are used to realize the dimensionality conversion of the initial weighted feature image and the fusion of multiple converted initial weighted feature images.
[0119] (3): Extract the region of interest in the fused initial feature image and use the region of interest as the fused target feature image.
[0120] In this embodiment, the fused initial feature image is convolved to obtain a fused target feature image, wherein the region of interest is the region where the photographed object is located in the visible light image.
[0121] (4): Perform dimension transformation on the fusion target feature image to obtain a target fusion image, and the dimension of the target fusion image is the same as that of the visible light image.
[0122] In this embodiment, dot product processing and normalization processing are performed on the fusion target feature image, and the fusion target feature image after the dot product and normalization processing is multiplied by a preset weight matrix to obtain a first feature image.
[0123] Further, the first feature image is convolved to obtain a second feature image, and the dimension of the second feature image is the same as the dimension of the visible light image. In this embodiment, the first feature image is mapped to a 3-channel output using a 3×3 convolution layer to obtain a 3-channel second feature image. The second feature image is then processed using an activation function to generate a target fused image. In this embodiment, the second feature image is processed using a Tanh activation function to obtain a target fused image.
[0124] Compared with the above-mentioned embodiments, this embodiment uses a channel attention mechanism to process multi-channel intermediate images to obtain a channel feature image containing channel feature information, and a spatial attention mechanism to process multi-channel intermediate images to obtain a spatial feature image containing spatial feature information. Avoid using a single channel attention mechanism or a spatial attention mechanism to process multi-channel intermediate images, and avoid using a series of channel-space attention mechanisms to process multi-channel intermediate images to obtain a fused weighted feature image that loses part of the channel feature information or part of the spatial feature information. Then use a self-attention mechanism to process multiple fused weighted feature images to focus on key information (for example, the location information of the target in the image). Thereby, a high-fidelity, high-definition target fused image is obtained.
[0125] like Figure 8 The figure shows a flow chart of the steps of detecting the target fusion image in this application. The steps include:
[0126] Step S301: Calculate a first loss value of a target fused image.
[0127] In this embodiment, a loss function is used to detect the target fused image. The loss function includes an intensity loss function and a structural similarity loss function. The intensity loss function is used to calculate the first loss value of the target fused image, and the intensity loss function is:
[0128]
[0129] Among them, F i is the first loss value, H is the height of the image, W is the width of the image, and I f is the target fusion image, I r is the infrared image, I v is a visible light image, ||…|| F is the matrix Frobenius norm, and α is the regularization parameter.
[0130] Step S302: Calculate a second loss value of the target fused image.
[0131] In this embodiment, a structural similarity loss function is used to calculate the second loss value of the target fusion function. The structural similarity loss function is:
[0132] F s =1-SSIM(I f ,I v )
[0133] Among them, F s is the second loss value, and SSIM(…) is the structural similarity measure.
[0134] Step S303: Calculate a target loss value based on the first loss value and the second loss value.
[0135] In this embodiment, F is the target loss value.
[0136] F=F i +μF s
[0137] Here, μ represents the regularization parameter.
[0138] Step S304: Use the target loss value to update the model parameters of the denoising prediction network until the denoising prediction network training is completed.
[0139] The embodiment of the present application can calculate the target loss value based on the intensity loss function and the structural similarity loss function. Then use the target loss value to update the model parameters of the denoising prediction network until the denoising prediction network training is completed. Thus, the denoising prediction network is used to process the multi-channel original image to obtain a more accurate and high-fidelity fused image.
[0140] Please refer to Fig. 9 , is a schematic diagram of the hardware structure of the electronic device 1000 provided in the embodiment of the present application. Fig. 9 As shown, the electronic device 1000 may include a processor 1001 and a memory 1002. The memory 1002 is used to store one or more computer programs 1003. The one or more computer programs 1003 are configured to be executed by the processor 1001. The one or more computer programs 1003 include instructions, and the above instructions can be used to implement the above method in the electronic device 1000.
[0141] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 1000. In other embodiments, the electronic device 1000 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.
[0142] The processor 1001 may include one or more processing units, for example, the processor 1001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0143] The processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. The memory may store instructions or data that the processor 1001 has just used or circulated. If the processor 1001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1001, and thus improves the efficiency of the system.
[0144] In some embodiments, the processor 1001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.
[0145] In some embodiments, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0146] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.
[0147] Among them, the electronic device and computer storage medium provided in this embodiment are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0148] In practical applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0149] In several embodiments provided in the present application, the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0150] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.
[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent printout, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software printout, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.
[0153] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.
Claims
1. A method for fusion of infrared and visible light images, characterized in that: include: Acquire a visible light image and an infrared light image of a location to be photographed; Perform channel stitching on the visible light image and the infrared light image to obtain a multi-channel original image; Performing a denoising operation on the multi-channel original image after adding noise, and then performing a denoising operation to obtain a denoising prediction network, wherein the denoising prediction network is a U-Net network, and the denoising prediction network includes N stages, and the jump connection of each stage is provided with a channel attention mechanism and a spatial attention mechanism in parallel, and N is an integer greater than or equal to 1; Inputting the multi-channel original image into the denoising prediction network, so that each of the N stages of the denoising prediction network outputs N multi-channel intermediate images based on the multi-channel original image, and after the multi-channel intermediate image is output at the i-th stage, the multi-channel intermediate image is converted into a channel feature image based on the channel attention mechanism corresponding to the i-th stage, wherein i is an integer from 1 to N; Converting the multi-channel intermediate image into a spatial feature image based on the spatial attention mechanism corresponding to the i-th stage; Respectively splicing the channel feature image, the spatial feature image and the multi-channel intermediate image of each stage to obtain N fused weighted feature images; The N fused weighted feature images are fused to obtain a target fused image.
2. The infrared-visible light image fusion method according to claim 1, characterized in that: The fusing N fused weighted feature images to obtain a target fused image includes: Extracting an initial weighted feature image from each of the fused weighted feature images; Performing dimension conversion on each of the initial weighted feature images, and fusing the converted multiple initial weighted feature images into a fused initial feature image, wherein the dimension of the fused initial feature image is the same as the dimension of the multi-channel original image; Extracting a region of interest from the fused initial feature image, and obtaining the target fused image based on the region of interest, wherein the region of interest is a region where the photographed object is located in the visible light image.
3. The infrared-visible light image fusion method according to claim 2, characterized in that: The extracting the region of interest in the fused initial feature image and obtaining the target fused image based on the region of interest includes: Extracting a region of interest from the fused initial feature image, and using the region of interest as a fused target feature image; The fusion target feature image is dimensionally transformed to obtain the target fusion image, wherein the dimension of the target fusion image is the same as the dimension of the visible light image.
4. The infrared-visible light image fusion method according to claim 1, characterized in that: The converting the multi-channel intermediate image into a channel feature image based on the channel attention mechanism corresponding to the i-th stage includes: Calculating the average pooling value of the multi-channel intermediate image according to the spatial dimension; Based on the average pooling value, converting the multi-channel intermediate image into a first channel image; Calculating the maximum pooling value of the multi-channel intermediate image according to the spatial dimension; Based on the maximum pooling value, converting the multi-channel intermediate image into a second channel image; Extracting a first eigenvector in the first channel image and a second eigenvector in the second channel image; Calculating channel attention weights of the first feature vector and the second feature vector; Based on the channel attention weights and the multi-channel intermediate image, the channel feature image is obtained.
5. The infrared-visible light image fusion method according to claim 1, characterized in that: The step of converting the multi-channel intermediate image into a spatial feature image based on the spatial attention mechanism corresponding to the i-th stage includes: Calculating the average pooling value of the multi-channel intermediate image according to the channel dimension; Based on the average pooling value, converting the multi-channel intermediate image into a first spatial image; Calculating the maximum pooling value of the multi-channel intermediate image according to the channel dimension; Based on the maximum pooling value, converting the multi-channel intermediate image into a second spatial image; splicing the first spatial image and the second spatial image to obtain a spliced spatial image; Calculating a spatial attention weight of the spliced spatial image; Based on the spatial attention weights and the multi-channel intermediate image, the spatial feature image is obtained.
6. The infrared-visible light image fusion method according to claim 1, characterized in that: After fusing the N fused weighted feature images to obtain a target fused image, the method further includes: Calculating a first loss value and a second loss value of the target fused image; Calculating a target loss value based on the first loss value and the second loss value; The target loss value is used to update the model parameters of the denoising prediction network until the denoising prediction network training is completed.
7. The infrared-visible light image fusion method according to claim 6, characterized in that: The method of calculating the first loss value of the target fused image is: Among them, F i is the first loss value, H is the height value of the image, W is the width value of the image, and I f is the target fused image, I r is the infrared light image, I v is the visible light image, ||…|| F is the matrix Frobenius norm, α is the regularization parameter; The method of calculating the second loss value of the target fused image is: F s =1-SSIM(I f ,I v ) Among them, F s is the second loss value, and SSIM(…) is the structural similarity measure.
8. An infrared and visible light image fusion method, characterized in that: include: Acquire a visible light image and an infrared light image of a location to be photographed; Perform channel stitching on the visible light image and the infrared light image to obtain a multi-channel original image; Performing a denoising operation on the multi-channel original image after adding noise, and then performing a denoising operation to obtain a denoising prediction network, wherein the denoising prediction network is a U-Net network, and the denoising prediction network includes N stages, and the skip connection of each stage is a channel attention mechanism and a spatial attention mechanism set in series, and N is an integer greater than or equal to 1; Inputting the multi-channel original image into the denoising prediction network, so that each of the N stages of the denoising prediction network outputs N multi-channel intermediate images based on the multi-channel original image, and after the multi-channel intermediate image is output at the i-th stage, the multi-channel intermediate image is converted into a channel feature image based on the channel attention mechanism corresponding to the i-th stage, wherein i is an integer from 1 to N; Processing the channel feature image based on the spatial attention mechanism corresponding to the i-th stage to obtain a spatial feature image; Splicing the spatial feature image and the multi-channel intermediate image at each stage to obtain N fused weighted feature images; The N fused weighted feature images are fused to obtain a target fused image.
9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the infrared-visible light image fusion method according to any one of claims 1 to 8.
10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the infrared-visible light image fusion method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Multi-band infrared image fusion method based on Cascade-GAN
CN113762277A
Fundus image optic disk detection method based on UNet network
CN116664546A
Infrared and visible light image fusion method based on progressive multi-branch and improved UNet3 + deep supervision
CN118229548A
Cited By
Significant attention and unsupervised image enhancement dual-drive target detection method
CN122289661A
