Method and system for fusion of infrared image and visible light image
Through multi-level feature extraction and cross-global attention mechanism, the problem of insufficient detail feature extraction in the fusion of infrared images and visible light images is solved, the effect and information richness of image fusion are improved, and the texture and structure information of the fusion image are enhanced.
Patent Information
- Application Number
- CN202311493612.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-11-10
AI Technical Summary
The existing infrared image and visible image fusion algorithms lack the extraction of image detail features, making it difficult to extract global information, reducing the image fusion effect.
Multi-level feature extraction steps are used to extract features using residual dense blocks, combined with the cross-global attention mechanism and channel attention mechanism, image features are restored through weighting operations and decoder, and the detailed description ability and information richness of image fusion are improved.
It improves the feature extraction capability of infrared and visible light images, enhances the texture and structure information of the fused image, and improves the performance and recognition effect of image fusion.
Smart Images

Figure CN117474782B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image fusion, and in particular to a method and system for fusing an infrared image with a visible light image. Background Art
[0002] Due to theoretical and technical limitations, images taken by a single modal sensor cannot effectively and comprehensively describe the imaging scene. Infrared sensors mainly capture the thermal radiation emitted by objects to obtain pictures that highlight the main target, but infrared images ignore textures and are easily affected by noise. In contrast, visible sensors capture reflected light information. Visible images usually contain rich texture and structural information, but are easily interfered by irrelevant information such as strong light. Due to the different imaging mechanisms of the two, visible images usually have higher spatial resolution and image contrast, which are suitable for human visual perception, but they are extremely susceptible to harsh conditions, such as insufficient brightness, heavy rain and other special climates. However, infrared images happen to have better scene anti-interference ability, and can more prominently display targets with temperatures higher than the environment, such as pedestrians. However, infrared images usually have low resolution and poor image detail. By fusing infrared images and visible light images, multiple information can be displayed on one image, highlighting the target, with richer details than a single image and the ability to resist harsh environments.
[0003] The purpose of infrared and visible light image fusion is to carefully fuse the infrared and visible light images of the same scene, while retaining the highlighted targets with thermal radiation information in the infrared image and the high-resolution background texture detail information in the visible light image, making the final fused image more information-rich and thus more conducive to human eye recognition and machine automatic detection.
[0004] However, the existing fusion algorithms of infrared images and visible light images are insufficient in extracting image detail features and have difficulty in extracting global information, thus reducing the effect of image fusion. Summary of the invention
[0005] The present application provides a method and system for fusing infrared images and visible light images. The existing fusion algorithm for infrared images and visible light images is insufficient in extracting image detail features and is difficult to extract global information, thereby reducing the technical problem of the effect of image fusion.
[0006] The present application provides a method for fusing an infrared image and a visible light image, comprising:
[0007] Collect visible light images and corresponding infrared images and build a data set;
[0008] Normalize the visible light image and the infrared image;
[0009] Multi-level feature extraction step: extracting features of the visible light image and the infrared image using the residual dense block to obtain a first visible light feature and a first infrared feature;
[0010] Modal fusion operation step: input the first infrared feature and the first visible light feature of each level after the residual dense block into the modal fusion module, and the modal fusion module obtains the second infrared feature and the second visible light feature through the cross global attention mechanism;
[0011] Weighted operation step: weighting the second visible light feature and the corresponding input first visible light feature and inputting them into the next layer of residual dense block, and at the same time weighting the second infrared feature and the corresponding input first infrared feature and inputting them into the next layer of residual dense block;
[0012] Determine whether the level of the residual dense block extraction branch network is less than the preset level. If so, loop the multi-level feature extraction step to the weighted operation step; if not, add and combine the second infrared feature and the second visible light feature outputted from the last level with the first infrared feature and the first visible light feature of the corresponding input respectively;
[0013] The added features are input into the decoder to restore the image features to obtain a fused image.
[0014] Optionally, the multi-level feature extraction step specifically includes:
[0015] Two 3×3 convolutions with residual dense connections are used as the main channels to preliminarily extract the features of visible light images and infrared images, respectively, to obtain primary visible light features and primary infrared features;
[0016] Use 1×1 convolution to control the number of channels and obtain the main channel visible light features and main channel infrared features;
[0017] The first-order derivative operator is used to extract the information in the visible light image and the infrared image to obtain the first-order visible light feature and the first-order infrared feature, as follows:
[0018]
[0019] Among them, G x and G y are the horizontal gradient and vertical gradient, f is the input image;
[0020] The second-order derivative operator is used to extract the information in the visible light image and the infrared image, and the second-order visible light features and the second-order infrared features are obtained, as follows:
[0021]
[0022] Where f is the input image, is the second-order derivative in the horizontal direction, is the second-order derivative in the vertical direction;
[0023] After the second-order visible light features and the second-order infrared features are processed by maximum pooling, they are added to the input visible light image and infrared image respectively to obtain the intermediate visible light features and the intermediate infrared features;
[0024] Multiplying the intermediate visible light feature with the first-order visible light feature, and multiplying the intermediate infrared feature with the first-order infrared feature;
[0025] After 1×1 convolution multiplication, enhanced infrared features and enhanced visible light features are obtained;
[0026] Adding the enhanced visible light feature to the main channel visible light feature, and adding the enhanced infrared feature to the main channel infrared feature, to obtain a first visible light feature and a first infrared feature;
[0027] The overall algorithm of the multi-level feature extraction step is:
[0028]
[0029] conv n (F i )=conv(C[conv(conv 1 (F i )),conv 1 (F i ),F i ])
[0030] conv 1 (F i )=C[conv(F i ),F i ]
[0031] Among them, conv(·) is the convolution operation, F i and F i+1 are the input and output of the module, ▽ and ▽ 2 They are sober and Laplace operators respectively, C is the concatenation operation, and MaxPool is the maximum pooling operation.
[0032] Optionally, in the modal fusion operation step, the step of acquiring the second visible light feature includes:
[0033] The first visible light feature and the first infrared feature of the input are convolved using a 3x3 convolution, and the key vectors K are obtained respectively. vi and K ir ;
[0034] Use 1x1 convolution to convolve the first visible light feature and the first infrared feature of the input, and obtain the value vector V respectively. vi and V ir ;
[0035] The first visible light feature of the input is taken as Q vi The bond vector K with the first infrared feature ir To splice;
[0036] The concatenated features are input into two consecutive 1×1 convolutions to generate the attention matrix;
[0037] The convolution result is processed by the softmax function and combined with the value vector V of the infrared feature ir multiply;
[0038] The multiplication result is combined with the key vector K of the first infrared feature ir A weighted processing is performed to obtain a second visible light feature.
[0039] Optionally, in the modal fusion operation step, the step of acquiring the second infrared feature includes:
[0040] The first visible light feature and the first infrared feature of the input are convolved using a 3x3 convolution, and the key vectors K are obtained respectively. vi and K ir ;
[0041] Use 1x1 convolution to convolve the first visible light feature and the first infrared feature of the input, and obtain the value vector V respectively. vi and V ir ;
[0042] The first infrared feature of the input is taken as Q ir The key vector K with the first visible light feature vi To splice;
[0043] The concatenated features are input into two consecutive 1×1 convolutions to generate the attention matrix;
[0044] The convolution result is processed by the softmax function and combined with the value vector V of the visible light image vi multiply;
[0045] The multiplication result is added to the key vector K of the first visible light feature vi A weighted processing is performed to obtain a second infrared feature.
[0046] Optionally, after the modal fusion operation step, the following steps are further included:
[0047] Performing a global average pooling operation on the second infrared feature and the second visible light feature;
[0048] Generate a feature map with channel attention based on the channel attention operation;
[0049] The feature map with channel attention generates two learnable parameters through the softmax function:
[0050] ω v ,ω i =Softmax(SE(C[P(F ir ),P(F vi )]))
[0051] Among them, ω v represents the learnable parameters of infrared images, ω i represents the learnable parameters of the visible light image, SE is the channel attention mechanism, C is the connection operation, P(·) is the global average pooling operation, and F ir and F vi The second infrared image and the second visible light image are input;
[0052] Multiplying the generated learnable parameter with the input second infrared feature and the second visible light feature;
[0053] The generated results are connected and added to the second infrared feature and the second visible light feature of the input, and the third visible light feature and the third infrared feature are output:
[0054]
[0055] in, is the weighted output result of the visible light image, is the weighted output result of the infrared image, C is the connection operation, ω v represents the learnable parameters of infrared images, ω i represents the learnable parameters of the visible light image, F ir Indicates the second infrared feature, F vi Represents the second visible light feature.
[0056] Optionally, in the step of performing a global average pooling operation on the second infrared feature and the second visible light feature:
[0057] Spatial feature compression is performed on the second infrared feature map and the second visible light feature map with an input dimension of H×W×C, and global average pooling is implemented in the spatial dimension to obtain a first visible light feature map and a first infrared feature map of 1×1×C.
[0058] Optionally, in the step of generating a feature map with channel attention based on the channel attention operation:
[0059] Through FC fully connected layer learning, the second visible light feature map and the second infrared feature map with channel attention are obtained, and their dimensions are still 1×1×C;
[0060] The first visible light feature map of the channel attention 1×1×C and the second visible light feature of the original input H×W×C are multiplied channel by channel by the learnable parameter w i , output the third visible light feature map with channel attention, and multiply the first infrared feature map with channel attention 1×1×C and the second infrared feature with original input H×W×C by the learnable parameter w channel by channel v , outputting the third infrared feature map with channel attention.
[0061] Optionally, in the step of respectively adding the second infrared feature and the second visible light feature outputted from the last stage to the first infrared feature and the first visible light feature of the corresponding input, and combining them and inputting them into the decoder to restore the image feature to obtain the fused image:
[0062] The combined features are input to the decoder, and convolution operations are performed in sequence using five-level 3×3 convolution kernels. The first four levels use the LeakyReLU activation function, and the last level uses the Tanh activation function to output values between [-1,1].
[0063] Correspondingly, the present application also provides an electronic device, which includes a memory and a processor, the memory being used to store executable program code; the processor being connected to the memory, and running a computer program corresponding to the executable program code by reading the executable program code, so as to execute the steps of the method for fusing infrared images and visible light images as described in any of the aforementioned items.
[0064] Correspondingly, the present application also provides a system for fusing infrared images and visible light images, which includes an electronic device, and the electronic device is any of the electronic devices described above.
[0065] The present application provides a method and system for fusing infrared images and visible light images. The residual connection feature extraction module uses a residual connection to connect the first-order derivative operator and the second-order derivative operator in parallel, thereby improving the feature extraction network's ability to describe details and effectively extracting infrared and visible light image features.
[0066] The cross-global attention mechanism cleverly alleviates the problems of CNN's insufficient ability to extract global information of images and the inability of image-specific encoders to describe the common features of visible light images and infrared images well by cross-modal information of infrared images and visible light images.
[0067] By adopting a feature selector to filter the result features of the cross-global attention, using the channel attention mechanism to filter some information that is not conducive to image fusion, the image fusion performance is improved, and the feature filter is combined with the cross-global attention to form a modal fusion block, finally making the generated feature map have more texture and structural information. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0069] Figure 1 It is a flowchart of the method for fusing infrared images and visible light images provided by the present application;
[0070] Figure 2 It is a flowchart of step S300 in the method for fusing infrared images and visible light images provided in the present application;
[0071] Figure 3 It is a flowchart of step S410 in the method for fusing infrared images and visible light images provided in the present application;
[0072] Figure 4 It is a flowchart of step S430 in the method for fusing infrared images and visible light images provided in the present application;
[0073] Figure 5a-5e It is a schematic diagram of a fused image in the method for fusing an infrared image and a visible light image provided in the present application. DETAILED DESCRIPTION
[0074] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. In addition, it should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application and are not used to limit the present application. In the present application, unless otherwise stated, the directional words used, such as "up", "down", "left", and "right", generally refer to the up, down, left, and right of the device in actual use or working state, specifically the drawing direction in the accompanying drawings.
[0075] The present application provides a method and system for fusing infrared images and visible light images, which are described in detail below. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments of the present application. In the following embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0076] See also Figure 1 The present application provides a method for fusing infrared images and visible light images. The method can carefully fuse infrared images and visible light images of the same scene, while retaining the highlighted targets with thermal radiation information in the infrared image and the background texture detail information with high resolution in the visible light image, so that the final fused image is more information-rich, which is more conducive to human eye recognition and machine automatic detection.
[0077] See also Figure 1-Figure 4 A method for fusing an infrared image with a visible light image specifically comprises the following steps:
[0078] S100, collecting visible light images and corresponding infrared images, and constructing a data set;
[0079] The corresponding visible light images and infrared images are collected as a group of data samples, and a data set is constructed using all the collected data samples. In this application, the data samples are divided into a training set and a test set at a ratio of 7:3.
[0080] S200, normalizing the visible light image and the infrared image;
[0081] S300, multi-level feature extraction step: extracting features of the visible light image and the infrared image using a residual dense block to obtain a first visible light feature and a first infrared feature;
[0082] This application utilizes a feature extraction block with residual dense connection, which makes up for the problem of insufficient image detail extraction of traditional convolution by using first-order and second-order derivative operators to restart features and using residual connection on the main feature extraction channel.
[0083] Step S300 specifically includes the following steps:
[0084] S310, using two 3×3 convolutions with residual dense connections as main channels to preliminarily extract features of the visible light image and the infrared image, respectively, to obtain primary visible light features and primary infrared features;
[0085] Two 3×3 convolutions with residual dense connections are used as the main channels, and LeakyReLU is selected as the activation function.
[0086] S320, using 1×1 convolution to control the number of channels, and obtaining the main channel visible light features and the main channel infrared features;
[0087] S330, using a first-order derivative operator to extract information from the visible light image and the infrared image, and obtaining first-order visible light features and first-order infrared features, as follows:
[0088]
[0089] Among them, G x and G y are the horizontal gradient and vertical gradient, f is the input image;
[0090] S340, using a second-order derivative operator to extract information from the visible light image and the infrared image, and obtaining second-order visible light features and second-order infrared features, as follows:
[0091]
[0092] Where f is the input image, is the second-order derivative in the horizontal direction, is the second-order derivative in the vertical direction;
[0093] In order to make up for the insufficient image feature extraction caused by the convolution operation, the first-order derivative operator and the second-order derivative operator are used in parallel to realize feature restart. The first-order sober operator of the image is an edge detection operator that can extract fine-grained detail information in the image. The second-order derivative operator of the image enhances finer details than the first-order derivative operator.
[0094] S350, performing maximum pooling processing on the second-order visible light features and the second-order infrared features, and then adding them to the input visible light image and the infrared image respectively to obtain intermediate visible light features and intermediate infrared features;
[0095] Although the second-order derivative operator is better than the first-order derivative operator in detail extraction, it is more sensitive to image noise. Therefore, the Laplace operator result is subjected to maximum pooling to retain the main detail features while reducing noise interference, and then added to the original image to eliminate the image discontinuity problem caused by the second-order derivative operator.
[0096] S360, multiplying the intermediate visible light feature with the first-order visible light feature, and multiplying the intermediate infrared feature with the first-order infrared feature;
[0097] Multiply the result with the result of the sober operator to enhance the information on edges and texture details.
[0098] S370, using the image after 1×1 convolution multiplication to obtain enhanced infrared features and enhanced visible light features;
[0099] S380, adding the enhanced visible light feature to the main channel visible light feature, and adding the enhanced infrared feature to the main channel infrared feature to obtain a first visible light feature and a first infrared feature;
[0100] The overall algorithm of the multi-level feature extraction step is:
[0101]
[0102] conv n (F i )=conv(C[conv(conv 1 (F i )),conv 1 (F i ),F i ])
[0103] conv 1 (F i )=C[conv(F i ),F i ]
[0104] Among them, conv(·) is the convolution operation, F i and F i+1 are the input and output of the module respectively. and They are sober and Laplace operators respectively, C is the concatenation operation, and MaxPool is the maximum pooling operation;
[0105] The residual connection feature extraction module uses the residual connection in parallel with the first-order derivative operator and the second-order derivative operator, which improves the feature extraction network's ability to describe details and effectively extracts features of infrared and visible light images.
[0106] S400, modal fusion operation step: inputting the first infrared feature and the first visible light feature of each level after the residual dense block into the modal fusion module, and the modal fusion module obtains the second infrared feature and the second visible light feature through the cross global attention mechanism;
[0107] The cross-global attention mechanism compensates for the encoder's insufficient extraction of multi-modal features by cross-infrared image and visible light image information.
[0108] Step 400 includes the steps of obtaining a second visible light feature and the steps of obtaining a second infrared feature. Step 410 of obtaining a second visible light feature specifically includes the following steps:
[0109] S411, use 3x3 convolution to perform convolution processing on the input first visible light feature and the first infrared feature, and obtain key vectors K respectively vi and K ir ;
[0110] S412, using 1x1 convolution to perform convolution processing on the input first visible light feature and the first infrared feature, and obtain value vectors V vi and V ir ;
[0111] S413, taking the first visible light feature input as Q vi The bond vector K with the first infrared feature ir To splice;
[0112] S414, input the concatenated features into two consecutive 1×1 convolutions to generate an attention matrix;
[0113] S415, processing the convolution result using the softmax function and combining it with the value vector V of the first infrared feature ir multiply;
[0114] S416, multiply the result and the key vector K of the first infrared feature ir Performing weighted processing to obtain a second visible light feature;
[0115] Using the cross global attention mechanism, the input first visible light feature and the first infrared feature will be modified into value vectors V by 1×1 and 3×3 convolutions respectively. vi and V ir and the key vector K vi and K ir For the first visible light feature, the first visible light feature is input as Q vi The bond vector K with the first infrared feature ir The generated feature map has both infrared image features and visible light image features. The feature is then input into two consecutive 1×1 convolutions to generate an attention matrix. The convolution result is then softmaxed and summed with the value vector V of the first infrared feature. ir Multiply them together to generate visible light features that combine the global information of the infrared image, and add this result to the key vector K ir The matrix of is weighted to finally obtain the visible light feature. This result allows the generated visible light feature to better focus on the modal common information provided by the infrared image, strengthens the semantic connection between the visible light image and the infrared image, and thus promotes the interaction of features and the integration of information. The infrared image is also input into the attention mechanism to obtain the infrared image cross-attention result.
[0116] The step 420 of acquiring the second infrared signature includes the following steps:
[0117] S421, use 3x3 convolution to perform convolution processing on the input first visible light feature and the first infrared feature, and obtain key vectors K respectively vi and K ir ;
[0118] S422, use 1x1 convolution to perform convolution processing on the input first visible light feature and the first infrared feature, and obtain value vectors V respectively vi and V ir ;
[0119] S423, taking the input first infrared feature as Q ir The key vector K with the first visible light feature vi To splice;
[0120] S424, input the concatenated features into two consecutive 1×1 convolutions to generate an attention matrix;
[0121] S425, process the convolution result using the softmax function and add it to the value vector V of the first visible light feature vi multiply;
[0122] S426, the multiplication result and the key vector K of the first visible light feature vi Performing weighted processing to obtain a second infrared feature;
[0123] The cross-global attention mechanism cleverly alleviates the problems of CNN's insufficient ability to extract global information of images and the inability of image-specific encoders to describe the common features of visible light images and infrared images well by cross-modal information of infrared images and visible light images.
[0124] S430, result feature screening step, which specifically includes the following steps:
[0125] S431, performing a global average pooling operation on the second infrared feature and the second visible light feature;
[0126] Spatial feature compression is performed on the second infrared feature map and the second visible light feature map with an input dimension of H×W×C, and global average pooling is implemented in the spatial dimension to obtain a first visible light feature map and a first infrared feature map of 1×1×C.
[0127] S432, generating a feature map with channel attention based on the channel attention operation;
[0128] Through FC fully connected layer learning, the second visible light feature map and the second infrared feature map with channel attention are obtained, and their dimensions are still 1×1×C;
[0129] The first visible light feature map of the channel attention 1×1×C and the second visible light feature of the original input H×W×C are multiplied channel by channel by the learnable parameter ω i , output the third visible light feature map with channel attention, and multiply the first infrared feature map with channel attention 1×1×C and the second infrared feature of the original input H×W×C by the learnable parameter ω channel by channel v , outputting the third infrared feature map with channel attention.
[0130] S433, the feature map with channel attention generates two learnable parameters through the softmax function:
[0131] ω v ,ω i =Softmax(SE(C[P(F ir ),P(F vi )]))
[0132] Among them, ω v represents the learnable parameters of infrared images, ω i represents the learnable parameters of the visible light image, SE is the channel attention mechanism, C is the connection operation, P(·) is the global average pooling operation, and F ir and F vi The second infrared feature and the second visible light feature are input;
[0133] S434, multiplying the generated learnable parameter with the input second infrared feature and second visible light feature;
[0134] S435, connect the generated results and add them to the input second infrared feature and second visible light feature, and output the third visible light feature and the third infrared feature:
[0135]
[0136] in, is the weighted output result of the visible light image, is the weighted output result of the infrared image, C is the connection operation, ω v represents the learnable parameters of infrared images, ω i represents the learnable parameters of the visible light image, F ir Indicates the second infrared feature, F vi represents a second visible light feature;
[0137] The generated learnable parameters are multiplied with the input infrared and visible light images to highlight important information, and then the generated results are concatenated and combined with the input feature F ir and F viBy adding them together, the output infrared and visible light images can better combine the common information between multi-modal images, alleviating the shortcomings of the dedicated encoder's insufficient ability to extract common features of multi-modal images.
[0138] For the feature filter, the infrared image and the visible light image are subjected to the attention mechanism, and the feature F ir and F vi Perform global average pooling and channel attention (SE) operations to generate a weight vector. Channel attention is an attention mechanism used to increase the weight of input channels so that the model pays more attention to important channel information.
[0139] By adopting a feature selector to filter the result features of the cross-global attention, using the channel attention mechanism to filter some information that is not conducive to image fusion, the image fusion performance is improved, and the feature filter is combined with the cross-global attention to form a modal fusion block, finally making the generated feature map have more texture and structural information.
[0140] S500, weighted operation step: weighting the second visible light feature and the corresponding input first visible light feature and inputting them into the next layer of residual dense block, and weighting the second infrared feature and the corresponding input first infrared feature and inputting them into the next layer of residual dense block;
[0141] S600, determining whether the level of the residual dense block extraction branch network is less than a preset level, if so, looping the multi-level feature extraction step to the weighted operation step; if not, respectively adding and combining the second infrared feature and the second visible light feature outputted from the last level with the first infrared feature and the first visible light feature of the corresponding input;
[0142] S700, inputting the added and combined features into a decoder to restore image features to obtain a fused image;
[0143] The combined features are input into the decoder, and convolution operations are performed in sequence using five-level 3×3 convolution kernels. The first four levels use the LeakyReLU activation function, and the last level uses the Tanh activation function to set the numerical output range to [-1,1], which matches the general image pixel range and facilitates image restoration.
[0144] In order to verify the rationality of the structure, this application uses the MSRS dataset for testing, divides the MSRS image into 7:3, sets the learning rate to 0.001, sets the batch size to 4, applies the training result parameters to the test set, and compares them with two other methods currently available, namely, image fusion in a high-level visual task cycle: a semantically perceived real-time infrared and visible light image fusion network (SeFusion), and a unified unsupervised image fusion network (U2Fusion). The comparative evaluation results are shown in Table 1, where entropy (EN), mutual information (MI), visual information fidelity (VIF), and image fusion quality (Q abf ) four indicators.
[0145] Figure 5 is a fusion result diagram, where (5a) is an infrared image; (5b) is a visible light image; (5c) is the fusion result of the method proposed in this patent; (5d) is the SeFusion fusion result; (5e) is the U2Fusion fusion result. The results of zooming in on some areas of the image show that the fusion result of our method has better details and edges, and has a better fusion effect.
[0146] Table 1 Comparative experimental indicators
[0147]
[0148] The present application also provides a system for fusing infrared images and visible light images, which includes an electronic device, which includes a memory and a processor, the memory being used to store executable program codes; the processor being connected to the memory, and running a computer program corresponding to the executable program codes by reading the executable program codes, so as to execute the steps of the aforementioned method for fusing infrared images and visible light images.
[0149] The above is a detailed introduction to the method and system for fusing infrared images and visible light images provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for fusing infrared images and visible light images. It is characterized in that include: Collect visible light images and corresponding infrared images to build a data set; Normalize the visible light image and the infrared image; Multi-level feature extraction step: extracting features of the visible light image and the infrared image using the residual dense block to obtain a first visible light feature and a first infrared feature; Modal fusion operation step: input the first infrared feature and the first visible light feature of each level after the residual dense block into the modal fusion module, and the modal fusion module obtains the second infrared feature and the second visible light feature through the cross global attention mechanism; Weighted operation step: weighting the second visible light feature and the corresponding input first visible light feature and inputting them into the next layer of residual dense block, and at the same time weighting the second infrared feature and the corresponding input first infrared feature and inputting them into the next layer of residual dense block; Determine whether the level of the residual dense block extraction branch network is less than the preset level. If so, loop the multi-level feature extraction step to the weighted operation step; if not, add and combine the second infrared feature and the second visible light feature outputted from the last level with the first infrared feature and the first visible light feature of the corresponding input respectively; The added features are input into the decoder to restore the image features to obtain a fused image.
2. The method for fusing infrared images and visible light images according to claim 1, It is characterized in that The multi-level feature extraction step specifically includes: Two 3×3 convolutions with residual dense connections are used as the main channels to preliminarily extract the features of visible light images and infrared images, respectively, to obtain primary visible light features and primary infrared features; Use 1×1 convolution to control the number of channels and obtain the main channel visible light features and main channel infrared features; The first-order derivative operator is used to extract the information in the visible light image and the infrared image to obtain the first-order visible light feature and the first-order infrared feature, as follows: Among them, G x and G y are the horizontal gradient and vertical gradient, f is the input image; The second-order derivative operator is used to extract the information in the visible light image and the infrared image to obtain the second-order visible light features and the second-order infrared features, as follows: Where f is the input image, is the second-order derivative in the horizontal direction, is the second-order derivative in the vertical direction; After the second-order visible light features and the second-order infrared features are processed by maximum pooling, they are added to the input visible light image and infrared image respectively to obtain the intermediate visible light features and the intermediate infrared features; Multiplying the intermediate visible light feature with the first-order visible light feature, and multiplying the intermediate infrared feature with the first-order infrared feature; Using the features after 1×1 convolution multiplication, enhanced infrared features and enhanced visible light features are obtained; The enhanced visible light feature is added to the main channel visible light feature, and the enhanced infrared feature is added to the main channel infrared feature to obtain a first visible light feature and a first infrared feature; The overall algorithm of the multi-level feature extraction step is: conv n (F i )=conv(C[conv(conv 1 (F i )),conv 1 (F i ),F i ]) conv 1 (F i )=C[conv(F i ),F i ] Among them, conv(·) is the convolution operation, F i and F i+1 are the input and output of the module respectively. and They are sober and Laplace operators respectively, C is the concatenation operation, and MaxPool is the maximum pooling operation.
3. The method for fusing infrared images and visible light images according to claim 1, It is characterized in that In the modal fusion operation step, the step of obtaining the second visible light feature includes: Use 3x3 convolution to convolve the first visible light feature and the first infrared feature of the input to obtain the key vector K vi and K ir ; Use 1x1 convolution to convolve the first visible light feature and the first infrared feature of the input to obtain the value vector V vi and V ir ; The first visible light feature of the input is taken as Q vi The bond vector K with the first infrared feature ir To splice; The concatenated features are input into two consecutive 1×1 convolutions to generate the attention matrix; The convolution result is processed by the softmax function and combined with the value vector V of the infrared image ir multiply; The multiplication result is combined with the key vector K of the first infrared feature ir Perform weighted processing to obtain the second visible light feature.
4. The method for fusing infrared images and visible light images according to claim 1, It is characterized in that In the modal fusion operation step, the step of obtaining the second infrared feature includes: Use 3x3 convolution to convolve the first visible light feature and the first infrared feature of the input to obtain the key vector K vi and K ir ; Use 1x1 convolution to convolve the first visible light feature and the first infrared feature of the input to obtain the value vector V vi and V ir ; The first infrared feature of the input is taken as Q ir The key vector K with the first visible light feature vi To splice; The concatenated features are input into two consecutive 1×1 convolutions to generate the attention matrix; The convolution result is processed by the softmax function and combined with the value vector V of the visible light image vi multiply; The multiplication result is added to the key vector K of the first visible light feature vi Perform weighted processing to obtain the second infrared feature.
5. The method for fusing infrared images and visible light images according to claim 1, It is characterized in that After the modal fusion operation step, the method further includes the following steps: Performing a global average pooling operation on the second infrared feature and the second visible light feature; Generate a feature map with channel attention based on the channel attention operation; The feature map with channel attention generates two learnable parameters through the softmax function: oh v ,oh i =Softmax(SE(C[P(F ir ),P(F vi )])) Among them, ω v represents the learnable parameters of infrared images, ω i represents the learnable parameters of the visible light image, SE is the channel attention mechanism, C is the connection operation, P(·) is the global average pooling operation, and F ir and F vi The second infrared feature and the second visible light feature are input; Multiplying the generated learnable parameter with the input second infrared feature and the second visible light feature; The generated results are connected and added to the second infrared feature and the second visible light feature of the input, and the third visible light feature and the third infrared feature are output: in, is the weighted output result of the visible light image, is the weighted output result of the infrared image, C is the connection operation, ω v represents the learnable parameters of infrared images, ω i represents the learnable parameters of the visible light image, F ir Indicates the second infrared feature, F vi Represents the second visible light feature.
6. The method for fusing infrared images and visible light images according to claim 5, It is characterized in that In the step of performing a global average pooling operation on the second infrared feature and the second visible light feature: Spatial feature compression is performed on the second infrared feature map and the second visible light feature map with an input dimension of H×W×C, and global average pooling is implemented in the spatial dimension to obtain a first visible light feature map and a first infrared feature map of 1×1×C.
7. The method for fusing infrared images and visible light images according to claim 6, It is characterized in that In the step of generating a feature map with channel attention based on the channel attention operation: Through FC fully connected layer learning, the second visible light feature map and the second infrared feature map with channel attention are obtained, and their dimensions are still 1×1×C; The first visible light feature map of the channel attention 1×1×C and the second visible light feature of the original input H×W×C are multiplied channel by channel by the learnable parameter ω i , output the third visible light feature map with channel attention, and multiply the first infrared feature map with channel attention 1×1×C and the second infrared feature of the original input H×W×C by the learnable parameter ω channel by channel v , outputting the third infrared feature map with channel attention.
8. The method for fusing infrared images and visible light images according to claim 1, It is characterized in that In the step of respectively adding the second infrared feature and the second visible light feature outputted from the last stage to the first infrared feature and the first visible light feature of the corresponding input, and combining them and inputting them into the decoder to restore the image feature to obtain a fused image: The combined features are input to the decoder, and convolution operations are performed in sequence using five-level 3×3 convolution kernels. The first four levels use the LeakyReLU activation function, and the last level uses the Tanh activation function to output values between [-1,1].
9. An electronic device, It is characterized in that include A memory for storing executable program codes; as well as A processor is connected to the memory, and runs a computer program corresponding to the executable program code by reading the executable program code, so as to execute the steps of the method for fusing infrared images and visible light images as described in any one of claims 1 to 8.
10. A fusion system of infrared image and visible light image, It is characterized in that An electronic device comprising the electronic device described in claim 9.
Citation Information
Cited By
A frequency domain sensing and region focusing RGB-infrared fusion network
CN123094254A
