A method for fusing infrared image and visible light image

Abstract

The present application provides a method for fusing infrared images and visible light images, which includes: inputting the infrared image and the visible light image into a single-modal feature extraction branch network respectively; obtaining infrared features and visible light features by using a feature extraction module with a residual connection; a hybrid attention mechanism module operation step; obtaining single-modal infrared features and single-modal visible light features; obtaining multi-modal infrared features and multi-modal visible light features; a hybrid modal feature extraction step; judging whether the level of the single-modal feature extraction branch network is less than a preset level, if so, looping the hybrid attention mechanism module operation step to the hybrid modal feature extraction step, if not, respectively adding the single-modal infrared features and single-modal visible light features obtained by the last level of convolution to the multi-modal infrared features and multi-modal visible light features obtained in the last layer of the Transformer module and then splicing them; inputting the spliced ​​features into a decoder network to generate a fused image.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method for fusing infrared images and visible light images. Background Art

[0002] Image fusion technology is to fuse complementary information from different source images into a high-quality image to facilitate subsequent visual tasks. The fusion of infrared and visible light images has been widely used in the fields of salient object detection, semantic segmentation, face recognition, etc.

[0003] In the prior art, due to the limitation of hardware equipment, the image of a sensor can only represent part of the information. For example, the reflected light information captured by the visible light sensor can describe the texture of the scene and has a high spatial resolution, but it is greatly affected by light and shadow. In addition, the thermal radiation information captured by the infrared sensor is not sensitive to light, can reflect the basic properties of the scene and objects, and can obtain significant information of high thermal radiation targets, but the spatial resolution is low and it is easily affected by noise.

[0004] Therefore, the traditional fusion of infrared and visible light images cannot highlight hot targets while maintaining texture details and high image resolution, and cannot meet downstream computer vision tasks. Summary of the invention

[0005] The present application provides a method for fusing infrared images and visible light images to solve the technical problem that the fusion of traditional infrared and visible light images cannot highlight thermal targets while maintaining texture details and high image resolution of the image, and cannot meet the requirements of downstream computer vision tasks.

[0006] The present application provides a method for fusing an infrared image and a visible light image, comprising:

[0007] The infrared image and the visible light image are normalized and respectively input into a single-modal feature extraction branch network, wherein the single-modal feature extraction branch network includes a feature extraction module and a hybrid attention mechanism module;

[0008] The feature extraction module of the residual connection is used to extract the basic features of the infrared image and the basic features of the visible light image, and the infrared features and the visible light features are obtained;

[0009] The operation steps of the hybrid attention mechanism module are as follows: the infrared features and the visible light features are input into the hybrid attention mechanism module, and the infrared global features and the infrared local features, as well as the visible light global features and the visible light local features are obtained respectively;

[0010] Perform matrix addition and matrix multiplication operations on the infrared global features and the infrared local features, concatenate and reduce the dimensions of the operation results, and then use a jump connection method to add the infrared features to obtain a single-mode infrared feature;

[0011] Perform matrix addition and matrix multiplication operations on the visible light global feature and the visible light local feature, concatenate and reduce the dimension of the operation results, and then use a jump connection to add the visible light feature to obtain a single-mode visible light feature;

[0012] The unimodal infrared features and unimodal visible light features obtained by the first-level convolution are added and input into the Transformer module for information integration to obtain multimodal infrared features and multimodal visible light features.

[0013] Mixed modal feature extraction step: Use the SIM module to generate corresponding correlation weights for the multimodal infrared features and the multimodal visible light features according to the single-modal infrared features and the single-modal visible light features extracted by the corresponding mixed attention mechanism module, and inject them into the single-modal infrared features and the single-modal visible light features respectively;

[0014] Determine whether the level of the unimodal feature extraction branch network is less than the preset level. If so, loop the hybrid attention mechanism module operation step to the hybrid modal feature extraction step. If not, add the unimodal infrared features and unimodal visible light features obtained by the last level of convolution to the multimodal infrared features and multimodal visible light features obtained in the last layer of the Transformer module and then perform splicing processing;

[0015] The concatenated features are input into the decoder network to generate a fused image.

[0016] Optionally, in the step of extracting basic features of the infrared image and basic features of the visible light image using the feature extraction module of the residual connection, and obtaining infrared features and visible light features,

[0017] Use 3x3 convolution to convolve the infrared image and visible light image respectively;

[0018] The convolved infrared image and visible light image are convolved using 1x1 convolution to obtain infrared features and visible light features.

[0019] Optionally, in the hybrid attention mechanism module operation step, the step of obtaining infrared global features includes:

[0020] Input the infrared features into the hybrid attention mechanism module and use them as the query vector Q;

[0021] Use 3x3 convolution to convolve the infrared features and obtain the key vector K;

[0022] Use 1x1 convolution to convolve the infrared features and obtain the value vector V;

[0023] Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0024] Transpose the attention matrix to obtain the transposed matrix;

[0025] Perform matrix multiplication operation on the attention matrix and the transposed matrix, and then perform softmax function operation to obtain the weight matrix of global channel attention;

[0026] Perform matrix transposition and transpose operations on the value vector V, and perform matrix multiplication operation on the resulting matrix and the weight matrix;

[0027] The matrix obtained by multiplication is successively transposed and transposed to obtain the infrared global feature.

[0028] Optionally, in the hybrid attention mechanism module operation step, the step of obtaining infrared local features includes:

[0029] Input the infrared features into the hybrid attention mechanism module and use them as the query vector Q;

[0030] Use 3x3 convolution to convolve the infrared features and obtain the key vector K;

[0031] Use 1x1 convolution to convolve the infrared features and obtain the value vector V;

[0032] Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0033] The attention matrix is ​​processed by global average pooling and global maximum pooling respectively, and the obtained feature vector is processed by channel splicing;

[0034] The concatenated matrix is ​​reduced in dimension using 1x1 convolution, and the resulting matrix is ​​element-wise multiplied with the value vector V to obtain the infrared local features of the local channel attention.

[0035] Optionally, in the hybrid attention mechanism module operation step, the step of obtaining the visible light global feature includes:

[0036] The visible light features are input into the hybrid attention mechanism module and used as the query vector Q;

[0037] Use 3x3 convolution to convolve the visible light features and obtain the key vector K;

[0038] Use 1x1 convolution to convolve the visible light features and obtain the value vector V;

[0039] Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0040] Transpose the attention matrix to obtain the transposed matrix;

[0041] Perform matrix multiplication operation on the attention matrix and the transposed matrix, and then perform softmax function operation to obtain the weight matrix of global channel attention;

[0042] Perform matrix transposition and transpose operations on the value vector V, and perform matrix multiplication operation on the resulting matrix and the weight matrix;

[0043] The matrix obtained by multiplication is successively transposed and transposed to obtain the global features of visible light.

[0044] Optionally, the hybrid attention mechanism module operation steps include:

[0045] The visible light features are input into the hybrid attention mechanism module and used as the query vector Q;

[0046] Use 3x3 convolution to convolve the visible light features and obtain the key vector K;

[0047] Use 1x1 convolution to convolve the visible light features and obtain the value vector V;

[0048] Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0049] The attention matrix is ​​processed by global average pooling and global maximum pooling respectively, and the obtained feature vector is processed by channel splicing;

[0050] The concatenated matrix is ​​reduced in dimension using a 1x1 convolution, and the resulting matrix is ​​element-wise multiplied with the value vector V to obtain the visible light local features of the local channel attention.

[0051] Optionally, the attention matrix is ​​enhanced using a feature enhancement module based on the LoG operator, which includes the following steps:

[0052] The infrared image is low-pass filtered using a two-dimensional LoG function with a Gaussian standard deviation of σ; the LoG function is:

[0053]

[0054] Where (x, y) represents the horizontal and vertical coordinates of the pixel, and σ represents the Gaussian standard deviation;

[0055] Subtract the LoG operator convolution infrared attention matrix from the doubled attention matrix and obtain the enhanced infrared attention matrix,

[0056]

[0057] represents the enhanced infrared attention matrix, F ir (x,y) represents the infrared attention matrix.

[0058] Optionally, the attention matrix is ​​enhanced using a feature enhancement module based on the LoG operator, which includes the following steps:

[0059] The infrared image is low-pass filtered using a two-dimensional LoG function with a Gaussian standard deviation of σ; the LoG function is:

[0060]

[0061] Where (x, y) represents the horizontal and vertical coordinates of the pixel, and σ represents the Gaussian standard deviation;

[0062] The attention matrix is ​​added to the LoG operator convolution visible light attention matrix, and the enhanced visible light attention matrix is ​​obtained.

[0063]

[0064] in, represents the enhanced visible light attention matrix, F vis (x,y) represents the visible light attention matrix.

[0065] Optionally, in the mixed modal feature extraction step, the feature extraction step of infrared features includes the following steps:

[0066] Perform 3x3 convolution operation on multimodal infrared features and multimodal visible light features;

[0067] The multi-modal infrared features after dimensionality reduction are channel-joined with the single-modal infrared features, and a 3x3 convolution operation is performed. At the same time, the multi-modal visible light features after dimensionality reduction are channel-joined with the single-modal visible light features, and a 3x3 convolution operation is performed.

[0068] The channel attention module is used to learn the convolutional infrared features and visible light features into the scaling matrix λ. The learning of the scaling matrix λ is:

[0069] λ=SE(Conv3×3 (C(F i out ,(Conv 3×3 (F i trans )))))

[0070] λ represents the scaling matrix, SE represents the channel attention module, and F i trans represents the output of the i-th level Transformer module, F i out represents the output of the i-th level of the unimodal feature extraction branch network, Conv represents the convolution operation, and C represents the channel concatenation operation;

[0071] Use 3x3 convolution to learn the bias matrix β of the multi-modal infrared features after dimensionality reduction and the multi-modal visible light features after dimensionality reduction. The learning of the bias matrix β is:

[0072] β=Conv 3×3 (Conv 3×3 (F i trans ))

[0073] β represents the bias matrix, F i trans represents the output of the i-th level Transformer module, and Conv represents the convolution operation;

[0074] The layer-normalized single-mode infrared feature is multiplied by the scaling matrix λ, and then added, and then the bias matrix β is added, so as to be injected into the next level of the single-mode infrared feature; at the same time, the layer-normalized single-mode visible light feature is multiplied by the scaling matrix λ, and then added, and then the bias matrix β is added, so as to be injected into the next level of the single-mode visible light feature;

[0075] The formula is:

[0076]

[0077] in, represents the input of the i+1th level of the unimodal feature extraction branch network, F i out represents the output of the i-th level of the unimodal feature extraction branch network, λ represents the scaling matrix, and β represents the bias matrix.

[0078] Optionally, after the step of inputting the spliced ​​features into the decoder network and generating a fused image, the step further includes:

[0079] Generate a mask M using the infrared image ir , mask M ir It is expressed as:

[0080]

[0081] Among them, I ir Indicates infrared image, I ir represents the mean value of the infrared image;

[0082] Calculate the Pearson correlation coefficient between the infrared image and the fused image or between the visible light image and the fused image. The formula is:

[0083]

[0084] Among them, (i, j) represents the vertical and horizontal coordinates of the pixel, H, W represents the height and width of the image, I f (i,j) represents the pixel value of the fused image at (i,j), represents the pixel mean of the fused image, I x (i, j) represents the pixel value of the infrared image or visible light image at (i, j), Represents the mean value of source image pixels;

[0085] Calculate the total loss function, which specifically includes the following steps:

[0086] The mask M ir Respectively with the source infrared image I ir and fused image I f Multiply and obtain the salient area of ​​the infrared image and the salient area of ​​the fused image;

[0087] Calculate the significant correlation coefficient between the significant area of ​​the infrared image and the significant area of ​​the fused image;

[0088] The texture information area of ​​the visible light image is multiplied by the visible light image and the fused image respectively. The texture information area of ​​the visible light image is (1-M ir );

[0089] Calculate the texture correlation coefficient between the texture information area of ​​the visible light image and the fused image;

[0090] The significant correlation coefficient is summed with the texture correlation coefficient, and the total similarity loss is obtained; the formula of the total similarity loss is:

[0091] L cc =1-CC(M ir i f ,M ir i ir )+1-CC((1-M ir )e f ,(1-M ir )e vi)

[0092] Among them, L cc represents the similarity loss, M ir Indicates mask, I f represents the fused image, e represents the multiplication of corresponding elements, I ir Indicates infrared image, I vi represents a visible light image;

[0093] Calculate the strength loss, which includes the following steps:

[0094] Calculate the first Manhattan distance of the difference between the salient area of ​​the infrared image and the salient area of ​​the fused image;

[0095] Calculate the second Manhattan distance of the difference between the texture information area of ​​the visible light image and the texture of the fused image;

[0096] The first Manhattan distance is added to the second Manhattan distance to obtain the intensity loss; the formula for the intensity loss is:

[0097] L pixel =||(M ir i f )-(M ir i ir )||1+||((1-M ir )e f )-((1-M ir )e vi )||1

[0098] Among them, L pixel Indicates strength loss, M ir Indicates mask, I f represents the fused image, e represents the multiplication of corresponding elements, I ir Indicates infrared image, I vi represents visible light image; || ||1 represents Manhattan distance;

[0099] Calculate the texture loss, the formula is:

[0100]

[0101] in represents the absolute value of the fused image gradient, Represents the absolute value of the visible light image gradient, Indicates the absolute value of the infrared image gradient, and max() indicates the maximum value between the two;

[0102] Calculate the total loss function, the formula is:

[0103] L total =aLcc +μL piexl +δL texture

[0104] Among them, a, μ, δ represent the setting parameters, L cc represents the similarity loss, L pixel Indicates strength loss, L texture Indicates texture loss.

[0105] The present application provides a method for fusing infrared images and visible light images. Texture enhancement and intensity enhancement convolution based on the LoG operator are designed on the hybrid attention mechanism module, and the global self-attention mechanism and the local self-attention mechanism are mixed for use, thereby improving the network's ability to describe fine-grained features and effectively extracting features from infrared images and visible light images.

[0106] A multimodal auxiliary network based on the Transformer module and the SIM module was designed, which cleverly solved the problem of no information interaction among different modal branches. Taking into account the similarity and complementarity between different modal information, multimodal information was injected into the single modal branch in an adaptive linear transformation manner, solving the problem of insufficient ability of the CNN network to extract remote information.

[0107] The mask-based similarity loss function enables the fusion result to well preserve the salient information of the infrared image and the detail information of the visible light image, thus improving the performance of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0109] Figure 1 It is a flow chart of the infrared image and visible light image fusion method provided by the present application;

[0110] Figure 2 It is a schematic diagram of the process of obtaining infrared global features in the infrared image and visible light image fusion method provided by the present application;

[0111] Figure 3 It is a schematic diagram of the process of obtaining infrared local features in the infrared image and visible light image fusion method provided by the present application;

[0112] Figure 4 It is a flowchart of step S600 in the infrared image and visible light image fusion method provided by the present application;

[0113] Figure 5 It is a flowchart of step S900 in the infrared image and visible light image fusion method provided by the present application;

[0114] FIG. 6 is a comparison diagram of image fusion results in the infrared image and visible light image fusion method provided in the present application. DETAILED DESCRIPTION

[0115] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. In addition, it should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application and are not used to limit the present application. In the present application, unless otherwise stated, the directional words used, such as "up", "down", "left", and "right", generally refer to the up, down, left, and right of the device in actual use or working state, specifically the drawing direction in the accompanying drawings.

[0116] The present application provides a method for fusing infrared images and visible light images, which are described in detail below. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments of the present application. In the following embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0117] See also Figure 1-Figure 5 The present application provides a method for fusing an infrared image and a visible light image, which specifically includes the following steps:

[0118] S100, normalizing the infrared image and the visible light image, and inputting them into a single-modal feature extraction branch network respectively, wherein the single-modal feature extraction branch network includes a feature extraction module and a hybrid attention mechanism module;

[0119] The unimodal feature extraction branch network can rely on limited attention resources to obtain important information from the features to be extracted, thereby quickly acquiring more important texture and structural information while reducing the interference of redundant irrelevant information.

[0120] S200, extracting basic features of the infrared image and basic features of the visible light image using a feature extraction module with residual connection, and obtaining infrared features and visible light features;

[0121] The feature extraction module is composed of 1x1 convolution and 3x3 convolution, which uses a simple skip connection structure to extract the basic information of the input features and enhance feature reuse.

[0122] Step S200 includes the following steps:

[0123] S210, performing convolution processing on the infrared image and the visible light image respectively by using 3x3 convolution;

[0124] S220, using 1x1 convolution to perform convolution processing on the convolved infrared image and the visible light image respectively, and obtain infrared features and visible light features;

[0125] S300, hybrid attention mechanism module operation step: inputting infrared features and visible light features into the hybrid attention mechanism module, and obtaining infrared global features and infrared local features, as well as visible light global features and visible light local features respectively;

[0126] The hybrid attention mechanism module contains dynamic context global channel attention mechanism and dynamic context local channel attention mechanism, so it can extract global features and local features of the image.

[0127] In step S300, the step of obtaining infrared global features includes:

[0128] S311, input the infrared feature into the hybrid attention mechanism module and use it as the query vector Q;

[0129] S312, performing convolution processing on the infrared feature using 3x3 convolution, and obtaining a key vector K;

[0130] S313, performing convolution processing on the infrared feature using 1x1 convolution, and obtaining a value vector V;

[0131] S314, concatenate the query vector Q and the key vector K through channels, and obtain an attention matrix after two consecutive 1x1 convolutions, so as to mine their internal contextual relationship;

[0132] S315, using a feature enhancement module based on the LoG operator to enhance the attention matrix, which can make up for the lack of image feature information extraction capability caused by pure convolution operation;

[0133] S316, transposing the enhanced attention matrix to obtain a transposed matrix;

[0134] S317, performing a matrix multiplication operation on the attention matrix and the transposed matrix, and performing a softmax function operation to obtain a weight matrix of the global channel attention;

[0135] S318, performing a matrix transposition operation and a transpose operation on the value vector V, and performing a matrix multiplication operation on the matrix obtained by the operation and the weight matrix;

[0136] S319, performing a transposition operation and a transpose operation on the matrix obtained by multiplication, and obtaining an infrared global feature;

[0137] The steps of obtaining infrared local features include:

[0138] S321, input the infrared feature into the hybrid attention mechanism module and use it as the query vector Q;

[0139] S322, performing convolution processing on the infrared feature using 3x3 convolution, and obtaining a key vector K;

[0140] S323, performing convolution processing on the infrared feature using 1x1 convolution, and obtaining a value vector V;

[0141] S324, concatenate the query vector Q and the key vector K through channels, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0142] S325, using a feature enhancement module based on the LoG operator to enhance the attention matrix, which can make up for the lack of image feature information extraction capability caused by pure convolution operation;

[0143] S326, performing global average pooling and global maximum pooling on the enhanced attention matrix, and performing channel splicing on the obtained feature vector;

[0144] S327, using 1x1 convolution to reduce the dimension of the concatenated matrix, and performing element-by-element multiplication operation on the obtained matrix and the value vector V to obtain infrared local features of local channel attention;

[0145] Step S315 and step S325 specifically include the following steps:

[0146] The infrared image is low-pass filtered using a two-dimensional LoG function with a Gaussian standard deviation of σ; the LoG function is:

[0147]

[0148] Where (x, y) represents the horizontal and vertical coordinates of the pixel, and σ represents the Gaussian standard deviation;

[0149] In order to avoid falling into the local field of view and extracting false edges, we use σ=1.4 and the convolution kernel size is 9x9, so we infer the convolution template as follows:

[0150]

[0151] Subtract the LoG operator convolution infrared attention matrix from the doubled attention matrix and obtain the enhanced infrared attention matrix,

[0152]

[0153] represents the enhanced infrared attention matrix, F ir (x,y) represents the infrared attention matrix;

[0154] Considering that there will be a lot of noise in infrared images, in order to avoid being affected by noise, we designed a feature enhancement module based on the LoG operator. The Laplace operator is a high-pass filter that can highlight the areas with rapid intensity changes in the image, but as a second-order derivative operator, it is very sensitive to noise and false edges. Therefore, in order to overcome the shortcomings of the Laplace operator, a Gaussian filter is usually used to low-pass filter the image, and then the Laplace operator is used to extract the edge. Therefore, the LoG operator can also be used as a high-pass filter. We first use the Gaussian filter to convolve with the Laplace filter, and then get the hybrid filter, which is a two-dimensional LoG function centered at 0 and with a Gaussian standard deviation of σ.

[0155] The steps for obtaining visible light global features are similar to those for obtaining infrared global features, and specifically include:

[0156] S331, input the visible light feature into the hybrid attention mechanism module and use it as the query vector Q;

[0157] S332, performing convolution processing on the visible light feature using 3x3 convolution, and obtaining a key vector K;

[0158] S333, performing convolution processing on the visible light feature using 1x1 convolution, and obtaining a value vector V;

[0159] S334, concatenate the query vector Q and the key vector K through channels, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0160] S335, using a feature enhancement module based on the LoG operator to enhance the attention matrix, which can make up for the lack of image feature information extraction capability caused by pure convolution operation;

[0161] S336, transposing the enhanced attention matrix to obtain a transposed matrix;

[0162] S337, performing matrix multiplication operation on the attention matrix and the transposed matrix, and performing softmax function operation to obtain a weight matrix of global channel attention;

[0163] S338, performing a matrix transposition operation and a transpose operation on the value vector V, and performing a matrix multiplication operation on the matrix obtained by the operation and the weight matrix;

[0164] S339, performing a transposition operation and a transpose operation on the matrix obtained by multiplication, and obtaining a visible light global feature;

[0165] The steps of obtaining the visible light local features are similar to the steps of obtaining the infrared local features, which specifically include:

[0166] S341, input the visible light feature into the hybrid attention mechanism module and use it as the query vector Q;

[0167] S342, performing convolution processing on the visible light feature using 3x3 convolution, and obtaining a key vector K;

[0168] S343, performing convolution processing on the visible light feature using 1x1 convolution, and obtaining a value vector V;

[0169] S344, concatenate the query vector Q and the key vector K through channels, and obtain the attention matrix after two consecutive 1x1 convolutions;

[0170] S345, using a feature enhancement module based on the LoG operator to enhance the attention matrix, which can make up for the lack of image feature information extraction capability caused by pure convolution operation;

[0171] S346, performing global average pooling and global maximum pooling on the enhanced attention matrix, and performing channel splicing on the obtained feature vector;

[0172] S347, using 1x1 convolution to reduce the dimension of the concatenated matrix, and performing element-by-element multiplication operation on the obtained matrix and the value vector V to obtain the visible light local features of the local channel attention;

[0173] Step S335 and step S345 specifically include the following steps:

[0174] The infrared image is low-pass filtered using a two-dimensional LoG function with a Gaussian standard deviation of σ; the LoG function is:

[0175]

[0176] Where (x, y) represents the horizontal and vertical coordinates of the pixel, and σ represents the Gaussian standard deviation;

[0177] The attention matrix is ​​added to the LoG operator convolution visible light attention matrix, and the enhanced visible light attention matrix is ​​obtained.

[0178]

[0179] in, represents the enhanced visible light attention matrix, F vis(x,y) represents the visible light attention matrix;

[0180] Infrared images and visible light images have different modal information. Specifically, we hope to retain more intensity information of infrared images and more detailed texture information of visible light images. Therefore, we have different operations on different modal information in the feature enhancement module.

[0181] S400, step of obtaining a single-mode infrared feature and a single-mode visible light feature: performing matrix addition and matrix multiplication operations on the infrared global feature and the infrared local feature, concatenating and reducing the dimension of the operation results, and then performing addition operations with the infrared feature by using a skip connection method to obtain a single-mode infrared feature;

[0182] Perform matrix addition and matrix multiplication operations on the visible light global feature and the visible light local feature, concatenate and reduce the dimension of the operation results, and then use a jump connection to add the visible light feature to obtain a single-mode visible light feature;

[0183]

[0184] F local =Conv 1×1 (C(GMP(F en ),GAP(F en )))⊙V

[0185]

[0186] Among them, R(·) represents the reshape operation, T(·) represents the transpose operation, S(·) represents the softmax operation, and Conv 1×1 (·) represents a 1x1 convolution operation, GMP(·) represents a global maximum pooling operation, GAP(·) represents a global average pooling operation, and F en represents the enhanced attention matrix, F global represents the global attention matrix, F local represents the local attention matrix.

[0187] S500, adding the unimodal infrared feature and the unimodal visible light feature obtained by the first-level convolution, and inputting them into the Transformer module for information integration to obtain a multimodal infrared feature and a multimodal visible light feature;

[0188] Since the unimodal feature extraction branch network cannot pay attention to global information and does not take into account the similarities and complementarities between modalities, the Transformer based on the multi-head attention mechanism can perform global information interaction on each position of the input feature and obtain the correlation weight of each feature with other features. Such global information consideration enables it to better perceive the global context and better capture the relationship between multimodal information.

[0189] S600, hybrid modal feature extraction step: using the SIM module to generate corresponding correlation weights for the multimodal infrared features and the multimodal visible light features according to the single modal infrared features and the single modal visible light features extracted by the corresponding hybrid attention mechanism module, and injecting them into the single modal infrared features and the single modal visible light features respectively;

[0190] Step S600 specifically includes the following steps:

[0191] S610, performing a 3x3 convolution operation on the multimodal infrared feature and the multimodal visible light feature;

[0192] S620, channel-joining the multi-modal infrared features after dimensionality reduction and the single-modal infrared features, and performing a 3x3 convolution operation; at the same time, channel-joining the multi-modal visible light features after dimensionality reduction and the single-modal visible light features, and performing a 3x3 convolution operation;

[0193] S630, respectively use the channel attention module to learn the convolved infrared features and visible light features into a scaling matrix λ, and the learning of the scaling matrix λ is:

[0194] λ=SE(Conv 3×3 (C(F i out ,(Conv 3×3 (F i trans )))))

[0195] λ represents the scaling matrix, SE represents the channel attention module, and F i trans represents the output of the i-th level Transformer module, F i out represents the output of the i-th level of the unimodal feature extraction branch network, Conv represents the convolution operation, and C represents the channel concatenation operation;

[0196] S640, using 3x3 convolution to respectively learn the bias matrix β of the multi-modal infrared features after dimensionality reduction and the multi-modal visible light features after dimensionality reduction, the learning of the bias matrix β is:

[0197] β=Conv 3×3 (Conv3×3 (F i trans ))

[0198] β represents the bias matrix, F i trans represents the output of the i-th level Transformer module, and Conv represents the convolution operation;

[0199] S650, multiplying the layer-normalized single-mode infrared feature by the scaling matrix λ, adding the resultant and then adding the bias matrix β, so as to be injected into the next level of the single-mode infrared feature; at the same time, multiplying the layer-normalized single-mode visible light feature by the scaling matrix λ, adding the resultant and then adding the bias matrix β, so as to be injected into the next level of the single-mode visible light feature;

[0200] The formula is:

[0201]

[0202] in, represents the input of the i+1th level of the unimodal feature extraction branch network, F i out represents the output of the i-th level of the unimodal feature extraction branch network, λ represents the scaling matrix, and β represents the bias matrix;

[0203] A total of three Transformer modules are designed in the multimodal auxiliary network, corresponding to the first three hybrid attention mechanism modules in the unimodal feature extraction branch network. We first add the infrared features and visible light features obtained by the first-level convolution in the unimodal branch network and input them into the auxiliary network. After the mixed information is integrated through the Transformer block, the SIM module is used to generate the corresponding similarity weights for feature injection based on the unimodal features extracted by the corresponding hybrid attention mechanism module to help the backbone network organize global information and multimodal features.

[0204] S700, determining whether the level of the unimodal feature extraction branch network is less than the preset level, if so, looping the hybrid attention mechanism module operation step to the hybrid modal feature extraction step, if not, respectively adding the unimodal infrared features and unimodal visible light features obtained by the last level of convolution to the multimodal infrared features and multimodal visible light features obtained in the last layer of the Transformer module and then performing splicing processing;

[0205] S800, input the spliced ​​features into the decoder network and generate a fused image;

[0206] S900, the step of calculating the total loss function specifically comprises the following steps:

[0207] S910, using the infrared image to generate a mask M ir , mask M ir It is expressed as:

[0208]

[0209] Among them, I ir represents an infrared image, represents the mean value of the infrared image;

[0210] In order to enhance the similarity between the fusion result and the infrared image and the visible light image, we design a mask-based correlation loss. Since the infrared image has good target saliency, we use the source infrared image to generate the mask M. ir .

[0211] S920, calculating the Pearson correlation coefficient between the infrared image and the fused image or between the visible light image and the fused image, the formula is:

[0212]

[0213] Among them, (i, j) represents the vertical and horizontal coordinates of the pixel, H, W represents the height and width of the image, I f (i,j) represents the pixel value of the fused image at (i,j), represents the pixel mean of the fused image, I x (i, j) represents the pixel value of the infrared image or visible light image at (i, j), Represents the mean value of source image pixels;

[0214] The Pearson correlation coefficient measures the strength of the linear correlation between two variables, and its value range is between -1 and 1. When the correlation coefficient is 1, it means that there is a completely positive correlation between the two variables. When the value is -1, it means that there is a completely negative correlation. A value of 0 indicates that there is no linear correlation.

[0215] S930: Calculate the similarity loss function, which specifically includes the following steps:

[0216] S931, set the mask M ir Respectively with the source infrared image I ir and fused image I f Multiply and obtain the salient area of ​​the infrared image and the salient area of ​​the fused image;

[0217] S932, calculating the significant correlation coefficient between the significant region of the infrared image and the significant region of the fused image;

[0218] S933, multiply the texture information area of ​​the visible light image by the visible light image and the fused image respectively, and the texture information area of ​​the visible light image is (1-M ir );

[0219] Because the texture information stored in visible light images is richer than that in infrared images, we need to calculate the correlation coefficient between the texture information of the fused image and the visible light image. We believe that there should be a lot of texture information in the non-salient area of ​​the infrared image, so we use (1-M ir ) represents the texture information area.

[0220] S934, calculating the texture correlation coefficient between the texture information area of ​​the visible light image and the fused image;

[0221] S935, summing the significant correlation coefficient and the texture correlation coefficient, and obtaining a total similarity loss; the formula for the total similarity loss is:

[0222] L cc =1-CC(M ir i f ,M ir i ir )+1-CC((1-M ir )e f ,(1-M ir )e vi )

[0223] Among them, L cc represents the similarity loss, M ir Indicates mask, I f represents the fused image, e represents the multiplication of corresponding elements, I ir Indicates infrared image, I vi represents a visible light image;

[0224] S940, calculating strength loss, which includes the following steps:

[0225] S941, calculating the first Manhattan distance of the difference between the salient area of ​​the infrared image and the salient area of ​​the fused image;

[0226] S942, calculating the second Manhattan distance between the difference between the texture information area of ​​the visible light image and the texture of the fused image;

[0227] S943. Add the first Manhattan distance and the second Manhattan distance to obtain the intensity loss. The formula of the intensity loss is:

[0228] L pixel =||(M ir i f )-(M ir iir )||1+||((1-M ir )e f )-((1-M ir )e vi )||1

[0229] Among them, L pixel Indicates strength loss, M ir Indicates mask, I f represents the fused image, e represents the multiplication of corresponding elements, I ir Indicates infrared image, I vi represents visible light image; ||||1 represents Manhattan distance;

[0230] S950, calculating texture loss, the formula is:

[0231]

[0232] in represents the absolute value of the fused image gradient, Represents the absolute value of the visible light image gradient, Indicates the absolute value of the infrared image gradient, and max() indicates the maximum value between the two;

[0233] We expect the fused image to maintain the best intensity distribution while retaining rich texture details, so we introduce texture loss so that the fused image contains more rich texture information.

[0234] S960, calculate the total loss function, the formula is:

[0235] L total =aL cc +μL piexl +δL texture

[0236] Among them, α, μ, and δ represent the setting parameters, which are set to 30, 7, 50, and L respectively. cc represents the similarity loss, L pixel Indicates strength loss, L texture Indicates texture loss.

[0237] In order to verify the rationality of the proposed algorithm, the MSRS public dataset was used for testing. The dataset was divided into 7:3 as mentioned above, the learning rate was set to 0.001, the batch size (batch_size) was set to 8, and 100 rounds (epochs) of training were performed. The parameter results learned from the training were applied to the test set. Compared with several other methods nowadays, they represent different high-level visual tasks: unified unsupervised image fusion network (U2Fusion), semantically-aware real-time infrared and visible light image fusion (SeFusion), and a progressive infrared and visible light image fusion network based on illumination perception (PIAFusion). A unified evaluation index was selected as an objective evaluation: average gradient (AG), entropy (EN), image fusion quality (Q abf ), spatial frequency (SF), visual information fidelity (VIF), the objective evaluation indicators of the comparative experiment can be seen in Table 1. The results show that the fusion method we proposed is more excellent.

[0238] Table 1 Objective indicators

[0239] method AG EN <![CDATA[Q abf ]]> SF VIF SeFusion 4.1361 7.205 0.6798 0.0482 1.0389 U2Fusion 2.6978 6.4698 0.451 0.030 0.7366 PIAFusion 4.2273 7.228 0.6681 0.0492 1.0558 Ours 4.9032 7.3769 0.6775 0.0577 1.0709

[0240] FIG6 is a comparison diagram of image fusion results, where (a) is an infrared image; (b) is a visible light image; (c) is the fusion result of the method proposed in this application; (d) is the SeFusion fusion result; (e) is the U2Fusion fusion result; and (f) is the PIAFusion fusion result.

[0241] The results of zooming in on some areas of the image show that the fusion result of our method has better details and edges, and does not produce artifacts while retaining the significant information of the source image, and has a better fusion effect. In addition, this patent has a brightening effect on the fusion results of images in dark environments. In the fusion result image, the fusion result proposed by this patent can clearly observe the texture structure of the trees.

[0242] The above is a detailed introduction to the infrared image and visible light image fusion method provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for fusing infrared images and visible light images, characterized in that: include: The infrared image and the visible light image are normalized and respectively input into a single-modal feature extraction branch network, wherein the single-modal feature extraction branch network includes a feature extraction module and a hybrid attention mechanism module; The feature extraction module of the residual connection is used to extract the basic features of the infrared image and the basic features of the visible light image, and the infrared features and the visible light features are obtained; The operation steps of the hybrid attention mechanism module are as follows: the infrared features and the visible light features are input into the hybrid attention mechanism module, and the infrared global features and the infrared local features, as well as the visible light global features and the visible light local features are obtained respectively; Perform matrix addition and matrix multiplication operations on the infrared global features and the infrared local features, concatenate and reduce the dimensions of the operation results, and then use a jump connection method to add the infrared features to obtain a single-mode infrared feature; Perform matrix addition and matrix multiplication operations on the visible light global feature and the visible light local feature, concatenate and reduce the dimension of the operation results, and then use a jump connection to add the visible light feature to obtain a single-mode visible light feature; The unimodal infrared features and unimodal visible light features obtained by the first-level convolution are added and input into the Transformer module for information integration to obtain multimodal infrared features and multimodal visible light features. Mixed modal feature extraction step: Use the SIM module to generate corresponding correlation weights for the multimodal infrared features and the multimodal visible light features according to the single-modal infrared features and the single-modal visible light features extracted by the corresponding mixed attention mechanism module, and inject them into the single-modal infrared features and the single-modal visible light features respectively; Determine whether the level of the unimodal feature extraction branch network is less than the preset level. If so, loop the hybrid attention mechanism module operation step to the hybrid modal feature extraction step. If not, add the unimodal infrared features and unimodal visible light features obtained by the last level of convolution to the multimodal infrared features and multimodal visible light features obtained in the last layer of the Transformer module and then perform splicing processing; The concatenated features are input into the decoder network to generate a fused image. Wherein, the mixed modal feature extraction step comprises the following steps: Perform 3x3 convolution operation on multimodal infrared features and multimodal visible light features; The multi-modal infrared features after dimensionality reduction are channel-joined with the single-modal infrared features, and a 3x3 convolution operation is performed. At the same time, the multi-modal visible light features after dimensionality reduction are channel-joined with the single-modal visible light features, and a 3x3 convolution operation is performed. The channel attention module is used to learn the convolutional infrared features and visible light features into the scaling matrix λ. The learning of the scaling matrix λ is: λ=SE(Conv 3×3 (C(F i out ,(Conv 3×3 (F i trans ))))) λ represents the scaling matrix, SE represents the channel attention module, and F i trans represents the output of the i-th level Transformer module, F i out represents the output of the i-th level of the unimodal feature extraction branch network, Conv represents the convolution operation, and C represents the channel concatenation operation; Use 3x3 convolution to learn the bias matrix β of the multi-modal infrared features after dimensionality reduction and the multi-modal visible light features after dimensionality reduction. The learning of the bias matrix β is: β=Conv 3×3 (Conv 3×3 (F i trans )) β represents the bias matrix, F i trans represents the output of the i-th level Transformer module, and Conv represents the convolution operation; The layer-normalized single-mode infrared feature is multiplied by the scaling matrix λ, and then added, and then the bias matrix β is added, so as to be injected into the next level of the single-mode infrared feature; at the same time, the layer-normalized single-mode visible light feature is multiplied by the scaling matrix λ, and then added, and then the bias matrix β is added, so as to be injected into the next level of the single-mode visible light feature; The formula is: in, represents the input of the i+1th level of the unimodal feature extraction branch network, F i out represents the output of the i-th level of the unimodal feature extraction branch network, λ represents the scaling matrix, and β represents the bias matrix.

2. The infrared image and visible light image fusion method according to claim 1, characterized in that: In the step of extracting the basic features of the infrared image and the basic features of the visible light image using the feature extraction module of the residual connection, and obtaining the infrared features and the visible light features, Use 3x3 convolution to convolve the infrared image and visible light image respectively; The convolved infrared image and visible light image are convolved using 1x1 convolution to obtain infrared features and visible light features.

3. The method for fusing infrared images and visible light images according to claim 1, characterized in that: In the hybrid attention mechanism module operation step, the step of obtaining infrared global features includes: Input the infrared features into the hybrid attention mechanism module and use them as the query vector Q; Use 3x3 convolution to convolve the infrared features and obtain the key vector K; Use 1x1 convolution to convolve the infrared features and obtain the value vector V; Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions; Transpose the attention matrix to obtain the transposed matrix; Perform matrix multiplication operation on the attention matrix and the transposed matrix, and then perform softmax function operation to obtain the weight matrix of global channel attention; Perform matrix transposition and transpose operations on the value vector V, and perform matrix multiplication operation on the resulting matrix and the weight matrix; The matrix obtained by multiplication is successively transposed and transposed to obtain the infrared global feature.

4. The infrared image and visible light image fusion method according to claim 1, characterized in that: In the hybrid attention mechanism module operation step, the step of obtaining infrared local features includes: Input the infrared features into the hybrid attention mechanism module and use them as the query vector Q; Use 3x3 convolution to convolve the infrared features and obtain the key vector K; Use 1x1 convolution to convolve the infrared features and obtain the value vector V; Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions; The attention matrix is ​​processed by global average pooling and global maximum pooling respectively, and the obtained feature vector is processed by channel splicing; The concatenated matrix is ​​reduced in dimension using 1x1 convolution, and the resulting matrix is ​​element-wise multiplied with the value vector V to obtain the infrared local features of the local channel attention.

5. The infrared image and visible light image fusion method according to claim 1, characterized in that: In the hybrid attention mechanism module operation step, the step of obtaining the visible light global feature includes: The visible light features are input into the hybrid attention mechanism module and used as the query vector Q; Use 3x3 convolution to convolve the visible light features and obtain the key vector K; Use 1x1 convolution to convolve the visible light features and obtain the value vector V; Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions; Transpose the attention matrix to obtain the transposed matrix; Perform matrix multiplication operation on the attention matrix and the transposed matrix, and then perform softmax function operation to obtain the weight matrix of global channel attention; Perform matrix transposition and transpose operations on the value vector V, and perform matrix multiplication operation on the resulting matrix and the weight matrix; The matrix obtained by multiplication is successively transposed and transposed to obtain the global features of visible light.

6. The infrared image and visible light image fusion method according to claim 1, characterized in that: In the hybrid attention mechanism module operation step, the step of obtaining the local features of visible light includes: The visible light features are input into the hybrid attention mechanism module and used as the query vector Q; Use 3x3 convolution to convolve the visible light features and obtain the key vector K; Use 1x1 convolution to convolve the visible light features and obtain the value vector V; Concatenate the query vector Q and the key vector K channel by channel, and obtain the attention matrix after two consecutive 1x1 convolutions; The attention matrix is ​​processed by global average pooling and global maximum pooling respectively, and the obtained feature vector is processed by channel splicing; The concatenated matrix is ​​reduced in dimension using a 1x1 convolution, and the resulting matrix is ​​element-wise multiplied with the value vector V to obtain the visible light local features of the local channel attention.

7. The method for fusing infrared images and visible light images according to claim 3 or 4, characterized in that: The attention matrix is ​​enhanced using a feature enhancement module based on the LoG operator, which includes the following steps: The infrared image is low-pass filtered using a two-dimensional LoG function with a Gaussian standard deviation of σ; the LoG function is: Where (x, y) represents the horizontal and vertical coordinates of the pixel, and σ represents the Gaussian standard deviation; Subtract the LoG operator convolution infrared attention matrix from the doubled attention matrix and obtain the enhanced infrared attention matrix, represents the enhanced infrared attention matrix, F ir (x,y) represents the infrared attention matrix.

8. The method for fusing infrared images and visible light images according to claim 5 or 6, characterized in that: The attention matrix is ​​enhanced using a feature enhancement module based on the LoG operator, which includes the following steps: The visible light image is low-pass filtered using a two-dimensional LoG function with a Gaussian standard deviation of σ; the LoG function is: Where (x, y) represents the horizontal and vertical coordinates of the pixel, and σ represents the Gaussian standard deviation; The attention matrix is ​​added to the LoG operator convolution visible light attention matrix, and the enhanced visible light attention matrix is ​​obtained. in, represents the enhanced visible light attention matrix, F vis (x,y) represents the visible light attention matrix.

9. The infrared image and visible light image fusion method according to claim 1, characterized in that: After the step of inputting the spliced ​​features into the decoder network and generating a fused image, the method further includes the following steps: Generate a mask M using the infrared image ir , mask M ir It is expressed as: Among them, I ir represents an infrared image, represents the mean value of the infrared image; Calculate the Pearson correlation coefficient between the infrared image and the fused image or between the visible light image and the fused image. The formula is: Among them, (i, j) represents the vertical and horizontal coordinates of the pixel, H, W represents the height and width of the image, I f (i,j) represents the pixel value of the fused image at (i,j), represents the pixel mean of the fused image, I x (i, j) represents the pixel value of the infrared image or visible light image at (i, j), Represents the mean value of source image pixels; Calculate the similarity loss function, which specifically includes the following steps: The mask M ir Respectively with the source infrared image I ir and fused image I f Multiply and obtain the salient area of ​​the infrared image and the salient area of ​​the fused image; Calculate the significant correlation coefficient between the significant area of ​​the infrared image and the significant area of ​​the fused image; The texture information area of ​​the visible light image is multiplied by the visible light image and the fused image respectively. The texture information area of ​​the visible light image is (1-M ir ); Calculate the texture correlation coefficient between the texture information area of ​​the visible light image and the fused image; The significant correlation coefficient is summed with the texture correlation coefficient, and the total similarity loss is obtained; the formula of the total similarity loss is: L cc =1-CC(M ir e I f ,M ir e I ir )+1-CC((1-M ir )e I f ,(1-M ir )e I vi ) Among them, L cc represents the similarity loss, M ir Indicates mask, I f represents the fused image, e represents the multiplication of corresponding elements, I ir Indicates infrared image, I vi represents a visible light image; The strength loss is calculated, which includes the following steps: Calculate the first Manhattan distance of the difference between the salient area of ​​the infrared image and the salient area of ​​the fused image; Calculate the second Manhattan distance of the difference between the texture information area of ​​the visible light image and the texture of the fused image; The first Manhattan distance is added to the second Manhattan distance to obtain the intensity loss; the formula for the intensity loss is: L pixel =||(M ir e I f )-(M ir e I ir )||1+||((1-M ir )e I f )-((1-M ir )e I vi )||1 Among them, L pixel Indicates strength loss, M ir Indicates mask, I f represents the fused image, e represents the multiplication of corresponding elements, I ir Indicates infrared image, I vi represents visible light image; || ||1 represents Manhattan distance; Calculate the texture loss, the formula is: in represents the absolute value of the fused image gradient, Represents the absolute value of the visible light image gradient, Indicates the absolute value of the infrared image gradient, and max() indicates the maximum value between the two; Calculate the total loss function, the formula is: L total =aL cc +μL piexl +δL texture Among them, α, μ, δ represent the setting parameters, L cc represents the similarity loss, L pixel Indicates strength loss, L texture Indicates texture loss.

Images