An infrared and visible image fusion method focusing on inherent characteristics of multi-modal images

Abstract

The present application belongs to the field of image processing, and particularly relates to an infrared and visible light image fusion method focusing on inherent characteristics of multi-modal images; the method comprises obtaining an infrared image and a visible light image of a target image; the infrared image and the visible light image of the target image are input into a feature extractor, and a feature map of the infrared image and a feature map of the visible light image are extracted; the feature map of the infrared image and the feature map of the visible light image are processed using a channel attention module improved based on infrared and visible light image fusion characteristics, and an initial weight of channel attention is obtained in a global gradient mode. The features of the infrared image and the features of the visible light image are sent to a feature discriminator to strengthen the expression ability of the extracted features to fixed modal information; the final features of the infrared image and the final features of the visible light image are input into a reconstruction network to generate an image fusion image of the target image. The present application can better focus on the detail information of infrared and visible light images; the generated fusion image is rich in details and conforms to human visual perception.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to an infrared and visible light image fusion method focusing on inherent characteristics of multi-modal images. BACKGROUND

[0002] At present, image fusion technology has been widely applied in many fields. For example, in clinical diagnosis, important information in computed tomography (CT) and magnetic resonance (MR) images is fused to reduce the difficulty of accurate diagnosis; in digital photography, different exposure images or different focus images are fused to provide clearer and more realistic digital images; in remote sensing image processing, panchromatic (PAN) images and multispectral (MS) images are fused to obtain a fused image with higher spatial resolution and rich spectral content; in target detection, monitoring or tracking, infrared and visible light images are fused to realize all-weather operation of the system, and infrared target intensity and visible details can be well preserved in the fused image, which is easy to observe.

[0003] Many scholars have been committed to the research of infrared and visible light image fusion technology, which is one of the research hotspots in the field of image fusion, and has made great progress. Visible light and infrared images obtained by optical and infrared sensors describe scene information from two different angles, and there are obvious differences between them. Optical images capture images through reflection mechanism, and have high contrast, image resolution, and quite a lot of details and light-dark contrast due to good lighting. Therefore, visible light images are more suitable for human visual perception. However, their anti-interference ability is poor. When the lighting conditions are not ideal, such as "smoke", "fog" and "obstruction", the imaging effect is poor. The loss of texture, details and occluded information in the scene has a great impact on subsequent detection and recognition. Infrared images are imaged by thermal radiation principle, and the higher the target temperature, the more obvious the description of scene information by infrared images. Not only does it reduce the dependence on lighting and surrounding environment, but also improves its anti-interference performance, and can work uninterruptedly for 24 hours, and can collect information anytime and anywhere. However, problems such as low image contrast, poor clarity and poor visual effect make it difficult to use infrared images in target recognition and analysis.

[0004] According to the analysis of the characteristics of visible light images and infrared light images, the use of any single image sensor cannot perfectly describe the information in the scene. At present, two kinds of images with complementary characteristics can be used to integrate their respective characteristics, complement the advantages of both, and obtain a fused image with prominent target, good visual effect and strong robustness. The fusion result of the two images has the characteristics of the source images, and the fusion result can more reliably analyze the information content of the image. The fusion of infrared images and visible light images has wide application and very important practical significance.

[0005] Infrared and visible light images are typically two images from different modalities. Infrared images are captured using infrared sensors, and visible light images are captured using visible light sensors. Visible light images are usually of good quality, rich in detail texture, and consistent with human visual perception, but are susceptible to adverse weather. In contrast, infrared images usually capture the thermal radiation information of objects, can highlight the contours of objects, and are more robust. The purpose of multi-modal image fusion is to effectively fuse the effective information of both, so that the generated fusion image can effectively highlight the useful and key information of the two modal images.

[0006] In traditional GAN-based fusion methods, discriminators are often used to make the fusion image more similar to the infrared and visible light images. However, GAN methods are mostly used in production tasks, and when introduced into image fusion tasks, they will inevitably bring some false information that is generated and not from the source images. Moreover, in infrared and visible light fusion tasks, the generated fusion image is expected to combine the multi-modal useful information of the corresponding source images. It is not enough to simply judge the fusion image as an infrared image or a visible light image. First, since GAN is a generative model, some non-existent and unrealistic information will be produced in the fusion image, but in image fusion tasks, the information of the generated fusion image must strictly come from the source images. Furthermore, the generated fusion image will be fed into the discriminator, which is expected to judge it as a visible light image (or an infrared image), but in reality, the generated fusion image is not simply "like" a general visible light image, but is expected to have the characteristics of the specific visible light image of the source image. In addition, the existing methods mostly do not pay attention to the importance of multi-modal information in the fusion of extracted source image features, resulting in the final generated fusion image not being able to well represent the key and important characteristics of the source infrared and visible light images, and there being too much redundant and ineffective information in the fusion image. SUMMARY

[0007] Based on the problems existing in the prior art, the present application proposes an infrared and visible light image fusion method focusing on the inherent characteristics of multi-modal images, which considers the importance of multi-modal information, so that the final generated fusion image can well represent the key and important characteristics of the source infrared and visible light images, and there is more effective information in the fusion image.

[0008] The present application achieves the above-mentioned purposes through the following solutions:

[0009] An infrared and visible light image fusion method focusing on the inherent characteristics of multi-modal images, the method comprising:

[0010] obtaining an infrared image and a visible light image of a target image;

[0011] The infrared image and the visible light image of the target image are input into a feature extractor, and a feature map of the infrared image and a feature map of the visible light image are extracted;

[0012] The feature map of the infrared image and the feature map of the visible light image are processed by using a channel attention module, and global gradient is adopted to obtain the final feature of the infrared image and the final feature of the visible light image;

[0013] The final feature of the infrared image and the final feature of the visible light image are input into a reconstruction network to generate an image fusion image of the target image.

[0014] The beneficial effects of the present application are as follows:

[0015] 1、The mutual attention of the feature map of the infrared image and the feature map of the visible light image is calculated, so that the extracted feature information of the infrared image and the feature information of the visible light image can be more comprehensive, and the mutual influence relationship between the feature information of the infrared image and the feature information of the visible light image can be fully mined.

[0016] 2、The present application discards the way of obtaining the initial channel weight value by average pooling or maximum pooling in the traditional technology, and adopts the global gradient as the initial channel weight value. The channel attention module can better focus on the detail information of the infrared and visible light images; and the generated fusion image is rich in details and conforms to human visual perception.

[0017] 3、The present application changes the way of directly judging the source image category in the traditional technology to the feature of the feature extractor, which can promote the feature extractor to better extract the multi-modal information capable of expressing the corresponding source image. Only when the feature extracted by the feature extractor contains more unique information of the source image, the discriminator will judge it as the corresponding feature with a greater probability. The generated fusion image better combines the key useful information of both. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flow chart of the image fusion method of the embodiment of the present application;

[0019] Figure 2 is a feature extractor of the embodiment of the present application;

[0020] Figure 3 is a channel attention module of the embodiment of the present application;

[0021] Figure 4 is a reconstruction network of the embodiment of the present application;

[0022] Figure 5 is a flow chart of the image fusion method of the preferred embodiment of the present application;

[0023] Figure 6 is a feature discriminator of an embodiment of the present application;

[0024] Figure 7 is a subjective comparison chart of an embodiment of the present application; Figure 7 (a) and Figure 7 (b) are visible light images and infrared images employed in an embodiment of the present application; Figure 7 (d)- Figure 7 (i) Sequential image fusion results of visible light images and infrared images under GANMcC, FusionGAN, PMGI, DenseFuse, MSVD and the method of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] Figure 1 is a flow chart of an image fusion method of an embodiment of the present application, as Figure 1 The method comprises:

[0027] 101, obtaining an infrared image and a visible light image of a target image;

[0028] In an embodiment of the present application, the infrared image and the visible light image of the target image comprise fixing infrared images and visible light images of the same background and the same object to the same size, performing pairing processing on the infrared images and the visible light images, that is, one infrared image corresponds to one visible light image, and the infrared images and the visible light images can be obtained from a public platform or can be obtained by using a camera, an infrared sensor or other devices, which are not specifically limited by the present application.

[0029] In a preferred embodiment of the present application, the infrared images and the visible light images of the same background and the same object can also be cropped to the same size, for example, 256x256 size, and the images can also be subjected to common image preprocessing such as denoising.

[0030] In a preferred embodiment of the present invention, the present invention can smooth the acquired infrared and visible light images using Gaussian filtering to remove image noise; then, after spatial transformation of the smoothed visible light image, Gaussian filtering is performed again to remove redundant noise from the image; then, opening operations combined with dilation and erosion operations in computer morphology are used to eliminate scattered noise points enhanced by morphological operations; finally, bandpass filtering is performed using the Laplacian operator. The final result can effectively eliminate noise pixels in the visible light image and irrelevant noise in the infrared image introduced by the shooting equipment, shooting environment, etc.

[0031] To avoid the loss of detail in visible light images when using the Laplacian operator for bandpass filtering, this embodiment also utilizes a region summation table from the initial node (0,0) to the current node (x,y) to calculate the value of any large rectangular region in the image within a fixed time period. This region summation table can process large-area images. Furthermore, this embodiment uses an edge operator and nonlinear bilateral filtering to process the image, thereby eliminating noise and preserving the edge information of the visible light image as much as possible.

[0032] 102. Input the infrared and visible light images of the target image into the feature extractor to extract the feature maps of the infrared image and the visible light image;

[0033] In this embodiment of the invention, the main function of the feature extractor is to extract feature maps that represent the inherent characteristics of infrared and visible light images. The specific network structure of the feature extractor is as follows: Figure 2 As shown, the entire feature extractor consists of five consecutive convolutional units. Each convolutional unit comprises a convolutional layer, a batch normalization layer, and an activation layer. Because downsampling inevitably leads to information loss, the stride of all convolutional layers in this embodiment can be set to 1. The activation function used is LeakyReLU. The kernel size of the first four layers is 3x3, and the last layer uses a 1x1 kernel to change the number of layers in the feature map.

[0034] In a preferred embodiment of the present invention, considering that the traditional method extracts feature maps only through convolutional coding, this method does not calculate the mutual influence weights between the feature maps of the infrared image and the visible light image, resulting in incomplete feature information and insufficient exploration of the relationships between feature information; based on this, this embodiment performs convolution operations on the feature maps of infrared image i and visible light image j to obtain D feature maps, and performs linear operations on the D feature maps of infrared image i and visible light image j to obtain D-dimensional vectors, i.e., vector a. i D a j D, a i D , a j D Copy, in turn, the vector b i D and b j D ; transpose b i D and b j D , a i D , a j D corresponding multiplication, the mutual attention weight matrix is obtained; the feature map of the infrared image i and the visible light image j is obtained by re-encoding according to the mutual attention weight matrix.

[0035] The calculation formula of the mutual attention weight matrix is represented as:

[0036]

[0037] Wherein, W ij indicates the mutual attention weight between the feature map of the infrared image i and the feature map of the visible light image j; indicates the d-dimensional vector of the feature map of the infrared image i; indicates the d-dimensional vector of the visible light image j; T indicates the matrix transpose.

[0038] The formula of the re-encoding according to the mutual attention weight matrix is represented as:

[0039] F′ i = W ij F i +(1-W ij )F j

[0040] F′ j = W ij F j+ (1-W ij )F i

[0041] Wherein, F′ i indicates the re-encoded feature map of the infrared image i, F′ j indicates the re-encoded feature map of the visible light image j, F i indicates the feature map of the infrared image i; F j indicates the feature map of the visible light image j.

[0042] The preferred embodiment calculates mutual attention for the feature maps of the infrared image and the feature maps of the visible light image, so that the extracted feature information of the infrared image and the feature information of the visible light image can be more comprehensive, and the mutual influence relationship between the feature information of the infrared image and the feature information of the visible light image can be fully mined.

[0043] 103. The feature maps of the infrared image and the feature maps of the visible light image are processed using a channel attention module, and global gradient is used to obtain the final features of the infrared image and the final features of the visible light image.

[0044] In the embodiment, the source images obtain corresponding feature maps after passing through the feature extractor. Traditional feature fusion algorithms mostly simply splice the features of the infrared and visible light images together. However, in fact, the importance of features of different layers in the feature maps extracted by the application is different, and the application hopes to pay more attention to important features of the source images in the fusion process and weaken the expression of redundant and useless information. This also conforms to an important rule of image fusion, that is, only useful information of two images is fused.

[0045] In fact, the channel attention mechanism provides a good idea for the feature fusion of the application. Although there are methods that apply the channel attention mechanism to the infrared and visible light image fusion task, the existing methods only simply apply the attention mechanism to the fusion module without fully considering the particularity and inherent characteristics of the infrared and visible light image fusion task. In the infrared and visible light image fusion task, the expression of detail information is particularly important.

[0046] The application generally considers that the expression detail information of the fusion image should come from the visible light image, but in fact, the infrared image often has clear details and is often captured by the visible light image. Fully extracting and fusing the detail information of the two source images can make the finally generated fusion image clear and bright.

[0047] The existing channel attention mechanism for the image fusion task mostly follows the traditional way to obtain the initial channel weight value through average pooling and maximum pooling. The average pooling can reduce the problem of increased variance of estimated values caused by the limitation of neighborhood size, and more retain the overall characteristics of the data, and the maximum pooling can reduce the problem of estimated mean deviation caused by the error of convolution layer parameters. However, the two ways cannot be well applied to the infrared and visible light image fusion task. The average pooling has a smoothing effect on the feature map, and the maximum pooling is easily disturbed by the noise commonly seen in the infrared image. For the infrared and visible light image fusion task, we need to make the final fusion image contain more useful information of the two source images.

[0048] To better fuse useful information from the source images through details and textures, this invention employs global gradients as the initial channel weights. Different weights are assigned to different feature maps; if a feature map contains more texture details, its initial weight value obtained from the global gradient is larger, and consequently, more of the information contained in that feature map will be reflected in the final fused image. The structure of the entire channel attention module is as follows: Figure 3 As shown, the initial weight values ​​are obtained by calculating the global gradient of the feature maps of infrared and visible light images. Then, the initial weights are fine-tuned through a fully connected layer. Finally, the final weight values ​​are obtained by mutual attention and applied to the corresponding feature maps. The improved channel attention module can assign different importance to different feature maps of the source image, highlighting the expression of effective information and reducing redundant and useless information.

[0049] In this embodiment of the invention, the calculation method of the global gradient of the feature map includes calculating the local gradient of each feature point within the range of its neighboring nodes, averaging the local gradients of all feature points in the feature map, and obtaining the global gradient of the feature map. The calculation formula is expressed as follows:

[0050]

[0051]

[0052] in, Let N represent the global gradient of feature map I, and let N represent the number of feature points in feature map I. f(x,y) represents the local gradient of the feature point (x,y) in feature map I; f(x,y) represents the pixel value of the feature point.

[0053] After passing through a feature extractor, feature maps of infrared and visible light images are obtained. The global gradient of each feature map is calculated to obtain initial channel attention weights. These weights are then passed through a fully connected layer to obtain final channel attention weights. These final channel attention weights are multiplied by the corresponding feature maps to obtain the final features. Finally, the features of the infrared and visible light images obtained through the channel attention mechanism are concatenated to obtain the final fused features. This invention improves the method of calculating initial feature weights by replacing max pooling with global gradient calculation. This allows the network to more effectively express the detailed information of the two source images during the fusion process. The generated fused image is rich in detail and conforms to human visual perception.

[0054] 104. Input the final features of the infrared image and the final features of the visible light image into the reconstruction network to generate an image fusion image of the target image.

[0055] In the embodiment of the present application, the reconstruction network reconstructs the features of the fused source image into the final fused image. The specific structure of the reconstruction network is as follows Figure 4 As shown in the figure, our reconstruction network is composed of three consecutive convolution units. The first two convolution units are both composed of a 3*3 convolution layer, a batch normalization layer, and an activation layer using a LeakyRelu activation function. In order to facilitate the generation of the final fused image, the last convolution unit uses a 1*1 convolution kernel size and uses tanh as the activation function.

[0056] In the preferred embodiment of the present application, in order to further improve the accuracy of the fusion method of the present application, the present patent can be divided into two stages: the first stage is the training stage, which trains the network in a generative adversarial manner, and the second stage is the test stage, in which the final fused image is generated.

[0057] The training stage is implemented in the following steps:

[0058] Step one, select a number of matched infrared and visible light image pairs from the standard training set for network training.

[0059] Step two, send the infrared and visible light images selected in step one into the feature extractor of the network for feature extraction. The source image obtains the corresponding feature map after passing through the feature extractor.

[0060] Step three, send the feature map extracted in step two into the feature discriminator to obtain the predicted probability value. Calculate the loss L of the feature extractor F .

[0061] Step four, process and splice the extracted features of the infrared and visible light images using the channel attention module. And send it into the reconstruction network to generate a fused image.

[0062] Step five, calculate the content loss L content of the fused image. The final loss is the feature loss plus the content loss L=L feature +L content . Update the parameters of the feature extractor and the reconstruction network. After training, save the parameters of the network.

[0063] Step six, load the network parameters saved in step five into the test network. Input the original infrared and visible light images to obtain the final fused image.

[0064] The flow chart of the whole image processing is shown in Figure 5 , wherein the dashed part is the feature discrimination module in the training stage, and there is no such module in the test stage. Next, the components of the system will be analyzed.

[0065] In order to make the generated fusion image have more characteristics of the source image, the existing method learns from the idea of GAN network generation confrontation, and sends the fusion image generated by the generator into the discriminator, so that the discriminator can distinguish the fusion image as the original infrared or visible light image. Meanwhile, the source image and the generated fusion image are used to train the discriminator, so that the discriminator can distinguish the generated fusion image from the real source image.

[0066] However, in the image fusion task, the generated fusion image of the present application is expected to have matching characteristics of the infrared and visible light image pair, instead of being simply distinguished by the discriminator as a general infrared or visible light image. Meanwhile, due to the generation characteristics of the GAN network, the fusion image generated in this way often has a lot of useless information generated in the source image, which affects the quality of the generated fusion image.

[0067] The present application learns from the idea of using the discriminator to distinguish the fusion image in the previous method, but changes the position of the discriminator in view of the drawbacks of simply distinguishing the fusion image as the source image by the discriminator. The fusion image is distinguished as the generated feature. The original infrared and visible light images are obtained after the feature extractor, and the corresponding infrared and visible light image features are obtained. The extracted features are sent to the feature discriminator. The features of the infrared image need to be distinguished by the discriminator as infrared features, and the corresponding features of the visible light image need to be distinguished as visible light image features. In this way, the feature extraction network of the present application can better extract the multi-modal information of the corresponding source image. Only the features extracted by the feature extractor contain more unique information of the source image, the discriminator will distinguish it as the corresponding feature with a higher probability.

[0068] The method proposed in the present application avoids the shortcomings of the above-mentioned generation confrontation method, and better extracts the multi-modal information of the two images. The generated fusion image better combines the key useful information of the two images. The structure of the feature discriminator is as follows Figure 6 The extracted feature map is changed into a one-dimensional vector after two times of downsampling, and finally the final feature discrimination probability is obtained after two times of full connection.

[0069] In the embodiment of the present application, the loss function of the network is composed of two parts. One part is the content loss L content content loss L featureThe loss of the feature discriminator can better promote the feature extractor to better extract the source image features. The thermal radiation information of the infrared image is mainly embodied by pixel intensity, and the information of the visible light image is mainly embodied by feature texture. Therefore, the application uses the pixel intensity difference between the fusion image and the infrared image and the texture difference between the fusion image and the visible light image as the content loss. The corresponding content loss L content may be represented as:

[0070]

[0071] Where H and W represent the height and width of the input image respectively, and ||·|| F represents the F norm of the matrix, represents the gradient operation, and I f represents the feature map of the infrared image, v represents the feature map of the visible light image, represents the global gradient of the feature map I f , and represents the global gradient of the feature map , and δ represents an adjustable parameter. The first item of the content loss L content wants the fusion image to maintain the thermal radiation information of the original infrared image. The second item of the content loss wants the fusion image to contain more detail information of the visible light image.

[0072] Another part of the loss of the generation network is the feature loss L feature . The application wants the extracted infrared image feature F i to be discriminated as 1, and the extracted visible light image feature F v to be discriminated as 0. The application adopts the mean square error as the loss function. For the network feature loss L feature , the batch normalization quantity n can be represented as follows:

[0073]

[0074] In order to verify the effectiveness of the method of the application, the application makes qualitative and quantitative comparative experiments. First, the application introduces the data and evaluation indexes used by the application. Then, the application shows the subjective and objective comparison results of the method of the application and other six methods.

[0075] (1) Subjective effect comparison.

[0076] The infrared and visible light images used to test the performance of the method of the application come from the TNO data set. The application first qualitatively compares the performance of the method of the application, and directly observes the differences between different fusion algorithms. The application selects two pairs of typical images for qualitative evaluation. The fusion results are shown in Figure 7 . As Figure 7The shown is a typical example of infrared image affected by noise, so the generated fusion image tends to have the noise of the infrared image. Figure 7 (a) and Figure 7 (b) are visible light image and infrared image used in the embodiments of the present application; Figure 7 (d) ~ Figure 7 (i) Image fusion results of visible light image and infrared image in turn under GANMcC, FusionGAN, PMGI, DenseFuse, MSVD and the method of the present application; the present application uses a frame line to show the example of shrub area, and enlarges it and places it in the lower right corner. It can be seen that although all the methods can highlight the targets of the infrared image, they are all affected by noise. The method of the present application can extract the targets highlighted by the infrared image and filter the noise. At the same time, as shown by the dotted box. The image generated by the present application can not be contaminated by the thermal radiation information in the sky area. Through these intuitive comparisons, on the one hand, the fusion image generated by the method of the present application can extract the targets highlighted by the infrared image while suppressing the noise, and on the other hand, the image of the present application retains the rich texture details in the visible light image, and the generated image is more consistent with human visual perception.

[0077] (2) Objective effect comparison

[0078] In order to better verify the method proposed by the present application, the present application performs quantitative comparison. The present application uses 20 image data sets selected from TNO to perform objective evaluation. Table 1 shows the performance of the six methods on the above-mentioned six common infrared and visible light indexes. It can be found that the method of the present application achieves the best result on all test indexes except EN and MI. The quantitative comparison of objective indexes also shows the superiority of the method proposed by the present application.

[0079] Table 1 objective effect comparison

[0080]

[0081]

[0082] Those skilled in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware by a program, and the program can be stored in a computer readable storage medium, which can include ROM, RAM, magnetic disk or optical disk, etc.

[0083] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An infrared and visible image fusion method focusing on inherent characteristics of multi-modal images, characterized by, The method comprises: obtaining an infrared image and a visible light image of a target image; The infrared image and the visible light image of the target image are input into a feature extractor to extract a feature map of the infrared image and a feature map of the visible light image; comprising: performing convolution operation on the feature map of the infrared image i and the feature map of the visible light image j respectively to obtain D feature maps, performing linear operation on the D feature maps of the infrared image i and the D feature maps of the visible light image j respectively to obtain D-dimensional vectors, that is, vectors a i D , a j D , a i D , a j D Copy a, and sequentially obtain vectors b i D and b j D ; after transposing b i D and b j D , multiply a i D , a j D correspondingly to obtain a mutual attention weight matrix; and re-encoding the mutual attention weight matrix to obtain the feature map of the infrared image i and the visible light image j. processing feature maps of the infrared image and the visible light image using a channel attention module, and obtaining final features of the infrared image and the visible light image in a global gradient manner; including: calculating global gradients of the feature maps of each infrared image and each visible light image respectively, obtaining initial channel attention weights according to the calculated global gradients, obtaining final channel attention weights after a fully connected layer, and multiplying the obtained channel attention weights to the corresponding feature maps as the final features of the infrared image and the visible light image; inputting the final features of the infrared image and the visible light image into a reconstruction network to generate an image fusion image of the target image; including: inputting the feature maps of the infrared image and the visible light image into a feature discriminator respectively to obtain feature discrimination probabilities; screening the feature maps of the infrared image and the feature maps of the visible light image according to the feature discrimination probabilities, and eliminating irrelevant feature maps of the infrared image and irrelevant feature maps of the visible light image.

2. The method according to claim 1, wherein the method is characterized by, The calculation formula of the mutual attention weight matrix is represented as: ； wherein W ij denotes the cross-attention weight between the feature map of the infrared image i and the feature map of the visible light image j; denotes the d-th dimensional vector of the feature map of the infrared image i; denotes the d-th dimensional vector of the visible light image j; T denotes the matrix transpose. 3.The method of claim 1, wherein, The formula of the re-encoding according to the mutual attention weight matrix is represented as: ； ； wherein, denotes the feature map of the re-encoded infrared image i, W ij denotes the cross-attention weight between the feature map of the infrared image i and the feature map of the visible light image j, denotes the feature map of the re-encoded visible light image j, denotes the feature map of the infrared image i; denotes the feature map of the visible light image j.

4. The method of claim 1, wherein the method is focused on intrinsic characteristics of multi-modal images. The calculation method of the global gradient of the feature map includes calculating the local gradient of each feature point within the neighborhood node range, averaging the local gradients of all feature points in the feature map, and obtaining the global gradient of the feature map.

Images