An Image Fusion Method Based on the Modal Difference between Infrared and Visible Light Images

Abstract

The present invention claims protection for an image fusion method based on the modal differences between infrared and visible light images, which includes the following two stages: The first stage is the training stage, in which the network's ability to separate and extract the high and low frequency features of images and reconstruct images from the combined high and low frequency features is trained by using the method of pre-training with a single natural image; the training stage includes three stages: the feature extraction stage, the feature combination stage, and the reconstruction stage; the second stage is the testing stage, in which the image processing network structure also includes three main stages: the frequency decomposition stage, the order feature fusion stage, and the reconstruction stage; the final fused image is generated in the testing stage. The present invention extracts the useful information of the two images according to the features brought by the different modalities of the two images, and while filtering redundant or even useless information, improves the utilization rate of valuable information.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and more particularly to an infrared and visible light image fusion method. Background Art

[0002] An image is a capture and freeze-frame of the real world using a specific sensor. The real world is real and rich, but due to the limitations of a single sensor, images of a specific modality can only record partial scene information of the real world. And often restricted by the imaging principle, a single sensor cannot work effectively in a specific scenario. With the rapid development of information technology, especially the rapid progress of imaging detection technology, the ways people obtain images are becoming more and more diversified. The single-modal images generated by a single sensor often cannot meet the actual needs. Multi-modal image fusion technology aims to integrate the complementary information of multiple input images (source images) of the same imaging scene into a composite image (fusion image), so that the composite image contains more information, is more in line with human visual perception, overcomes the limitations of single-modal images, and thus obtains a complete description of the imaging scene. These different modalities of images, such as infrared and visible light, CT and MRI, multi-spectral and panchromatic images, are acquired by different sensors, overcoming many limitations of images acquired by a single device, providing more information, and fully reflecting the scene information. A typical application is the fusion of infrared and visible light images. Visible images are acquired by visible light sensors, which image by capturing reflected visible light. The imaging principle of visible light sensors is similar to that of the human eye, so visible images are suitable for human visual perception, with high resolution and rich details. However, the disadvantages of visible light sensors are also obvious. They have relatively high requirements for light, and both too dark and too strong light will affect the imaging of visible light sensors. Infrared images are captured by infrared sensors, which image by capturing invisible infrared light. Infrared images describe the thermal radiation of objects, so they can effectively highlight the target and can stably resist interference. However, they usually have a low resolution and lack fine texture. It is precisely due to the inherent characteristics brought by the different modalities of infrared and visible light images that integrating and balancing the complementary and differential information in the two source images can enable the generated fusion image to effectively highlight the target, contain rich texture details, and at the same time have stronger robustness and more fully and completely express the real scene information.

[0003] Infrared images and visible light images can complement each other well. By extracting the target information in the infrared image and then obtaining the important information in the visible light image, more comprehensive visual information of important targets can be obtained. Due to the universality and complementarity of infrared and visible light images, infrared and visible light image fusion algorithms are used in many applications, such as autonomous driving, target detection, pedestrian tracking, and video surveillance.

[0004] Due to different imaging principles, infrared images capture the thermal radiation of objects, highlighting the targets but lacking texture details and usually having a low spatial resolution. Visible light images capture the reflected light of objects, with rich texture details and a high spatial resolution. Therefore, in the frequency domain, the information of infrared images is mainly concentrated in the low-frequency band. In contrast, the information of visible light images is mainly concentrated in the high frequencies. Although traditional methods and deep learning-based methods have achieved good performance in the fusion task, most methods ignore a key issue, that is, infrared images and visible light images come from different modalities, so they will have different characteristics due to the modality differences. In the fusion task, this difference should be fully considered. However, traditional infrared and visible light fusion algorithms usually use the same transformation or representation method to extract the features of the source images. Most CNN-based methods use the same feature extraction network for the two images. In many cases, the weights are even shared between the two networks. Obviously, these feature extraction methods introduce a lot of redundant and even invalid information. Therefore, valuable information will inevitably be weakened in the fused image. Based on this, we propose the method of the present invention. A fusion framework that can be processed according to the modality differences between infrared and visible light images. The image fusion task can be regarded as an image enhancement technology, which should be able to combine the advantages of infrared and visible light images in the fused image. A high-quality fused image should be able to highlight the necessary targets while showing rich texture details. Therefore, it is necessary to extract rich texture details from the visible light image and extract prominent object contours from the infrared image. The texture detail information of the image is usually concentrated in the high-frequency band, while the object contours and background information are mainly concentrated in the low-frequency band. The present invention first decomposes the source images in frequency and uses frequency doubling convolution for frequency interaction, extracts the high-frequency features of the visible light image and the low-frequency features of the infrared image respectively, and finally generates the final fused image through a reconstruction network. The present invention extracts the useful information of the two images according to the features brought by the different modalities of the two images, and while filtering redundant and even useless information, improves the utilization rate of valuable information. Summary of the Invention

[0005] The present invention aims to solve the above problems of the prior art. A method for image fusion based on the modality differences between infrared and visible light images is proposed. The technical solution of the present invention is as follows:

[0006] A method for image fusion based on the modality differences between infrared and visible light images, which includes the following two stages:

[0007] The first stage is the training stage, which uses a single natural image pre-training method to train the network's ability to separate and extract high- and low-frequency features of the image and reconstruct the image from the combination of high- and low-frequency features. The training stage includes three stages: feature extraction stage, feature combination stage, and reconstruction stage. The feature extraction stage is used to train the network's ability to separate and extract high- and low-frequency features of the image; the feature combination stage is used to combine the high- and low-frequency features of the same image; and the reconstruction stage is used to train the network's ability to reconstruct the image from the combination of high- and low-frequency features.

[0008] The second stage is the testing stage. During the testing stage, the image processing network structure also includes three main stages: frequency decomposition stage, feature fusion stage and reconstruction stage. During the testing stage, the final fused image is generated. The frequency decomposition stage is used to separate and extract the high- and low-frequency information of visible light and infrared images. The feature fusion stage is used to selectively fuse the low-frequency features of infrared images and the high-frequency features of visible light images. The reconstruction stage is used to generate the final fused image based on the high- and low-frequency features of the combined infrared and visible light images.

[0009] Furthermore, the feature extraction stage in the training stage specifically includes the following steps:

[0010] S1. First, two independent conventional convolution operations are performed on each input image. For each input image X, two sets of feature maps are obtained. Then, an average pooling operation is performed on one of the feature maps to smooth the image and filter high-frequency noise.

[0011] S2. Then, the two sets of features Send it to the frequency doubling convolution layer; the role of the frequency doubling convolution is to process the low-frequency and high-frequency information in its corresponding frequency tensor while achieving effective inter-frequency communication; let Y H , Y L Respectively represent the decomposition characteristics of the output; then, from To Y = {Y H ,Y L There are two communication methods for the conversion of}; intra-frequency interaction refers to the convolution between features of the same dimension, similar to our regular convolution; inter-frequency interaction refers to the conversion between high-frequency features and low-frequency features; first, the high-frequency feature X H The dimension is reduced by average pooling and then the feature is merged into the low-frequency feature after convolution; the low-frequency feature X L Upsample by bilinear interpolation and merge it into high-frequency features after convolution;

[0012] S3, after the frequency doubling convolution layer, we get Y = {Y H ,Y L}; Then, the two sets of features Y = {Y H ,Y L}Apply a set of conventional convolutions; in order to enable high-frequency features and low-frequency features to be concatenated in the same dimension, interpolate the low-frequency features to the same size as the high-frequency features; finally, for each source image, obtain the decomposed high- and low-frequency features Z = {Z H ,Z L} from the source image.

[0013] Furthermore, the entire process of step S1 can be expressed as:

[0014] X H = f(X; W H→H ) #(3-1)

[0015] X L = pooling(f(X; W H→L )) #(3-2)

[0016] where f(X; W) represents convolution using the convolution kernel parameter W, and pooling(X) represents average pooling operation with a convolution kernel size of 2×2 and a stride of 2. X H , X L represent the results obtained from the two operation series shown in formulas (3-1) and (3-2) respectively.

[0017] Furthermore, the entire process of the frequency doubling convolution in step S2 can be expressed as:

[0018] Y H = f(X H ; W H→H ) + upsample(f(X L ; W L→H )) #(3-3)

[0019] Y L = f(X L ; W L→L ) + pooling(f(X H ; W H→L )) #(3-4)

[0020] where upsample(X; W) is an upsampling operation with a factor of 2 by bilinear interpolation; the frequency doubling convolution runs directly on the decomposed tensor .

[0021] Furthermore, the entire operation process of step S3 can be expressed as:

[0022] Z H = f(Y H ; W H→H ) #(3-5)

[0023] ZL = upsample(f(Y L ; W L→L ))#(3 - 6).

[0024] Furthermore, the feature combination stage in the training phase specifically includes:

[0025] After passing through the feature extraction network, the high - and low - frequency features Z = {Z H , Z L} of the same input image obtained by splicing are used to reconstruct the input image;

[0026] The reconstruction stage in the training phase specifically includes:

[0027] After obtaining the combined features of the low - frequency and high - frequency features of the input image, it is necessary to reconstruct the combined high - and low - frequency features back into the input image to train its ability to obtain the ability to reconstruct the original input image from the combined high - and low - frequency features; The reconstruction network consists of four consecutive convolutional units and is responsible for reconstructing the combined features into the input image.

[0028] Furthermore, the frequency decomposition stage specifically includes: using a trained feature extraction network containing frequency - doubling convolution to extract the features of infrared and visible light images respectively; for the infrared image X i , obtaining the combined high - and low - frequency features For the visible light image X v , also obtaining the combined high - and low - frequency features

[0029] Furthermore, in the feature fusion stage of the test phase, a new feature fusion strategy is proposed, namely selective splicing; selectively splicing the high - frequency features of the visible light image and the low - frequency features of the infrared image together to generate the final fused image; This feature fusion method makes full use of the respective feature information of visible light and infrared images and also avoids some noise and distortion.

[0030] The advantages and beneficial effects of the present invention are as follows:

[0031] Based on the analysis and exploration of the characteristics and differences in the frequency domain distribution of infrared and visible light images, a new feature extraction method is designed. For the first time, a method of using a convolutional neural network for frequency decomposition is proposed to perform discriminative processing on images of different modalities. A simple and effective fusion network is proposed. First, the high-frequency and low-frequency features of the source image are extracted through frequency decomposition. Then, the high-frequency features of the visible light image and the low-frequency features of the infrared image are selectively fused. Finally, the fused features are reconstructed to obtain the final fused image. The proposed method filters out redundant information and noise while extracting valuable information. Qualitative and quantitative experiments conducted on a public experimental set show that the images generated by the present invention can effectively extract the beneficial information of the source image, with prominent targets, rich and clear texture details, and conforming to human visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is the flowchart of image processing in the training stage of the preferred embodiment provided by the present invention

[0033] Figure 2 is the visualization effect diagram in the network training stage;

[0034] Figure 3 is the result diagram of frequency analysis for feature extraction;

[0035] Figure 4 is the flowchart of image processing in the test stage. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and detailedly described in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0037] The technical solution for the present invention to solve the above technical problems is:

[0038] The present invention can be divided into two stages: the first stage is the training stage, in which the network is trained by pre-training with a single natural image to separate and extract the high-frequency and low-frequency features of the image and the ability to reconstruct the image from the high-frequency and low-frequency combined features. The second stage is the test stage, in which the final fused image is generated.

[0039] The training stage is implemented according to the following steps:

[0040] Training stage:

[0041] The flowchart of the entire image processing during the training process is as Figure 1 shown. In the training stage, the network structure for image processing mainly includes three stages: the feature extraction stage, the feature combination stage, and the reconstruction stage. Next, we will analyze the components in the system.

[0042] (1) Feature extraction stage

[0043] First, two independent conventional convolution operations are performed on each input image. For each input image X, two sets of feature maps will be obtained. Then, an average pooling operation is performed on one of the sets of feature maps, which will reduce the image by one dimension in the spatial dimension. At the same time, this operation will smooth the image and filter out high-frequency noise. The whole process can be expressed as:

[0044] X H = f(X; W H→H )#(3-1)

[0045] X L = pooling(f(X; W H→L ))#(3-2)

[0046] where f(X; W) represents convolution using the convolution kernel parameter W, and pooling(X) represents an average pooling operation with a convolution kernel size of 2×2 and a stride of 2. X H , X L represent the results obtained from the two operation series respectively.

[0047] Then, these two sets of features are fed into the frequency doubling convolution layer. The blue part represents the processing process on X H , and the brown part represents the processing process on X L . The role of frequency doubling convolution is to efficiently process the low-frequency and high-frequency information in its corresponding frequency tensors while achieving effective inter-frequency communication. Let Y H , Y L represent the output decomposed features respectively. Then, there are two communication methods for the conversion from to Y = {Y H , Y L}. Intra-frequency interaction refers to the convolution between features of the same dimension, similar to our conventional convolution. Inter-frequency interaction refers to the conversion between high-frequency and low-frequency features. First, the high-frequency feature X H is reduced in dimension through average pooling, and then this feature is merged into the low-frequency feature after convolution. The low-frequency feature X L is upsampled through bilinear interpolation, and then it is merged into the high-frequency feature after convolution. The whole process of frequency doubling convolution can be expressed as:

[0048] Y H = f(X H ; W H→H ) + upsample(f(X L ; W L→H ))#(3-3)

[0049] Y L = f(X L ; W L→L ) + pooling(f(X H ; W H→L ))#(3 - 4)

[0050] where upsample(X; W) performs an upsampling operation with a factor of 2 by bilinear interpolation. Doubled-frequency convolution can run directly on the decomposed tensors without any additional computation or memory overhead. The larger receptive field of the low-frequency feature maps is another benefit brought by the frequency decomposition approach. The doubled-frequency convolutional layer can capture more context information from distant positions in the low-frequency part of the convolution, thus effectively doubling the receptive field compared to normal convolution. Therefore, the low-frequency part of the network can better extract the global structure and contour information of the infrared image.

[0051] After the doubled-frequency convolutional layer, we obtain Y = {Y H , Y L}. Then, we apply a set of conventional convolutions to the two sets of features Y = {Y H , Y L} obtained. To enable the high-frequency and low-frequency features to be concatenated in the same dimension, the low-frequency features are interpolated to the same size as the high-frequency features. Finally, for each source image, the decomposed high- and low-frequency features Z = {Z H , Z L} are obtained from the source image. The entire operation process can be expressed as:

[0052] Z H = f(Y H ; W H→H )#(3 - 5)

[0053] Z L = upsample(f(Y L ; W L→L ))#(3 - 6)

[0054] Such as Figure 2As shown, we visualized the training process of the proposed method. The first row shows the original input image, the second row shows the low-frequency features extracted by the network, and the third row shows the high-frequency features extracted by the network. It can be seen that after the feature extraction network, the original input image is well decomposed into high-frequency and low-frequency features. The last row is the image reconstructed by combining the high-frequency and low-frequency features. The slight difference between the reconstructed image and the original image indicates that our reconstruction network can well reconstruct the image according to the combined high-frequency and low-frequency features. In fact, during the training phase of the network, our feature extractor obtains the low-frequency features of the original image through average pooling operation and nearest neighbor interpolation. However, the feature extraction network does not specifically process the high-frequency part of the image except for conventional convolution and frequency domain interaction. In fact, because the feature extraction network needs to provide sufficient information (high-frequency and low-frequency information) to reconstruct the original image. The extracted low-frequency feature maps of the network are sent to the image reconstruction module. In order to reconstruct the original input image, the high-frequency feature extraction branch of the network must tend to extract the high-frequency features of the image.

[0055] To perform frequency analysis on the extracted features, we performed Fourier transform on the feature map groups of 1000 images to obtain their spectrograms. Figure 3 The average result is shown. Figure 3 (b) shows the average result of the spectrograms of the low-frequency feature groups. We can see that in the low-frequency features, the information is mainly concentrated in the low-frequency region. Figure 3 (a) shows the average result of the spectrograms of the high-frequency feature groups. It can be found that the spectrograms contain a lot of high-frequency information not found in the low-frequency groups. The above frequency analysis illustrates the ability of the proposed network to decompose image features in the frequency domain.

[0056] (2) Feature combination stage

[0057] After passing through the feature extraction network, we concatenate the high- and low-frequency features Z = {Z H , Z L} of the same input image obtained for reconstructing the input image.

[0058] (3) Reconstruction stage

[0059] After obtaining the combined features of the low-frequency and high-frequency features of the input image, it is necessary to reconstruct the combined high- and low-frequency features back into the input image to train its ability to obtain the reconstructed original input image from the combined high- and low-frequency features. The reconstruction network consists of four consecutive convolutional units, which are responsible for reconstructing the combined features into the input image.

[0060] Testing phase:

[0061] The proposed network has obtained the ability to separate and extract the high- and low-frequency information of images and reconstruct images from the combined high- and low-frequency features through supervised training with a large number of natural images during the training phase. In the testing phase, the proposed testing framework uses the pre-trained network to decompose and extract the features of infrared and visible light images and reconstruct the final fused image. As Figure 4 , in the testing phase, the image processing network structure also includes three main phases: the frequency decomposition phase, the feature fusion phase, and the reconstruction phase.

[0062] (1) Frequency decomposition phase

[0063] In the frequency decomposition phase, a trained feature extraction network containing frequency doubling is used to extract the features of infrared and visible light images respectively. For the infrared image X i , the combined features of high and low frequencies are obtained For the visible light image X v , the combined features of high and low frequencies are also obtained

[0064] (2) Feature fusion phase

[0065] In the feature fusion phase, we propose a new feature fusion strategy, namely selective splicing. In this strategy, the high-frequency features of the visible light image obtained after passing through the feature extraction network are selectively and the low-frequency features of the infrared image spliced together to generate the final fused image. The advantage of this feature fusion method is that it can make full use of the respective feature information of visible light and infrared images, thereby improving the quality and details of the fused image. At the same time, selectively fusing high- and low-frequency features can also avoid the influence of some noise and distortion.

[0066] (3) Reconstruction phase

[0067] After obtaining the fused features of the low-frequency features of the infrared image and the high-frequency features of the visible light image, it is necessary to reconstruct the fused features into an ideal fused image. In the training phase, the reconstruction network reconstructs the original input image through the combined high- and low-frequency features extracted from a single original input image. In the testing phase, we still input the combined high- and low-frequency features to the reconstruction network, but the low-frequency features come from the infrared image and the high-frequency features come from the visible light image. Since the extracted features are a combination of valuable information from the two images, the reconstructed image combines the advantages of the two modal images. At the same time, the interference of noise and the uncertainty brought by redundant information are also reduced.

[0068] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0069] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0070] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the element.

[0071] The above embodiments should be understood as being only for illustrative purposes of the present invention and not for limiting the protection scope of the present invention. After reading the content described in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. An image fusion method based on the modal difference between infrared and visible light images, characterized in that, It includes the following two stages: The first stage is the training stage. In this stage, the network is trained to separate and extract the high and low frequency features of images and reconstruct images from the combined high and low frequency features by using the method of pre-training with a single natural image; The training stage includes three sub-stages: the feature extraction stage, the feature combination stage, and the reconstruction stage. The feature extraction stage is used to train the network's ability to separate and extract the high and low frequency features of images. The feature combination stage is used to combine the high and low frequency features of the same image obtained; The reconstruction stage is used to train the network's ability to reconstruct images from the combined high and low frequency features; The second stage is the testing stage. In the testing stage, the image processing network structure also includes three main stages: the frequency decomposition stage, the feature fusion stage, and the reconstruction stage. In the testing stage, the final fused image is generated. The frequency decomposition stage is used to separately and extract the features of the infrared and visible light image modalities in a discriminatory manner. The feature fusion stage is used to selectively fuse the low frequency features of the infrared image and the high frequency features of the visible light image; The reconstruction stage is used to generate the final fused image based on the combined high and low frequency features of the infrared and visible light images; The feature extraction stage in the training stage specifically includes the following steps: S1. First, perform two independent conventional convolution operations on each input image. For each input image X, two sets of feature maps will be obtained. Then, perform an average pooling operation on one of the sets of feature maps to smooth the image and filter out high frequency noise; S2. Then, send the two sets of features to the frequency doubling convolutional layer; the role of frequency doubling convolution is to process the low-frequency and high-frequency information in its corresponding frequency tensor while achieving effective inter-frequency communication; let Y H , Y L respectively represent the decomposed features of the output; then, from to Y = {Y H , Y L}, there are two communication methods for the conversion; intra-frequency interaction refers to the convolution between features of the same dimension, and inter-frequency interaction refers to the conversion between high-frequency and low-frequency features; first, the high-frequency feature X H is reduced in dimension through average pooling and then merged into the low-frequency feature after convolution; the low-frequency feature X L is upsampled through bilinear interpolation and merged into the high-frequency feature after convolution; S3. After the frequency doubling convolution layer, we obtain Y = {Y H , Y L}; then, apply a set of conventional convolutions to the two sets of features Y = {Y H , Y L}; in order to enable the high-frequency features and low-frequency features to be concatenated in the same dimension, interpolate the low-frequency features to the same size as the high-frequency features; finally, for each source image, obtain the decomposed high- and low-frequency features Z = {Z H , Z L}; The whole process of step S1 can be expressed as: X H = f(X; W H→H ) (3-1) X L = pooling(f(X; W H→L )) (3-2) Among them, f(X; W) represents convolution using the convolution kernel parameter W, pooling(X) represents average pooling operation with a convolution kernel size of 2×2 and a stride of 2, and X H , X L respectively represent the results obtained from the operation series shown in formulas (3-1) and (3-2); The whole process of the frequency doubling convolution in step S2 can be expressed as: Y H = f(X H ; W H→H ) + upsample(f(X L ; W L→H )) (3 - 3) Y L = f(X L ; W L→L ) + pooling(f(X H ; W H→L )) (3 - 4) Among them, upsample(X; W) performs an upsampling operation with a factor of 2 through bilinear interpolation; the frequency doubling convolution runs directly on the decomposed tensors above; The whole operation process of step S3 can be expressed as: Z H = f(Y H ; W H→H ) (3 - 5) Z L = upsample(f(Y L ; W L→L )) (3-6) The feature combination stage in the training stage specifically includes: After passing through the feature extraction network, the high-frequency and low-frequency features Z = {Z H , Z L} of the same input image obtained by splicing are used to reconstruct the input image; The reconstruction stage in the training stage specifically includes: After obtaining the combined features of the low frequency features and high frequency features of the input image, it is necessary to reconstruct the combined high and low frequency features back into the input image to train its ability to obtain the original input image from the combined high and low frequency features. The reconstruction network consists of four consecutive convolutional units and is responsible for reconstructing the combined features into the input image.

2. The image fusion method based on the modality difference between infrared and visible light images according to claim 1, wherein, The specific steps of the frequency decomposition stage in the testing process are as follows: Use a trained feature extraction network containing frequency doubling convolution to extract the features of infrared and visible light images respectively; for the infrared image X i , obtain the combined high-frequency and low-frequency features For the visible light image X v , also obtain the combined high-frequency and low-frequency features 3. The image fusion method based on the modal difference between infrared and visible light images according to claim 1, wherein, In the feature fusion stage of the testing phase, a new feature fusion strategy, namely selective splicing, is proposed; selectively splice the high-frequency features of the visible light image obtained after passing through the feature extraction network and the low-frequency features of the infrared image together to generate the final fused image.

Images