A lightweight complex scene image fusion model and real-time image fusion method

Abstract

The present application belongs to the field of image processing technology, and discloses a lightweight complex scene image fusion model and a real-time image fusion method, which utilize the complementary capabilities of two modal images in the frequency domain to guide network learning, and can extract key feature information even in the case of severe pixel interference. At the same time, spatial domain branches are used to improve local detail resolution and solve the potential omission problem inherent in frequency domain analysis, so that significant pixel information of different modal images can be effectively captured under the interference of complex factors, thereby improving the quality of the fused image; in addition, the model has a simple structure, high computational efficiency, and high image processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a lightweight complex scene image fusion model and a real-time image fusion method. Background Art

[0002] The fusion of infrared images and visible light images can integrate valuable information from the two images to present the scene more comprehensively and accurately. Therefore, it is widely used in application scenarios such as object detection, semantic segmentation, and autonomous driving.

[0003] Existing infrared image and visible light image fusion methods for complex scenes (such as bad weather, low light environment, noisy environment, etc.) generally adopt a two-stage learning process. First, the image restoration and fusion tasks are processed separately, and then they are interactively optimized. This method has the following problems:

[0004] 1) Error accumulation is prone to occur. Residual interfering pixels or detail loss in the image restoration stage are easily propagated to the fusion stage. Existing fusion models are usually trained under ideal conditions and may misunderstand these interfering features as valuable features, which may exacerbate these problems. For example, in the image restoration task, raindrop textures that blur scene information may be mistakenly enhanced as salient features during the fusion process.

[0005] 2) Introducing erroneous or false features. The fusion of infrared and visible light images attempts to take advantage of the complementary advantages of different modal images, but under low-light conditions, the visible light modality captures very little information, and mainly relies on infrared images. Using low-light enhancement algorithms for preprocessing may inadvertently introduce false or erroneous features in visible light images.

[0006] 3) The model has many parameters and the calculation is complex, the reasoning cost is high, and the processing efficiency is low. Summary of the invention

[0007] The purpose of this application is to provide a lightweight complex scene image fusion model and a real-time image fusion method, which can effectively capture the significant pixel information of different modal images under the interference of complex factors, improve the quality of the fused image, and have high image processing efficiency.

[0008] In the first aspect, the present application provides a lightweight complex scene image fusion model, comprising a first input layer, a frequency domain channel formed by sequentially connecting a first convolutional layer, a first ReLU activation function layer and a frequency domain fusion module, a spatial domain channel formed by sequentially connecting a second convolutional layer, a second ReLU activation function layer and a spatial domain fusion module, a third convolutional layer, a third ReLU activation function layer and a first output layer; the first input layer is used to input mutually aligned infrared light images and visible light images, the input end of the frequency domain channel and the input end of the spatial domain channel are both connected to the output end of the first input layer, the third convolutional layer and the third ReLU activation function layer are connected front and back, the output of the frequency domain channel and the output of the spatial domain channel are added element by element and then input into the third convolutional layer, and the output of the third ReLU activation function layer is added element by element to the output of the first input layer and then input into the first output layer;

[0009] The frequency domain fusion module is used to perform Fourier transform on the infrared light feature map and the visible light feature map output by the first ReLU activation function layer to obtain an infrared light phase spectrum, an infrared light amplitude spectrum, a visible light phase spectrum and a visible light amplitude spectrum, use the infrared light amplitude spectrum to guide the visible light amplitude spectrum to perform amplitude recovery, and use the visible light phase spectrum to guide the infrared light phase spectrum to perform phase recovery, and use the infrared light phase spectrum, the infrared light amplitude spectrum, the visible light phase spectrum, the visible light amplitude spectrum, the visible light amplitude spectrum after amplitude recovery, and the infrared light phase spectrum after phase recovery to generate a frequency domain channel fusion feature map;

[0010] The spatial domain fusion module is used to perform sparse feature extraction and fusion on the infrared light feature map and the visible light feature map output by the second ReLU activation function layer to generate a spatial domain channel fusion feature map.

[0011] This lightweight complex scene image fusion model uses the complementary capabilities of two modal images in the frequency domain to guide network learning. Even in the case of severe pixel interference, it can extract key feature information. At the same time, it uses spatial domain branches to improve local detail resolution and solve the potential omission problem inherent in frequency domain analysis. It can effectively capture significant pixel information of different modal images under the interference of complex factors and improve the quality of the fused image. In addition, the model has a simple structure, high computational efficiency, and high image processing efficiency.

[0012] Preferably, the frequency domain fusion module includes a second input layer, a third input layer, a Fourier transform module, a guide network, an inverse Fourier transform module, a first convolution block and a second output layer;

[0013] The second input layer and the third input layer are used to input the visible light feature map and the infrared light feature map output by the first ReLU activation function layer, respectively, and the output end of the second input layer and the output end of the third input layer are both connected to the input end of the Fourier transform module;

[0014] The Fourier transform module is used to perform Fourier transform on the infrared light feature map and the visible light feature map output by the first ReLU activation function layer to obtain an infrared light phase spectrum, an infrared light amplitude spectrum, a visible light phase spectrum and a visible light amplitude spectrum, and then input them into the guidance network;

[0015] The guiding network is used to use the infrared light amplitude spectrum to guide the visible light amplitude spectrum for amplitude recovery, and use the visible light phase spectrum to guide the infrared light phase spectrum for phase recovery, and use the infrared light phase spectrum, the infrared light amplitude spectrum, the visible light phase spectrum, the visible light amplitude spectrum, the visible light amplitude spectrum after amplitude recovery, and the infrared light phase spectrum after phase recovery to generate fused frequency domain features, and input the fused frequency domain features into the inverse Fourier transform module;

[0016] The inverse Fourier transform module is used to convert the fused frequency domain features into a spatial domain feature map through inverse Fourier transform, and input the map into the first convolution block;

[0017] The first convolution block is used to perform convolution processing on the spatial domain feature map to obtain a frequency domain channel fusion feature map, and input the frequency domain channel fusion feature map into the second output layer.

[0018] In images degraded by interference from bad weather, weak light environment, noisy environment, etc., the representation space of degraded pixels is usually limited to the amplitude of the image. In addition, compared with the amplitude spectrum, the phase spectrum represents the relative position or offset of the signal and mainly contains the structural information of the image. In the image fusion process, the infrared light amplitude spectrum, which is less affected by interference, is used to guide the visible light amplitude spectrum for restoration, which can effectively reduce the influence of interference information on the fusion result. In addition, the visible light phase spectrum is used to guide the infrared light phase spectrum for phase recovery, so that the fusion result retains more valuable detail information.

[0019] Preferably, the guide network includes a deep frequency domain filtering module, a local attention module, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer and a seventh convolutional layer;

[0020] The infrared light amplitude spectrum is input into the deep frequency domain filtering module to obtain a first guided attention map output by the deep frequency domain filtering module, the first guided attention map is multiplied element by element with the visible light amplitude spectrum to obtain a visible light amplitude spectrum after amplitude restoration, the visible light amplitude spectrum after amplitude restoration is input into the fifth convolutional layer, and the output of the fifth convolutional layer is added element by element with the visible light amplitude spectrum to obtain a visible light amplitude feature;

[0021] The visible light phase spectrum is input into the fourth convolution layer to obtain the visible light phase feature output by the fourth convolution layer;

[0022] The visible light amplitude feature and the visible light phase feature are combined to generate a visible light frequency domain feature;

[0023] The visible light phase spectrum is also input into the local attention module to obtain a second guided attention map output by the local attention module, the second guided attention map is multiplied element by element with the infrared light phase spectrum to obtain a phase-recovered infrared light phase spectrum, the phase-recovered infrared light phase spectrum is input into the sixth convolutional layer, and the output of the sixth convolutional layer is added element by element with the infrared light phase spectrum to obtain an infrared light phase feature;

[0024] The infrared light amplitude spectrum is also input into the seventh convolution layer to obtain the infrared light amplitude feature output by the seventh convolution layer;

[0025] The infrared light amplitude feature and the infrared light phase feature are combined to generate an infrared light frequency domain feature;

[0026] The visible light frequency domain feature and the infrared light frequency domain feature are added to obtain the fused frequency domain feature.

[0027] The deep frequency domain filtering module is used to generate the first guided attention map to guide the visible light amplitude spectrum for amplitude recovery, which can enhance the transferable frequency components and suppress the frequency components in the latent space that are not conducive to generalization; the local attention module is used to generate the second guided attention map to guide the infrared light phase spectrum for phase recovery, which can better extract the weak texture information in the infrared light image.

[0028] Preferably, the first convolution block includes an eighth convolution layer, a fourth ReLU activation function layer, a ninth convolution layer, a fifth ReLU activation function layer, a tenth convolution layer and a first Sigmoid activation function layer connected sequentially from front to back.

[0029] Preferably, the deep frequency domain filtering module includes an eleventh convolution layer, a BN layer, a sixth ReLU activation function layer, a first maximum pooling layer, a global average pooling layer, a twelfth convolution layer and a second Sigmoid activation function layer;

[0030] The eleventh convolution layer, the BN layer and the sixth ReLU activation function layer are connected in sequence, and the input end of the eleventh convolution layer serves as the input end of the deep frequency domain filtering module; the twelfth convolution layer and the second Sigmoid activation function layer are connected in sequence; the output end of the sixth ReLU activation function layer is respectively connected to the input end of the first maximum pooling layer and the input end of the global average pooling layer, and the output end of the first maximum pooling layer and the output end of the global average pooling layer are connected to the input end of the twelfth convolution layer after channel cascading; the output of the sixth ReLU activation function layer is multiplied element by element with the output of the second Sigmoid activation function layer as the output of the deep frequency domain filtering module.

[0031] Preferably, the local attention module includes a second maximum pooling layer and a third Sigmoid activation function layer connected front and back.

[0032] Preferably, the spatial domain fusion module includes a fourth input layer and a thirteenth convolution layer connected front and back, a fifth input layer and a fourteenth convolution layer connected front and back, a third maximum pooling layer, a fifteenth convolution layer and a seventh ReLU activation function layer connected in sequence from front to back, a convolution path formed by a plurality of second convolution blocks connected in series, and a third output layer;

[0033] The output of the thirteenth convolutional layer and the output of the fourteenth convolutional layer are added element by element to obtain a first fused feature map, the first fused feature map is input into the third maximum pooling layer, the output of the seventh ReLU activation function layer is multiplied element by element with the first fused feature map to obtain a second fused feature map, the second fused feature map is input into the convolution path, and the output of the convolution path is added element by element with the second fused feature map and then input into the third output layer.

[0034] Preferably, the convolution path includes three second convolution blocks, and the second convolution block includes a sixteenth convolution layer and an eighth ReLU activation function layer connected front and back.

[0035] In a second aspect, the present application provides a real-time image fusion method, comprising the steps of:

[0036] A1. Constructing an initial lightweight complex scene image fusion model; the initial lightweight complex scene image fusion model is the lightweight complex scene image fusion model described above;

[0037] A2. training the initial lightweight complex scene image fusion model to obtain a trained lightweight complex scene image fusion model;

[0038] A3. Input the mutually registered infrared image to be fused and the visible light image to be fused into the trained lightweight complex scene image fusion model to obtain a fused image output by the trained lightweight complex scene image fusion model.

[0039] Preferably, in step A2, the initial lightweight complex scene image fusion model is trained based on the following total loss function:

[0040] ;

[0041] ;

[0042] ;

[0043] in, is the total loss function, is the spatial domain loss function, is the frequency domain loss function, is the output image of the initial lightweight complex scene image fusion model, The infrared light image in the training sample of the initial lightweight complex scene image fusion model is input, The visible light image in the training sample of the initial lightweight complex scene image fusion model is input, Represents a Fast Fourier Transform operation.

[0044] Beneficial effects: The lightweight complex scene image fusion model and real-time image fusion method provided by the present application utilize the complementary capabilities of two modal images in the frequency domain to guide network learning, and can extract key feature information even in the case of severe pixel interference. At the same time, spatial domain branches are used to improve local detail resolution and solve the potential omission problem inherent in frequency domain analysis, so that significant pixel information of different modal images can be effectively captured under the interference of complex factors, thereby improving the quality of the fused image; in addition, the model structure is simple, the computational efficiency is high, and the image processing efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of a lightweight complex scene image fusion model provided in an embodiment of the present application.

[0046] Figure 2 It is a structural diagram of the frequency domain fusion module.

[0047] Figure 3 Schematic diagram of the structure of the deep frequency domain filtering module.

[0048] Figure 4 Schematic diagram of the structure of the local attention module.

[0049] Figure 5 It is a structural diagram of the spatial domain fusion module.

[0050] Figure 6 A flowchart of a real-time image fusion method provided in an embodiment of the present application.

[0051] Figure 7 This is a comparison chart of the first set of experimental results.

[0052] Figure 8 This is a comparison chart of the second set of experimental results.

[0053] Fig. 9 This is a comparison chart of the third set of experimental results.

[0054] Fig.10 This is a comparison chart of the fourth set of experimental results.

[0055] Explanation of reference numerals: 1. first input layer; 2. first convolution layer; 3. first ReLU activation function layer; 4. frequency domain fusion module; 401. second input layer; 402. third input layer; 403. Fourier transform module; 404. guided network; 405. inverse Fourier transform module; 406. first convolution block; 407. second output layer; 5. second convolution layer; 6. second ReLU activation function layer; 7. spatial domain fusion module; 701. fourth input layer; 702. thirteenth convolution layer; 703. fifth input layer; 704. fourteenth convolution layer; 705. third maximum pooling layer; 706. fifteenth convolution layer; 707. seventh ReLU activation function layer; 708. second convolution block; 7081. sixteenth convolution layer; 7082. eighth ReLU activation function layer; 709. convolution path; 710. third output layer; 8. The third convolutional layer; 9, the third ReLU activation function layer; 10, the first output layer; 11, the deep frequency domain filtering module; 1101, the eleventh convolutional layer; 1102, the BN layer; 1103, the sixth ReLU activation function layer; 1104, the first maximum pooling layer; 1105, the global average pooling layer; 1106, the twelfth convolutional layer; 1107, the second Sigmoid activation function layer; 12, the local attention module; 1201, the second maximum pooling layer; 1202, the third Sigmoid activation function layer; 13, the fourth convolutional layer; 14, the fifth convolutional layer; 15, the sixth convolutional layer; 16, the seventh convolutional layer; 17, the eighth convolutional layer; 18, the fourth ReLU activation function layer; 19, the ninth convolutional layer; 20, the fifth ReLU activation function layer; 21, the tenth convolutional layer; 22, the first Sigmoid activation function layer. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0057] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0058] Please refer to Figure 1 , Figure 1 It is a lightweight complex scene image fusion model in some embodiments of the present application (for fusing infrared light images and visible light images), including a first input layer 1, a frequency domain channel formed by connecting a first convolution layer 2, a first ReLU activation function layer 3 and a frequency domain fusion module 4 in sequence, a spatial domain channel formed by connecting a second convolution layer 5, a second ReLU activation function layer 6 and a spatial domain fusion module 7 in sequence, a third convolution layer 8, a third ReLU activation function layer 9 and a first output layer 10; the first input layer 1 is used to input mutually aligned infrared light images and visible light images, the input end of the frequency domain channel and the input end of the spatial domain channel are both connected to the output end of the first input layer 1 (specifically, the input end of the first convolution layer 2 and the input end of the second convolution layer 5 are both connected to the output end of the first input layer 1), the third convolution layer 8 and the third ReLU activation function layer 9 are connected front and back, the output of the frequency domain channel and the output of the spatial domain channel are added element by element and then input to the third convolution layer 8, and the output of the third ReLU activation function layer 9 is added element by element to the output of the first input layer 1 and then input to the first output layer 10;

[0059] The frequency domain fusion module 4 is used to perform Fourier transform on the infrared light feature map and the visible light feature map output by the first ReLU activation function layer 3 to obtain the infrared light phase spectrum , infrared light amplitude spectrum , visible light phase spectrum and the visible light amplitude spectrum , using infrared light amplitude spectroscopy Guiding the visible light amplitude spectrum Perform amplitude recovery and use visible light phase spectroscopy Guided infrared phase spectroscopy Phase retrieval and infrared phase spectroscopy , infrared light amplitude spectrum , visible light phase spectrum , visible light amplitude spectrum , Visible light amplitude spectrum after amplitude recovery And the infrared light phase spectrum after phase recovery Generate frequency domain channel fusion feature map;

[0060] The spatial domain fusion module 7 is used to perform sparse feature extraction and fusion on the infrared light feature map and the visible light feature map output by the second ReLU activation function layer 6 to generate a spatial domain channel fusion feature map.

[0061] This lightweight complex scene image fusion model uses the complementary capabilities of two modal images (visible light images and infrared light images) in the frequency domain to guide network learning. Even in the case of severe pixel interference, it can extract key feature information. At the same time, it uses spatial domain branches (i.e., spatial domain channels) to improve local detail resolution and solve the potential omission problem inherent in frequency domain analysis, so that it can effectively capture significant pixel information of different modal images under the interference of complex factors and improve the quality of the fused image. In addition, the model has a simple structure (therefore, lightweight essentially means that the model structure is simple and the model parameters involved are small in scale), high computational efficiency, and high image processing efficiency.

[0062] Preferably, the first convolutional layer 2, the second convolutional layer 5 and the third convolutional layer 8 are all 3×3 convolutional layers (ie, their convolution kernel sizes are 3×3).

[0063] Specifically, see Figure 2 , the frequency domain fusion module 4 includes a second input layer 401, a third input layer 402, a Fourier transform module 403, a guide network 404, an inverse Fourier transform module 405, a first convolution block 406 and a second output layer 407;

[0064] The second input layer 401 and the third input layer 402 are used to input the visible light feature map and the infrared light feature map output by the first ReLU activation function layer 3, respectively. The output end of the second input layer 401 and the output end of the third input layer 402 are both connected to the input end of the Fourier transform module 403;

[0065] The Fourier transform module 403 is used to perform Fourier transform on the infrared light characteristic map and the visible light characteristic map output by the first ReLU activation function layer 3 to obtain the infrared light phase spectrum. , infrared light amplitude spectrum , visible light phase spectrum and the visible light amplitude spectrum (Specifically, the infrared light feature map output by the first ReLU activation function layer 3 is subjected to Fourier transform to obtain the first frequency domain feature, and then the infrared light phase spectrum is calculated based on the real part and imaginary part of the first frequency domain feature. and infrared light amplitude spectrum , the visible light feature map output by the first ReLU activation function layer 3 is Fourier transformed to obtain the second frequency domain feature, and then the visible light phase spectrum is calculated based on the real part and imaginary part of the second frequency domain feature and the visible light amplitude spectrum , the specific calculation method is the existing technology and will not be described in detail here), and then input into the guidance network 404;

[0066] Guiding network 404 for utilizing infrared light amplitude spectrum Guiding the visible light amplitude spectrum Perform amplitude recovery and use visible light phase spectroscopy Guided infrared phase spectroscopy Phase retrieval and infrared phase spectroscopy , infrared light amplitude spectrum , visible light phase spectrum , visible light amplitude spectrum , Visible light amplitude spectrum after amplitude recovery And the infrared light phase spectrum after phase recovery Generate fused frequency domain features , the fusion frequency domain features Input inverse Fourier transform module 405;

[0067] The inverse Fourier transform module 405 is used to transform the fused frequency domain features by inverse Fourier transform. Converted into a spatial domain feature map and input into the first convolution block 406;

[0068] The first convolution block 406 is used to perform convolution processing on the spatial domain feature map to obtain a frequency domain channel fusion feature map, and input it into the second output layer 407.

[0069] In images degraded by interference from bad weather, weak light environment, noisy environment, etc., the representation space of degraded pixels is usually limited to the amplitude of the image. In addition, compared with the amplitude spectrum, the phase spectrum represents the relative position or offset of the signal and mainly contains the structural information of the image. In the image fusion process, the infrared light amplitude spectrum, which is less affected by interference, is used to guide the visible light amplitude spectrum for restoration, which can effectively reduce the influence of interference information on the fusion result. In addition, the visible light phase spectrum is used to guide the infrared light phase spectrum for phase recovery, so that the fusion result retains more valuable detail information.

[0070] Further, see Figure 2, the guide network 404 includes a deep frequency domain filtering module 11, a local attention module 12, a fourth convolutional layer 13, a fifth convolutional layer 14, a sixth convolutional layer 15 and a seventh convolutional layer 16;

[0071] Infrared Amplitude Spectrum Input the deep frequency domain filtering module 11 to obtain the first attention-guiding map output by the deep frequency domain filtering module 11 , the first guided attention map Visible light amplitude spectrum Multiply element by element to get the amplitude spectrum of visible light after amplitude restoration , the visible light amplitude spectrum after amplitude recovery Input the fifth convolution layer 14, the output of the fifth convolution layer 14 and the visible light amplitude spectrum Add element by element to get the visible light amplitude characteristics ;

[0072] Visible light phase spectrum Input the fourth convolution layer 13 to obtain the visible light phase feature output by the fourth convolution layer 13 ;

[0073] Visible light amplitude characteristics and visible light phase characteristics Combined to generate visible light frequency domain features (Specifically, according to the visible light amplitude characteristics and visible light phase characteristics Calculate the visible light frequency domain characteristics the real and imaginary parts of );

[0074] Visible light phase spectrum The local attention module 12 is also input to obtain the second guided attention map output by the local attention module 12 , the second guided attention map Infrared phase spectrum Multiply element by element to get the infrared light phase spectrum after phase recovery , infrared light phase spectrum after phase recovery Input the sixth convolution layer 15, the output of the sixth convolution layer 15 and the infrared light phase spectrum Add element by element to get the infrared light phase characteristics ;

[0075] Infrared Amplitude Spectrum It is also input into the seventh convolutional layer 16 to obtain the infrared light amplitude feature output by the seventh convolutional layer 16 ;

[0076] Infrared light amplitude characteristics and infrared phase characteristics Combined to generate infrared light frequency domain features (Specifically, according to the infrared light amplitude characteristics and infrared phase characteristics Calculate the infrared light frequency domain characteristics the real and imaginary parts of );

[0077] Visible light frequency domain characteristics and infrared light frequency domain characteristics Add together to get the fused frequency domain features (Specifically, using the visible light frequency domain features The real part and infrared light frequency domain characteristics Add the real part of The real part of the visible light frequency domain feature The imaginary part and infrared light frequency domain characteristics Add the imaginary parts of of the imaginary part).

[0078] The first guided attention map is generated using the deep frequency domain filtering module 11 To guide the visible light amplitude spectrum Amplitude recovery can enhance the transferable frequency components and suppress the frequency components in the latent space that are not conducive to generalization; the local attention module 12 is used to generate the second guided attention map To guide the infrared light phase spectrum Phase recovery can better extract weak texture information in infrared images.

[0079] Among them, the fourth convolutional layer 13, the fifth convolutional layer 14, the sixth convolutional layer 15 and the seventh convolutional layer 16 are preferably 1×1 convolutional layers.

[0080] Further, see Figure 2 , the first convolution block 406 includes an eighth convolution layer 17, a fourth ReLU activation function layer 18, a ninth convolution layer 19, a fifth ReLU activation function layer 20, a tenth convolution layer 21 and a first Sigmoid activation function layer 22, which are sequentially connected from front to back. Among them, the eighth convolution layer 17, the ninth convolution layer 19 and the tenth convolution layer 21 are preferably 3×3 convolution layers.

[0081] In this embodiment, see Figure 3 , the deep frequency domain filtering module 11 includes an eleventh convolution layer 1101, a BN layer 1102 (i.e., a batch normalization layer), a sixth ReLU activation function layer 1103, a first maximum pooling layer 1104, a global average pooling layer 1105, a twelfth convolution layer 1106, and a second Sigmoid activation function layer 1107;

[0082] The eleventh convolutional layer 1101, the BN layer 1102 and the sixth ReLU activation function layer 1103 are connected in sequence, and the input end of the eleventh convolutional layer 1101 is used as the input end of the deep frequency domain filtering module 11; the twelfth convolutional layer 1106 and the second Sigmoid activation function layer 1107 are connected in sequence; the output end of the sixth ReLU activation function layer 1103 is respectively connected to the input end of the first maximum pooling layer 1104 and the input end of the global average pooling layer 1105, and the output end of the first maximum pooling layer 1104 and the output end of the global average pooling layer 1105 are connected to the input end of the twelfth convolutional layer 1106 after channel cascading; the output of the sixth ReLU activation function layer 1103 is multiplied element by element with the output of the second Sigmoid activation function layer 1107 as the output of the deep frequency domain filtering module 11.

[0083] The deep frequency domain filtering module 11 using this structure for guidance can effectively enhance the transferable frequency components and suppress the frequency components in the latent space that are not conducive to generalization.

[0084] Among them, the eleventh convolution layer 1101 is preferably a 1×1 convolution layer, and the twelfth convolution layer 1106 is preferably a 9×9 convolution layer.

[0085] Further, see Figure 4 The local attention module 12 includes a second maximum pooling layer 1201 and a third Sigmoid activation function layer 1202 connected front and back. The local attention module 12 with this structure can be used for guidance to effectively extract weak texture information in infrared images.

[0086] Specifically, see Figure 5 , the spatial domain fusion module 7 includes a fourth input layer 701 and a thirteenth convolution layer 702 connected front and back, a fifth input layer 703 and a fourteenth convolution layer 704 connected front and back, a third maximum pooling layer 705, a fifteenth convolution layer 706 and a seventh ReLU activation function layer 707 connected from front to back, a convolution path 709 formed by a plurality of second convolution blocks 708 connected in series, and a third output layer 710;

[0087] The output of the thirteenth convolutional layer 702 and the output of the fourteenth convolutional layer 704 are added element by element to obtain the first fused feature map , the first fusion feature map Input the third maximum pooling layer 705, the output of the seventh ReLU activation function layer 707 and the first fusion feature map Multiply element by element to get the second fusion feature map , the second fusion feature map The convolution path 709 is input, and the output of the convolution path 709 is element-by-element with the second fusion feature map. After addition (after addition, the spatial domain channel fusion feature map is obtained), it is input into the third output layer 710.

[0088] Spatial domain information is used as compensation for details ignored in frequency domain learning, and only sparse and significant pixel information needs to be captured. Maximum pooling is a simple and effective tool. Maximum pooling selects the feature with the highest response in each window while discarding other weak details. This mechanism enables the model to not generate redundant features and save computing resources. Therefore, the third maximum pooling layer 705 is introduced in the spatial domain fusion module 7.

[0089] In some embodiments, see Figure 5 The convolution path 709 includes three second convolution blocks 708, and the second convolution block 708 includes a sixteenth convolution layer 7081 and an eighth ReLU activation function layer 7082 connected front and back.

[0090] Among them, the thirteenth convolution layer 702, the fourteenth convolution layer 704 and the sixteenth convolution layer 7081 are preferably 3×3 convolution layers, and the fifteenth convolution layer 706 is preferably a 1×1 convolution layer.

[0091] refer to Figure 6 The present application provides a real-time image fusion method (for fusing infrared light images and visible light images), comprising the steps of:

[0092] A1. Construct an initial lightweight complex scene image fusion model; the initial lightweight complex scene image fusion model is the lightweight complex scene image fusion model mentioned above (that is, the structure of the initial lightweight complex scene image fusion model is as follows Figure 1 shown);

[0093] A2. training the initial lightweight complex scene image fusion model to obtain a trained lightweight complex scene image fusion model;

[0094] A3. Input the mutually registered infrared image to be fused and the visible light image to be fused into the trained lightweight complex scene image fusion model to obtain a fused image output by the trained lightweight complex scene image fusion model.

[0095] Since a lightweight complex scene image fusion model is used for image fusion, the fusion efficiency is high and the real-time performance of the fusion processing can be improved.

[0096] Preferably, in step A2, the initial lightweight complex scene image fusion model is trained based on the following total loss function:

[0097] ;

[0098] ;

[0099] ;

[0100] in, is the total loss function, is the spatial domain loss function, is the frequency domain loss function, is the output image of the initial lightweight complex scene image fusion model, The infrared image in the training sample of the initial lightweight complex scene image fusion model is input. The visible light image in the training sample of the initial lightweight complex scene image fusion model is input. Represents the fast Fourier transform operation. The above total loss function takes into account the losses in the spatial domain and the frequency domain, which can make the fused image output by the trained lightweight complex scene image fusion model have higher quality.

[0101] Specifically, step A2 includes:

[0102] A201. Obtain a training sample data set; the training sample data set includes a plurality of training samples, each training sample includes an infrared image and a visible light image that are registered with each other;

[0103] A202. Select an unused training sample from the training sample data set, input the infrared image and visible light image of the selected training sample into the initial lightweight complex scene image fusion model, and obtain an output image of the initial lightweight complex scene image fusion model;

[0104] A203. Calculate the total loss function according to the output image of the initial lightweight complex scene image fusion model and the infrared image and visible light image of the corresponding training sample (see the previous text for the specific calculation formula);

[0105] A204. Update the model parameters of the initial lightweight complex scene image fusion model according to the total loss function to obtain an updated lightweight complex scene image fusion model;

[0106] A205. If the preset stop condition is met, stop the training and use the updated lightweight complex scene image fusion model as the trained lightweight complex scene image fusion model; otherwise, return to execute step A202.

[0107] Among them, in order to ensure that the trained lightweight complex scene image fusion model can achieve good fusion effect for the fusion of infrared images and visible light images of various complex scenes, the training samples in the training sample data set include training samples containing noise (that is, the source image contains noise, and the source image is the infrared image and visible light image that are registered with each other), training samples containing rain streaks (that is, the source image contains rain streaks), overexposed training samples (that is, the source image is overexposed) and training samples in low-light environments (that is, the source image is an image taken in a low-light environment, such as a night image).

[0108] In step A204, existing technologies may be used to update model parameters, such as gradient descent method, Adam optimization algorithm, etc., but not limited thereto.

[0109] Among them, the preset stopping condition can be: the total loss function calculated for N consecutive times (N is a positive integer, which can be set according to actual needs) is less than a preset loss function threshold (which can be set according to actual needs), or the number of training iterations reaches a preset upper limit of the number of iterations (which can be set according to actual needs).

[0110] The real-time image fusion method of the present application is used to conduct a comparative experiment with several existing image fusion algorithms (hereinafter referred to as the comparative algorithms). In the experiment, these methods are used to fuse the same set of source images. The comparative results of the experimental results are shown in the figure below. Figure 7-Figure 10 As shown, the comparison algorithms include CoConet algorithm, Text-IF algorithm, CDDFuse algorithm, DeFusion algorithm, IGNet algorithm, LRRNet algorithm and Tgfuse algorithm. Figure 7 In , N1 and N2 represent the visible light image and infrared light image in the source image containing noise respectively; Figure 8 In , R1 and R2 represent the visible light image and infrared light image in the source image containing rain streaks, respectively; Fig. 9 In , O1 and O2 represent the visible light image and infrared light image in the overexposed source image respectively; Fig.10 In , L1 and L2 represent the visible light image and infrared light image in the source image of the low-light environment respectively; Figure 7-Figure 10 In the figure, a represents the fusion result of the CoConet algorithm, b represents the fusion result of the Text-IF algorithm, c represents the fusion result of the CDDFuse algorithm, d represents the fusion result of the DeFusion algorithm, e represents the fusion result of the IGNet algorithm, f represents the fusion result of the LRRNet algorithm, g represents the fusion result of the Tgfuse algorithm, and h represents the fusion result of the real-time image fusion method of the present application.

[0111] from Figure 7-Figure 10It can be seen that the real-time image fusion method of the present application can effectively remove noise and rain streaks on the source image, and can effectively restore the contrast of the image and reconstruct detail information when facing overexposure or low light interference; in the four complex scenes of noise, rain, overexposure and low light, the real-time image fusion method of the present application has the best fusion performance.

[0112] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0113] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0115] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0116] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A lightweight complex scene image fusion model, characterized in that: The invention comprises a first input layer (1), a frequency domain channel formed by sequentially connecting a first convolution layer (2), a first ReLU activation function layer (3) and a frequency domain fusion module (4), a spatial domain channel formed by sequentially connecting a second convolution layer (5), a second ReLU activation function layer (6) and a spatial domain fusion module (7), a third convolution layer (8), a third ReLU activation function layer (9) and a first output layer (10); the first input layer (1) is used to input an infrared image and a visible light image that are registered with each other, the input end of the frequency domain channel and the input end of the spatial domain channel are both connected to the output end of the first input layer (1), the third convolution layer (8) and the third ReLU activation function layer (9) are connected front to back, the output of the frequency domain channel and the output of the spatial domain channel are added element by element and then input to the third convolution layer (8), and the output of the third ReLU activation function layer (9) is added element by element to the output of the first input layer (1) and then input to the first output layer (10); The frequency domain fusion module (4) is used to perform Fourier transform on the infrared light feature map and the visible light feature map output by the first ReLU activation function layer (3) to obtain an infrared light phase spectrum, an infrared light amplitude spectrum, a visible light phase spectrum and a visible light amplitude spectrum, use the infrared light amplitude spectrum to guide the visible light amplitude spectrum to perform amplitude recovery, use the visible light phase spectrum to guide the infrared light phase spectrum to perform phase recovery, and use the infrared light phase spectrum, the infrared light amplitude spectrum, the visible light phase spectrum, the visible light amplitude spectrum, the visible light amplitude spectrum after amplitude recovery and the infrared light phase spectrum after phase recovery to generate a frequency domain channel fusion feature map; The spatial domain fusion module (7) is used to perform sparse feature extraction and fusion on the infrared light feature map and the visible light feature map output by the second ReLU activation function layer (6), so as to generate a spatial domain channel fusion feature map; The frequency domain fusion module (4) comprises a second input layer (401), a third input layer (402), a Fourier transform module (403), a guide network (404), an inverse Fourier transform module (405), a first convolution block (406) and a second output layer (407); The second input layer (401) and the third input layer (402) are respectively used to input the visible light feature map and the infrared light feature map output by the first ReLU activation function layer (3), and the output end of the second input layer (401) and the output end of the third input layer (402) are both connected to the input end of the Fourier transform module (403); The Fourier transform module (403) is used to perform Fourier transform on the infrared light characteristic map and the visible light characteristic map output by the first ReLU activation function layer (3) to obtain an infrared light phase spectrum, an infrared light amplitude spectrum, a visible light phase spectrum and a visible light amplitude spectrum, and then input them into the guidance network (404); The guiding network (404) is used to use the infrared light amplitude spectrum to guide the visible light amplitude spectrum for amplitude recovery, and use the visible light phase spectrum to guide the infrared light phase spectrum for phase recovery, and use the infrared light phase spectrum, the infrared light amplitude spectrum, the visible light phase spectrum, the visible light amplitude spectrum, the visible light amplitude spectrum after amplitude recovery, and the infrared light phase spectrum after phase recovery to generate fused frequency domain features, and input the fused frequency domain features into the inverse Fourier transform module (405); The inverse Fourier transform module (405) is used to transform the fused frequency domain features into a spatial domain feature map through inverse Fourier transform, and input the map into the first convolution block (406); The first convolution block (406) is used to perform convolution processing on the spatial domain feature map to obtain a frequency domain channel fusion feature map, and input the frequency domain channel fusion feature map into the second output layer (407).

2. The lightweight complex scene image fusion model according to claim 1, characterized in that: The guide network (404) includes a deep frequency domain filtering module (11), a local attention module (12), a fourth convolutional layer (13), a fifth convolutional layer (14), a sixth convolutional layer (15) and a seventh convolutional layer (16); The infrared light amplitude spectrum is input into the deep frequency domain filtering module (11) to obtain a first guided attention map output by the deep frequency domain filtering module (11); the first guided attention map is element-by-element multiplied with the visible light amplitude spectrum to obtain an amplitude-restored visible light amplitude spectrum; the amplitude-restored visible light amplitude spectrum is input into the fifth convolution layer (14); the output of the fifth convolution layer (14) is element-by-element added with the visible light amplitude spectrum to obtain a visible light amplitude feature; The visible light phase spectrum is input into the fourth convolution layer (13) to obtain a visible light phase feature output by the fourth convolution layer (13); The visible light amplitude feature and the visible light phase feature are combined to generate a visible light frequency domain feature; The visible light phase spectrum is also input into the local attention module (12) to obtain a second guided attention map output by the local attention module (12); the second guided attention map is element-by-element multiplied with the infrared light phase spectrum to obtain a phase-recovered infrared light phase spectrum; the phase-recovered infrared light phase spectrum is input into the sixth convolutional layer (15); the output of the sixth convolutional layer (15) is element-by-element added with the infrared light phase spectrum to obtain an infrared light phase feature; The infrared light amplitude spectrum is also input into the seventh convolution layer (16) to obtain the infrared light amplitude characteristics output by the seventh convolution layer (16); The infrared light amplitude feature and the infrared light phase feature are combined to generate an infrared light frequency domain feature; The visible light frequency domain feature and the infrared light frequency domain feature are added to obtain the fused frequency domain feature.

3. The lightweight complex scene image fusion model according to claim 1, characterized in that: The first convolution block (406) includes an eighth convolution layer (17), a fourth ReLU activation function layer (18), a ninth convolution layer (19), a fifth ReLU activation function layer (20), a tenth convolution layer (21), and a first Sigmoid activation function layer (22) which are sequentially connected from front to back.

4. The lightweight complex scene image fusion model according to claim 2, characterized in that: The deep frequency domain filtering module (11) comprises an eleventh convolution layer (1101), a BN layer (1102), a sixth ReLU activation function layer (1103), a first maximum pooling layer (1104), a global average pooling layer (1105), a twelfth convolution layer (1106) and a second Sigmoid activation function layer (1107); The eleventh convolution layer (1101), the BN layer (1102) and the sixth ReLU activation function layer (1103) are connected in sequence, and the input end of the eleventh convolution layer (1101) serves as the input end of the deep frequency domain filtering module (11); the twelfth convolution layer (1106) and the second Sigmoid activation function layer (1107) are connected in sequence; the output end of the sixth ReLU activation function layer (1103) is respectively connected to the first maximum pooling layer (11 The input end of the first maximum pooling layer (1104) is connected to the input end of the global average pooling layer (1105), the output end of the first maximum pooling layer (1104) and the output end of the global average pooling layer (1105) are connected to the input end of the twelfth convolutional layer (1106) after channel cascading; the output of the sixth ReLU activation function layer (1103) and the output of the second Sigmoid activation function layer (1107) are multiplied element by element as the output of the deep frequency domain filtering module (11).

5. The lightweight complex scene image fusion model according to claim 2, characterized in that: The local attention module (12) includes a second maximum pooling layer (1201) and a third Sigmoid activation function layer (1202) connected front and back.

6. The lightweight complex scene image fusion model according to claim 1, characterized in that: The spatial domain fusion module (7) includes a fourth input layer (701) and a thirteenth convolution layer (702) connected front to back, a fifth input layer (703) and a fourteenth convolution layer (704) connected front to back, a third maximum pooling layer (705), a fifteenth convolution layer (706) and a seventh ReLU activation function (707) layer connected in sequence from front to back, a convolution path (709) formed by a plurality of second convolution blocks (708) connected in series, and a third output layer (710); The output of the thirteenth convolutional layer (702) and the output of the fourteenth convolutional layer (704) are added element by element to obtain a first fused feature map, the first fused feature map is input into the third maximum pooling layer (705), the output of the seventh ReLU activation function (707) layer is multiplied element by element with the first fused feature map to obtain a second fused feature map, the second fused feature map is input into the convolution path (709), and the output of the convolution path (709) is added element by element with the second fused feature map and then input into the third output layer (710).

7. The lightweight complex scene image fusion model according to claim 6, characterized in that: The convolution path (709) includes three second convolution blocks (708), and the second convolution block (708) includes a sixteenth convolution layer (7081) and an eighth ReLU activation function layer (7082) connected front and back.

8. A real-time image fusion method, characterized in that: Includes steps: A1. Constructing an initial lightweight complex scene image fusion model; the initial lightweight complex scene image fusion model is the lightweight complex scene image fusion model according to any one of claims 1-7; A2. training the initial lightweight complex scene image fusion model to obtain a trained lightweight complex scene image fusion model; A3. Input the mutually registered infrared image to be fused and the visible light image to be fused into the trained lightweight complex scene image fusion model to obtain a fused image output by the trained lightweight complex scene image fusion model.

9. The real-time image fusion method according to claim 8, characterized in that: In step A2, the initial lightweight complex scene image fusion model is trained based on the following total loss function: ； ； ； in, is the total loss function, is the spatial domain loss function, is the frequency domain loss function, is the output image of the initial lightweight complex scene image fusion model, The infrared light image in the training sample of the initial lightweight complex scene image fusion model is input, The visible light image in the training sample of the initial lightweight complex scene image fusion model is input, Represents a Fast Fourier Transform operation.

Images