Frequency domain enhancement fusion network method for night infrared and visible light images

Abstract

The invention discloses a frequency domain enhancement fusion network method for night infrared and visible light images, and relates to the technical field of image processing. According to the method, a coarse enhancement network (CENet) is constructed, the night visible light feature extraction capability is improved through deep convolution and point-by-point convolution, a channel importance score is calculated in combination with SENet, finally, a pixel-level enhancement image is generated by using a Tanh activation function, the enhanced visible light feature image and an infrared light image are respectively sent to FDFNet, and the pixel-level enhancement image is obtained. In FDFNet, a U-shaped feature extractor is constructed based on EFA and D3FN modules of a frequency domain, visible light significant information is injected into an infrared feature map through SENet, so that the feature learning ability of a U-Net network is enhanced, finally, images passing through the FDFNet are spliced, a final fused image is obtained through image reconstruction, and the fusion effect is improved. The method provided by the invention is superior to the most advanced method in the aspect of LLVIP data set, shows the potential of further improving the fusion image precision in the future, and is helpful to promote the progress of advanced visual tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a frequency domain enhancement fusion network method for nighttime infrared and visible light images. Background Art

[0002] Due to the limitations of different shooting environments and sensor hardware conditions, a single image often lacks comprehensive information. Therefore, image fusion technology has attracted widespread attention. It extracts useful information from different source images and combines them into a single image, enhancing its subsequent application. The fusion of infrared and visible light images is a current hot research direction. Specifically, infrared images can provide important targets such as prominent pedestrians and vehicles even at night or under other extreme conditions, while visible light images contain rich texture information and have stronger adaptability in visual perception. For these reasons, infrared and visible light images can not only reduce data redundancy, but also generate high-quality images with high contrast and rich texture details. Therefore, they have a wide range of applications in military surveillance, vehicle navigation, and target detection.

[0003] Existing infrared and visible light image fusion methods can be roughly divided into two categories: traditional methods and deep learning-based methods. Traditional methods mainly rely on mathematical transformations. First, they use specific transformations to extract features from the source image in the feature extraction stage. Subsequently, manually created fusion rules are applied to fuse these features. Finally, the fused features are reconstructed into an image through the corresponding inverse transformation. Therefore, based on mathematical transformation, traditional methods can be further divided into four categories: sparse representation-based methods, multi-scale decomposition-based methods, spatial clustering-based methods, and optimization-based methods.

[0004] Deep learning methods usually achieve better performance than traditional methods, providing more possibilities for the development of image fusion. According to the network structure, deep learning methods can be further divided into three categories: AE-based methods, CNN-based methods and GAN-based methods. AE-based methods use autoencoders to complete feature extraction and feature reconstruction, and achieve feature fusion through specific fusion strategies. CNN-based methods design end-to-end network structures, avoid the complexity of manually designing fusion strategies, and achieve feature extraction, feature fusion and feature reconstruction through the constraints of loss functions, thereby producing unique fusion results. GAN-based methods introduce generative adversarial mechanisms into the field of image fusion. Specifically, they use discriminators to constrain the distribution of fusion results generated by the generator, aiming to make the results as close to the source image as possible in an unsupervised manner. Existing deep learning models are basically built on the basis of CNN or GAN networks. Deep learning-based image fusion methods use CNN for specific operations in the feature extraction stage.

[0005] However, existing deep learning-based image fusion methods ignore the impact of extreme environments. In nighttime environments, severe darkness can cause visible light images to degrade dramatically. Therefore, low-light enhancement of low-light visible light images is required.

[0006] Low-light image enhancement methods based on deep learning can be roughly divided into two categories: reference-based methods and no-reference-based methods. However, reference-based methods require a lot of data. In contrast, no-reference methods are more practical and convenient. The characteristic of no-reference methods is that they do not require data selection, but emphasize exploring the potential information in the image. No-reference training methods can be further divided into unsupervised learning and zero-shot learning. HEP provides an effective histogram equalization prior in an unsupervised manner. Specifically, they enhance the similarity between images and restored images by regularizing histogram equalization. For zero-shot learning, ExCNet designs an S-curve to map the brightness to the appropriate range. Zero-DCE also uses a unique curve map to iteratively multiply with the original image to obtain color adaptation results.

[0007] In summary, the prior art has the following technical problems:

[0008] 1) Existing methods are designed for normal lighting conditions, thus ignoring the difficulty of illumination attenuation in nighttime visible images. Specifically, in low-light conditions, previous fusion methods only use infrared information to fill in the scene defects caused by illumination degradation in visible images. As a result, the rich scene information in nighttime visible light images cannot be expressed in the fused image, which deviates from the original intention of the infrared and visible light image fusion task.

[0009] 2) Existing infrared and visible light image fusion methods mainly focus on the spatial domain, using various algorithms and techniques to process and merge image data from two different sensors. These methods usually include steps such as feature extraction, feature fusion, and feature reconstruction, aiming to enhance the visual effect and information content of the image. However, these traditional methods often ignore the valuable information in the frequency domain.

[0010] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0011] The purpose of the present invention is to provide a frequency domain enhancement fusion network method for nighttime infrared and visible light images to solve the technical problems raised in the background technology.

[0012] To achieve the above object, the present invention provides the following technical solution: a frequency domain enhancement fusion network method for nighttime infrared and visible light images, comprising at least the following steps:

[0013] S1: A coarse enhancement network is constructed, the coarse enhancement network is CENet, the CENet cooperates with SENet and Tanh activation function to generate a pixel-level enhancement map, the pixel-level enhancement map at least includes a pixel-level visible light enhancement map, and the SENet is a squeeze and excitation network;

[0014] S2: construct a frequency domain fusion network, and send the enhanced pixel-level visible light enhancement image and infrared light image to the frequency domain fusion network respectively, and the frequency domain fusion network is FDFNet;

[0015] S3: Finally, the images passed through FDFNet are stitched together, and the fused image is reconstructed through the reconstruction network to obtain the final fused image.

[0016] Furthermore, the CENet is a U-Net network composed of 8 depthwise separable convolution blocks, each of which is composed of a depthwise convolution with a stride of 1 and a size of 3×3 and a pointwise convolution with a stride of 1 and a size of 1×1, and uses LeakyReLU as an activation function. The purpose of using depthwise separable convolution is to combine and fuse features through depthwise convolution and pointwise convolution, improve the feature learning ability of the coarse enhancement network, and extract more nighttime visible light image features.

[0017] Furthermore, the S2 at least includes the following steps:

[0018] Given a pair of strictly registered infrared images I ir and visible light image I vis ;

[0019] First, the visible light image I vis Send to CENet;

[0020] Use SENet to calculate the importance score of each feature channel and apply it as a weight to the subsequent depthwise separable convolution to further enhance the feature information;

[0021] Finally, the pixel-level visible light enhancement image I is obtained through the Tanh activation function. map .

[0022] Furthermore, in order to obtain the final roughly enhanced visible light image, a light enhancement curve is used to adjust the brightness of the visible light image, and the visible light enhancement image I is estimated by the light enhancement curve. map , and then reused in different iteration stages to obtain a coarsely enhanced visible light image. This process can be expressed as

[0023] LE n (I vis )=LE n-1 (I vis )+Imap *LE n-1 (I vis )*(1-LE n-1 (I vis ))

[0024] Among them: LE n (·) represents the enhancement curve process, which is monotonic to maintain the contrast and natural color of the original image.

[0025] Furthermore, the SENet includes a global average pooling layer, a fully connected layer, a LeakyReLU layer, another fully connected layer and a Sigmoid layer.

[0026] Furthermore, the FDFNet in S2 uses frequency domain multiplication to replace spatial domain correlation operations, and the FDFNet includes an efficient frequency attention and a discriminative frequency feedforward network, so that the network can selectively extract the required frequency domain from infrared and coarse enhanced visible light images, the efficient frequency attention is EFA, and the discriminative frequency feedforward network is D3FN;

[0027] D3FN is used in the encoder to filter out the interference of useless information, and EFA is added in the decoder to form an asymmetric structure to extract more comprehensive features and further enhance image information;

[0028] The EFA and D3FN jointly construct a U-shaped feature extractor, using the D3FN network to filter out useless information, and using EFA to globally extract features to enhance overall visual perception;

[0029] The FDFNet also captures the attention map of the visible light image through SENet and multiplies it with the infrared feature map element by element, injecting the significant information captured by the visible light into the infrared features, enhancing the feature learning ability of U-Net.

[0030] Furthermore, the EEA is to effectively estimate the attention map by multiplication of elements in the frequency domain, and the construction of the EEA includes at least the following steps:

[0031] First, use 1×1 pointwise convolution and 3×3 depthwise convolution to obtain Q, K, and V;

[0032] Use Fast Fourier Transform to convert Q and K into the frequency domain and estimate the correlation between Q and K:

[0033]

[0034] Where: FFT(·) means fast Fourier transform, IFFT(·) means inverse fast Fourier transform, represents element-wise multiplication;

[0035] In addition, considering that deep convolution shares the weights of spatial dimensions while focusing on channels, it provides complementary cues for attention and enhances the exploration of contextual information. Therefore, in order to enhance the modeling ability of attention in the channel dimension, channel interaction is designed to generate channel dynamic weights;

[0036] The channel interaction includes a global average pooling layer, two consecutive 1×1 convolutional layers with unnormalization and an activation function in the middle, and finally Sigmoid is used to generate attention in the channel dimension, and then interacts with V, see the following formula:

[0037]

[0038] Where: D represents the value generated by the deep convolution branch, C inter represents channel interaction;

[0039] The activation function is GELU;

[0040] Finally, the newly generated V is used to estimate the aggregate features, and the formula is as follows:

[0041] V att =L(M)V

[0042] Where L(·) represents the normalization of M;

[0043] Then, spatial interactions are designed, which allow spatial relationships to flow out from the attention branch and provide strong spatial information to the deep convolution branch;

[0044] The spatial interaction consists of two 1×1 convolutions with Batch Normalization and GELU between the two convolution kernels;

[0045] Through two convolution kernels, V att The number of channels is reduced to 1, and the spatial attention map is generated through the Sigmoid layer, and then multiplied with the deep convolution branch, see the following formula:

[0046]

[0047] Finally, the present invention connects the attention branch and the depthwise convolution branch and outputs enhanced features through 1×1 convolution.

[0048] X att =X+Conv(concat(V att ,D))

[0049] Among them: concat means concatenating two feature maps in the channel dimension, Conv means convolution operation, and the final output X att It is the enhanced feature map.

[0050] Furthermore, the D3FN is used to solve the problem that not all frequency information is useful in the coarsely enhanced visible light image. By introducing a learnable quantization matrix W, D3FN learns appropriate parameters from the dataset itself to filter the frequency domain representation of the feature map. The specific process is shown in the following formula:

[0051] X=Conv(L(X att ))

[0052] X = FFT(P(X))

[0053] x=IFFT(WX)

[0054] X1,X2=DWConv(P -1 (X)

[0055]

[0056] X out =X att +Conv(X)

[0057] Where P(·) and P -1 (·) represent the patch expansion and folding operations, respectively, and G(·) represents the GELU activation function.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention constructs a coarse enhancement network (CENet), which is composed of a U-Net network composed of 8 deeply separable convolution blocks. The nighttime visible light feature extraction capability is improved through deep convolution and point-by-point convolution, and the channel importance score is calculated in combination with SENet, and finally a pixel-level enhancement map is generated using the Tanh activation function. Then, the enhanced visible light feature map and the infrared light image are respectively sent to the frequency domain fusion network (FDFNet). In FDFNet, the present invention proposes to construct a U-shaped feature extractor based on the EFA and D3FN modules in the frequency domain, and injects the visible light significant information into the infrared feature map through SENet, thereby enhancing the feature learning ability of the U-Net network. Finally, the images passing through FDFNet are spliced, and the final fused image is obtained by image reconstruction. Experimental results show that the method proposed by the present invention is superior to the existing most advanced methods on the LLVIP dataset, demonstrating the potential for further improving the accuracy of fused images in the future, while helping to promote the progress of advanced visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0061] Figure 1 It is the overall framework diagram of the present invention;

[0062] Figure 2 It is a framework diagram of the coarse enhancement network of the present invention;

[0063] Figure 3 It is a framework diagram of the frequency domain fusion network of the present invention;

[0064] Figure 4 It is a composite schematic diagram of the present invention, wherein Figure 4 (a) is the efficient frequency attention of the present invention, Figure 4 (b) is the frequency discrimination feedforward network, Figure 4 (c) channel interaction and space interaction of the present invention;

[0065] Figure 5 Schematic diagram of qualitative comparison between FDEFusion of the present invention and six most advanced methods on the LLVIP dataset. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0067] The purpose of image fusion is to make the fused image maintain the advantages of different images. However, the existing fusion algorithm only considers the image fusion under normal illumination, ignoring the illumination attenuation of the visible image at night, resulting in the loss of fused image information. The previous image fusion algorithms and image fusion methods mainly focus on the spatial domain, ignoring the value of the image frequency domain. In order to solve these problems, the following embodiments are proposed.

[0068] See also Figure 1-Figure 4 , a frequency domain enhanced fusion network method for nighttime infrared and visible light images, comprising:

[0069] A coarse enhancement network (CENet) was constructed. CENet consists of a U-Net network composed of 8 depth-wise separable convolutional blocks. The feature extraction capability of visible light at night is improved through deep convolution and point-by-point convolution. SENet is combined with SENet to calculate the channel importance score, and finally the Tanh activation function is used to generate a pixel-level enhancement map.

[0070] Then, the enhanced visible light feature map and infrared light image are sent to the frequency domain fusion network (FDFNet) respectively.

[0071] In FDFNet, the present invention proposes to construct a U-shaped feature extractor based on the EFA and D3FN modules in the frequency domain, and injects visible light significant information into the infrared feature map through SENet, thereby enhancing the feature learning ability of the U-Net network.

[0072] Finally, the images passed through FDFNet are spliced ​​and the final fused image is obtained through image reconstruction.

[0073] Among them, the coarse enhancement network (CENet) and the frequency domain fusion network (FDFNet) are the key technologies of the present invention.

[0074] Coarse Enhanced Network (CENet)

[0075] Overall operation flow of the coarse enhancement network (such as Figure 2 As shown): CENet is a U-Net network composed of 8 depth-wise separable convolution blocks, each of which is composed of a depth-wise convolution with a stride of 1 and a size of 3×3 and a point-wise convolution with a stride of 1 and a size of 1×1, and uses LeakyReLU as an activation function. The purpose of using depth-wise separable convolution is to combine and fuse features through depth-wise convolution and point-wise convolution, improve the feature learning ability of the coarse enhancement network, and extract more nighttime visible light image features. In addition, the present invention also uses SENet to calculate the importance score of each feature channel, and applies it as a weight to subsequent depth-wise separable convolutions to further enhance the feature information. SENet includes a global average pooling layer, a fully connected layer, a LeakyReLU layer, another fully connected layer, and a Sigmoid layer. Finally, the Tanh activation function is used to obtain the pixel-level visible light enhancement image I map .

[0076] Frequency Domain Fusion Network (FDFNet)

[0077] The present invention uses frequency domain multiplication to replace the spatial domain correlation operation, and proposes a new efficient frequency attention (EFA) and discriminative frequency feedforward network (D3FN), such as Figure 4 As shown. The network can selectively extract the required frequency domain in infrared light and coarsely enhanced visible light images. EFA and D3FN jointly construct a U-shaped feature extractor. During the feature extraction process, the present invention also uses SENet to capture the attention map of the visible light image and multiply it element by element with the infrared light feature graph, so that it can inject the significant information captured in the visible light into the infrared features, thereby enhancing the feature learning capability of the U-net network.

[0078] Efficient frequency attention (such as Figure 4 (a) shows that after the feature map input is standardized, features are extracted through multiple convolutional layers. These features are then transformed into the frequency domain through the fast Fourier transform (FFT), and the information flow between channels is enhanced through the channel interaction module. After that, the features are restored to the spatial domain through the inverse fast Fourier transform (IFFT), and the spatial information is further integrated through the spatial interaction module. Finally, the residuals are added through 1×1 convolution and jump connection to achieve efficient feature integration. The high-frequency attention method effectively captures multi-scale features and important spatial details.

[0079] Discriminant frequency feedforward network (such as Figure 4 (b)): Figure 4 (b) shows a feature extraction and reconstruction process based on frequency domain operations: after the input is normalized and 1×1 convolved, the image is divided into blocks through the Patch Unfolding operation, and then the Fast Fourier Transform (FFT) is performed and multiplied with the learned weight w to selectively enhance specific frequency information. After being restored to the spatial domain through the Inverse Fast Fourier Transform (IFFT), the features are reconstructed through Patch Folding, and then further fused through deep convolution and 1×1 convolution to finally obtain the enhanced output. This structure improves the precision of feature extraction through frequency domain modulation and spatial domain integration.

[0080] Channel interaction and spatial interaction (such as Figure 4 (c)): Figure 4 In (c), channel interaction extracts global features of each channel through global average pooling (GAP) operation, and passes the features to 1×1 convolution and activation function layers (including BatchNormalization, GELU, Sigmoid), thereby generating importance weights for each channel to enhance the focus on key channels. This process ensures the priority allocation of features in the channel dimension, allowing the network to pay more attention to feature channels with high weights. Spatial interaction processes features in the spatial dimension, assigning weights to different spatial locations through 1×1 convolution and activation functions (BatchNormalization, GELU, Sigmoid), highlighting the features of salient areas. This operation generates an attention map in the spatial dimension to strengthen the focus on key information areas in the image, while suppressing irrelevant background areas and improving the network's spatial resolution.

[0081] Further evaluation of the technical effects of the above embodiments:

[0082] Experimental Configuration

[0083] In order to fully evaluate the proposed algorithm, the present invention conducts extensive quantitative and qualitative evaluations on the LLVIP dataset, where LLVIP is a dataset of paired infrared and color visible light images in low-light environments, containing 16,836 pairs of images. In order to verify the superior performance of the method, the present invention selects 50 classic infrared and visible light images from LLVIP for analysis. The results of the present invention are compared with six state-of-the-art SOTA fusion algorithms, including a GAN-based method FusionGAN, four CNN-based methods, namely SDNet, U2Fusion PIAFusion, SeAFusion, and a night infrared and visible light image fusion method DIVFusion. The above-mentioned image fusion algorithms and low-light enhancement algorithms are publicly available, and the present invention sets the parameters completely according to the original paper.

[0084] For quantitative evaluation, the present invention selects five indicators, including standard deviation (SD), visual information fidelity (VIF), average gradient (AG), information entropy (EN), and spatial frequency (SF), to objectively evaluate the fusion performance. SD indicates the distribution and contrast of the fused image. VIF measures information fidelity from the perspective of human visual perception. AG reflects the richness of image texture information. EN measures image information through pixel-level grayscale distribution. SF quantifies the grayscale change rate in the image. In addition, the larger the values ​​of these five indicators, the better the fusion performance.

[0085] Experimental details

[0086] Since the method of the present invention is a two-stage model, the present invention selects 240 pairs of night infrared and visible light images from the LLVIP data set to train the enhancement and fusion networks of the present invention respectively. In order to obtain more training data, the present invention crops these images into a large number of image blocks and normalizes them to [0,1] before sending them into the network. First, the present invention trains the coarse enhancement network, with the batch size set to 16 and the patch size set to 64×64. Then, the coarsely enhanced visible light image and infrared image are used as inputs of the frequency domain fusion network. The batch size of FDFNet is 4, and the patch size is set to 64×64. The parameters in all networks are updated by the Adam optimizer, and the weight decay is equal to 0.00001. The initial learning rate is set to 0.0001, and then decays with 20 epochs until it drops to 10-7 in the 60th epoch. In addition, the experiments were trained on NVIDIA Tesla V100 using the pytorch framework.

[0087] Comparative experiment

[0088] In order to fully evaluate the algorithm of the present invention, the present invention first compares the proposed FDEFusion with the other six algorithms on the LLVIP dataset.

[0089] Qualitative results: In order to visually observe the fusion performance of different algorithms on the LLVIP dataset, the present invention selects image pairs #010064 and #060193 for visualization, as shown in Figure 5 As shown. Figure 5 As shown in the red and green boxes, due to the uneven exposure and spectral pollution of the night scene, although various algorithms have achieved certain fusion effects, the fusion results of FusionGAN and SDNet have missed a lot of scene information, resulting in blurred vision. Although U2Fusion, PIAFusion and SeAFusion contain more texture information, they often make the overall scene submerged in the dark environment. Only the fusion results of the present invention and DIVFusion can obtain rich texture and scene information in a dark environment. However, the fusion results of DIVFusion generally tend to be white, introducing noise interference and color distortion, resulting in poor visibility. Only the method of the present invention retains the prominent targets of the infrared image and restores the potential information in the visible light image, so that the fused image achieves the best visual effect. The present invention uses the green and red boxes in images #010064 and #060193 to enlarge the dark area to illustrate this phenomenon.

[0090] Quantitative results: In order to verify the effectiveness of the method of the present invention, 50 images were randomly selected from LLVIP for further quantitative comparison. In terms of quantitative evaluation, the present invention selected five indicators, SD, VIF, AG, EN and SF, to objectively evaluate the fusion effect. The quantitative results are shown in Table 1.

[0091] Table 1 provides a quantitative comparison of five metrics on 50 images from the LLVIP dataset.

[0092]

[0093] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A frequency domain enhancement fusion network method for nighttime infrared and visible light images, characterized in that: At least the following steps are included: S1: A coarse enhancement network is constructed, the coarse enhancement network is CENet, the CENet cooperates with SENet and Tanh activation function to generate a pixel-level enhancement map, the pixel-level enhancement map at least includes a pixel-level visible light enhancement map, and the SENet is a squeeze and excitation network; S2: construct a frequency domain fusion network, and send the enhanced pixel-level visible light enhancement image and infrared light image to the frequency domain fusion network respectively, and the frequency domain fusion network is FDFNet; S3: Finally, the images passed through FDFNet are stitched together, and the fused image is reconstructed through the reconstruction network to obtain the final fused image.

2. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 1 is characterized in that: The CENet is a U-Net network composed of 8 depthwise separable convolution blocks. Each depthwise separable convolution is composed of a depthwise convolution with a stride of 1 and a size of 3×3 and a pointwise convolution with a stride of 1 and a size of 1×1, and uses LeakyReLU as an activation function. The purpose of using depthwise separable convolution is to combine and fuse features through depthwise convolution and pointwise convolution, improve the feature learning ability of the coarse enhancement network, and extract more nighttime visible light image features.

3. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 1 is characterized in that: The S2 at least comprises the following steps: Given a pair of strictly registered infrared images I ir and visible light image I vis ; First, the visible light image I vis Send to CENet; Use SENet to calculate the importance score of each feature channel and apply it as a weight to subsequent depthwise separable convolution to further enhance feature information; Finally, the pixel-level visible light enhancement image I is obtained through the Tanh activation function. map .

4. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 3 is characterized in that: In order to obtain the final coarse enhanced visible light image, a light enhancement curve is used to adjust the brightness of the visible light image, and the visible light enhancement image I is estimated by the light enhancement curve. map , and then repeatedly used in different iterative stages to obtain a coarsely enhanced visible light image. This process can be expressed as: THE n (I vis )=NO n-1 (I vis )+I map *THE n-1 (I vis )*(1-LE n-1 (I vis )) Among them: LE n (·) represents the enhancement curve process, which is monotonic to maintain the contrast and natural color of the original image.

5. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 3 is characterized by: The SENet includes a global average pooling layer, a fully connected layer, a LeakyReLU layer, another fully connected layer and a Sigmoid layer.

6. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 1 is characterized by: The FDFNet in S2 uses frequency domain multiplication to replace spatial domain correlation operations. The FDFNet includes an efficient frequency attention and a discriminative frequency feedforward network, which enables the network to selectively extract the required frequency domain from infrared and coarse enhanced visible light images. The efficient frequency attention is EFA, and the discriminative frequency feedforward network is D3FN. D3FN is used in the encoder to filter out the interference of useless information, and EFA is added in the decoder to form an asymmetric structure to extract more comprehensive features and further enhance image information; The EFA and D3FN jointly construct a U-shaped feature extractor, using the D3FN network to filter out useless information, and using EFA to globally extract features to enhance overall visual perception; The FDFNet also captures the attention map of the visible light image through SENet and multiplies it with the infrared feature map element by element, injecting the significant information captured by the visible light into the infrared features, enhancing the feature learning ability of U-Net.

7. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 6 is characterized by: The EEA is to effectively estimate the attention map by multiplying the elements in the frequency domain. The construction of the EEA includes at least the following steps: First, use 1×1 pointwise convolution and 3×3 depthwise convolution to obtain Q, K, and V; Use Fast Fourier Transform to convert Q and K into the frequency domain and estimate the correlation between Q and K: Where: FFT(·) represents fast Fourier transform, IFFT(·) represents inverse fast Fourier transform, represents element-wise multiplication; In addition, considering that deep convolution shares the weights of spatial dimensions while focusing on channels, it provides complementary cues for attention and enhances the exploration of contextual information. Therefore, in order to enhance the modeling ability of attention in the channel dimension, channel interaction is designed to generate channel dynamic weights; The channel interaction includes a global average pooling layer, two consecutive 1×1 convolutional layers with unnormalization and an activation function in the middle, and finally Sigmoid is used to generate attention in the channel dimension, and then interacts with V, see the following formula: Where: D represents the value generated by the deep convolution branch, C inter represents channel interaction; The activation function is GELU; Finally, the newly generated V is used to estimate the aggregate features, and the formula is as follows: V att =L(M)V Where L(·) represents the normalization of M; Then, spatial interactions are designed, which allow spatial relationships to flow out from the attention branch and provide strong spatial information to the deep convolution branch; The spatial interaction consists of two 1×1 convolutions with Batch Normalization and GELU between the two convolution kernels; Through two convolution kernels, V att The number of channels is reduced to 1, and the spatial attention map is generated through the Sigmoid layer, and then multiplied with the deep convolution branch, see the following formula: Finally, the present invention connects the attention branch and the depthwise convolution branch and outputs enhanced features through 1×1 convolution. X att =X+Conv(concat(V att ,D)) Among them: concat means concatenating two feature maps in the channel dimension, Conv means convolution operation, and the final output X att It is the enhanced feature map.

8. The frequency domain enhancement fusion network method for nighttime infrared and visible light images according to claim 7 is characterized in that: The D3FN is used to solve the problem of noise in coarsely enhanced visible light images and that not all frequency information is useful. By introducing a learnable quantization matrix W, D3FN learns appropriate parameters from the dataset itself to filter the frequency domain representation of the feature map. The specific process is shown in the following formula: X=Conv(L(X att )) X = FFT(P(X)) X=IFFT(WX) X1,X2=DWConv(P -1 (X)) X out =X att +Conv(X) Where P(·) and P -1 (·) represent the patch expansion and folding operations, respectively, and G(·) represents the GELU activation function.

Images