An adaptive feature extraction method for infrared and visible light image fusion

Abstract

The present invention relates to the field of computer vision technology, and specifically to an infrared and visible light image fusion method with adaptive feature extraction, comprising: using a single-modal spatial adaptive feature mixing module to perform multi-scale spatial adaptive feature extraction on two modal images; inputting the extracted features into a dual-branch feature extraction module to obtain low-frequency global features and high-frequency local features; then splicing the dual-modal features, and reconstructing the fused image using a dual-modal spatial adaptive feature fusion module. In order to solve the imbalance problem between modality-specific features and shared features, decomposition loss is introduced into the loss function to further optimize the fusion model. The present invention adopts a dual-branch feature extraction module to establish local and global contextual dependencies, and extracts potential features of the modalities at multiple scales to generate a fused image with rich details and global consistency, which can perform well even in harsh environments and improve the effects of downstream visual tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to an infrared and visible light image fusion method for adaptive feature extraction. Background Art

[0002] Image fusion is an image enhancement technology that aims to generate a fused image with robustness and rich semantic information based on data from different modalities, so as to facilitate the implementation and decision-making of downstream tasks. The key to image fusion lies in effective feature extraction and appropriate fusion strategies to ensure that rich information is obtained from the source images to generate a fused image without introducing artifacts. At present, image fusion is roughly divided into multimodal image fusion, digital image fusion and remote sensing image fusion. In multimodal image fusion, the fusion of infrared and visible light images has significant advantages. Visible light images have the characteristics of high resolution, rich colors, and natural perception, which are suitable for object recognition and scene perception. Since the human eye is sensitive to visible light, such images are easier to understand and interpret. Infrared images can capture the thermal radiation of objects and perform well in low light or no light conditions. They are suitable for night monitoring and target detection in special weather conditions. Infrared images also have good ability to penetrate atmospheric interference such as fog and smoke, and are more sensitive to stealth detection technology. By comprehensively utilizing the advantages of visible light and infrared images, the fusion of their information can enhance the richness of the image and make the system more comprehensive and effective in different environments and applications. This technology has broad application prospects in military, surveillance and medical fields.

[0003] Infrared and visible light image fusion methods can be roughly divided into two categories: traditional fusion methods and fusion models based on deep learning. Traditional fusion methods are mainly based on multi-scale transformation, sparse representation, image pyramid fusion, transform domain methods, pixel-level fusion and hybrid models. These methods are usually based on empirical and mathematically designed transformations, which can complete the task of image fusion to a certain extent. However, they have certain limitations in processing complex scenes and extracting high-level features. It is usually impractical to rely solely on manually designed fusion strategies. With the rise of deep learning technology, major changes have taken place in the field of image fusion. Deep learning methods, through end-to-end learning, can automatically learn abstract features and complex relationships in images, so as to better adapt to the needs of different tasks.

[0004] At present, deep learning-based fusion models can be divided into three categories: CNN-based fusion models, GAN-based fusion models, and transformer-based fusion models. Although these methods can produce better fused images, they still have problems such as information loss during the fusion process and artifacts in the fused image, which is disastrous for subsequent tasks. On the one hand, most of the existing deep learning-based fusion methods are based on convolutional neural networks, which tend to pay more attention to local features during the fusion process and ignore global contextual dependencies, resulting in the loss of detailed texture information in the fused image. At the same time, they face the challenges of cross-modal feature modeling and decomposition, and need to solve the imbalance between modality-specific features and modality-shared features. On the other hand, most of the existing image fusion methods rely on stacking network modules or using self-attention mechanisms to improve the performance of the fusion model, which leads to a large number of parameters and increased computational complexity, posing challenges to scenarios with limited resources or scarce domain-specific data.

[0005] The existing Chinese patent document CN202410690220.5 discloses a dual-branch infrared and visible light image fusion method and system, including: obtaining an infrared image and a visible light image to be fused; inputting the infrared image and the visible light image to be fused into an image fusion network for image fusion to obtain a fused image; wherein the image fusion network includes an encoder, a fusion module and a decoder, and the encoder includes a shallow feature extraction module, a local feature extraction module and a global feature extraction module; after the shallow features of the infrared image and the visible light image to be fused are respectively extracted by the shallow feature extraction module, the local features and the global features are extracted by the local feature extraction module and the global feature extraction module, and the local features and the global features are fused by the fusion module and the fused image is output through the decoder. The fused image generated by this method is insufficient in details and contrast, and due to the lack of multi-scale feature extraction, it fails to capture information at different levels in the image. In addition, dual-branch feature extraction increases the computational complexity. To solve these problems, the present invention introduces a spatially adaptive feature mixing module to extract representative features in a multi-scale manner, and combines the focused linear attention mechanism and structural reparameterization to reduce the amount of computational parameters, thereby speeding up the inference speed and generating high-quality fused images. Summary of the invention

[0006] The purpose of the present invention is to provide an infrared and visible light image fusion method with adaptive feature extraction. The present invention can effectively fuse low-frequency global features and high-frequency local features in infrared and visible light images, making low-frequency features correlated and high-frequency features uncorrelated, and extracting potential features between modalities at multiple scales to achieve end-to-end fused image generation. In order to solve the imbalance problem between modality-specific and shared features, decomposition loss is introduced into the loss function. The generated fused image is not only rich in details, globally consistent, high in contrast, and comprehensive in information integration, but also has excellent robustness and generalization ability, and is widely used, thereby promoting the realization of advanced visual tasks.

[0007] In order to solve the problems of insufficient details and contrast, failure to capture different levels of information in the image, and high computational complexity in the comparative documents, the present invention introduces a spatially adaptive feature mixing module to extract representative features in a multi-scale manner, and combines the focused linear attention mechanism and structural reparameterization to reduce the model complexity and improve the inference speed.

[0008] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0009] An infrared and visible light image fusion method based on adaptive feature extraction comprises the following steps:

[0010] S1: First, the acquired infrared and visible light image datasets are preprocessed, and then the preprocessed datasets are input into the fusion model for training;

[0011] S2: The shallow features extracted by convolution operation of the infrared and visible light image pairs in S1 are input into the single-modal spatial adaptive feature mixing module to perform multi-scale spatial adaptive feature extraction;

[0012] S3: The feature map extracted by S2 is input into the dual-branch feature extraction module, and the low-frequency texture features are extracted by the FTB module, and the high-frequency local detail features are extracted by the VB module;

[0013] S4: perform element-wise addition of the low-frequency texture features and high-frequency local features of the two modalities, perform channel concatenation of the features of the two modalities, and then use the dual-modal spatial adaptive feature fusion module to reconstruct the fused image;

[0014] S5: Calculate the loss value of the fused image obtained in S4 and the original infrared and visible light images, so as to optimize the fusion model;

[0015] S6: Save the trained fusion model;

[0016] S7: Input the infrared and visible light images to be fused into the fusion model to generate a fused image.

[0017] Furthermore, the single-mode spatial adaptive feature mixing module in step S2 specifically includes:

[0018] Considering that the goal of the multimodal image fusion task is to extract enough semantic information from the source images, in order to generate semantically rich fused images, the source images are first dimensionally expanded. Then, they are input into the unimodal spatial adaptive feature mixing module respectively to learn long-range dependencies from multi-scale features. This process helps to ensure that the fused image contains key semantic information from the source images. The spatial adaptive fusion module formula is expressed as follows:

[0019]

[0020]

[0021]

[0022]

[0023] in represents the features input to the spatial adaptive feature fusion module, LN represents the layer normalization operation, S represents the channel separation operation, and f' i represents the output feature of the middle layer, Dw- represents a 3×3 depth convolution with input channel C / 4 and output channel C / 4, U s represents upsampling to restore the original resolution size of the image, D s represents the maximum pooling with different downsampling rates, C represents the concatenation operation along the channel dimension, ⊙ represents the element-wise multiplication operation, GELU represents the GELU activation function, Represents the output features of the intermediate feature layer.

[0024] In order to further improve the performance and effectiveness of the model, a channel mixing module is also included. This module integrates spatial context information and enhances channel feature representation. The channel mixing module formula is as follows:

[0025]

[0026] in represents a 1×1 convolution of input channel 2C and output channel C, Represents the output features after the unimodal spatial adaptive feature mixing module.

[0027] Furthermore, the dual-branch feature extraction module in step S3 specifically includes:

[0028] The feature map extracted by the single-modal spatial adaptive feature hybrid module is input into the dual-branch feature extraction module. The FTB module is used to guide the network's attention to the long-range low-frequency dependencies by focusing linear attention. At the same time, the local high-frequency features of the two modalities are extracted through the VAB module with a minimalist structure. The FTB module formula is expressed as follows:

[0029]

[0030] Among them, F SA is the output of the middle layer, is the input of FTB, which contains independent representations of layer normalization LN and multi-head self-attention MSA.

[0031]

[0032] in is the output of FTB. The MLP consists of two layer normalization (LN) layers with a GELU nonlinear activation function between them.

[0033] The design of the VAB module uses structural reparameterization to train two 1×1 convolutional layers and activation functions in the initial training stage. The concept of structural reparameterization is used to merge the convolutional layer and the batch normalization layer, so that the activation function between the two convolutional layers gradually degenerates into an identity mapping during the training process. After training, the two convolutions are fused into one, thereby speeding up the inference speed of the model.

[0034] Furthermore, the dual-modal spatial adaptive feature fusion module in step S4 specifically includes:

[0035] The low-frequency texture features and high-frequency detail features obtained by the dual-branch feature extraction module are input into the dual-modal spatial adaptive feature fusion module, which performs spatial adaptive extraction and channel mixing on the features of the two modalities to reconstruct the fused image.

[0036] Furthermore, the total loss during the fusion model training process is defined as follows:

[0037]

[0038] in and denote the decomposition loss, gradient loss and structural similarity loss respectively. α and β are tuning parameters.

[0039] In order to solve the imbalance problem between modality-specific features and modality-shared features, the fused image generated by the guidance network realizes the correlation of low-frequency features and the irrelevance of high-frequency features. The image decomposition loss function is defined as follows:

[0040]

[0041] in, and are the background and detail feature maps of the visible light image, respectively. and are the background and detail feature maps of the infrared image, respectively. Φ is the tanh function, which is used to constrain the value to the interval (-1, 1), and γ is a hyperparameter.

[0042] In order to enhance the edge and detail information of the image, the gradient loss is introduced as follows:

[0043]

[0044] Among them I ir ,I vi ,I fu They represent infrared image, visible light image and fused image respectively, H and W represent the height and width of the image. MAX{,} represents the maximum selection of elements, and Δ represents the gradient operator.

[0045] To ensure that the generated image is similar to the original image in overall structure and retain the global consistency and structural integrity of low-frequency features, the structural similarity loss is defined as:

[0046]

[0047] Where SSIM stands for the structural similarity index metric, which evaluates distortion based on the similarity of brightness, contrast, and structure.

[0048] Beneficial effects of the present invention:

[0049] Compared with the prior art, the present invention mainly solves the problem that most current image fusion methods rely on convolutional neural networks for cross-modal local feature extraction, fail to fully utilize remote context information, and result in limited performance in complex scenarios. At the same time, the present invention introduces decomposition loss to solve the imbalance problem between modality-specific features and shared features, and realizes the processing of low-frequency feature correlation and high-frequency feature irrelevance. On the other hand, the dual-branch feature extraction module fully mines the complementary information of the source image, and dynamically extracts representative features from a multi-scale perspective through a multi-scale spatial adaptive feature fusion module, so that we can more deeply mine potential features when generating fused images, provide a more comprehensive description of the imaging scene, and have excellent robustness and generalization capabilities, highlighting its potential in advanced visual applications.

[0050] The present invention effectively ensures the separation and fusion of low-frequency and high-frequency features, thereby achieving efficient integration of information. In traditional image fusion methods, low-frequency features (such as the overall structure and contour of the image) and high-frequency features (such as details and edge information) are often unable to be effectively processed separately, resulting in information loss or artifacts. The present invention uses a dual-branch feature extraction module, respectively using the FTB module to focus on the extraction of low-frequency texture features, and using the VAB module to extract high-frequency local detail features. In order to solve the problems of increased computational complexity and slow reasoning speed brought by the dual-branch feature extraction module, the FTB module introduces a focused linear attention mechanism, which reduces the computational complexity while enabling the network to adaptively focus on remote low-frequency dependencies, thereby ensuring that important global features are effectively retained. At the same time, on the basis of structural reparameterization, the VAB module gradually transforms complex structures into efficient identity mappings through the initial multi-layer convolutional structure of the training, greatly improving the extraction efficiency of high-frequency features. Finally, the fused image not only retains rich details, but also shows high consistency and structural integrity globally, significantly improving the usability and visual quality of the image.

[0051] The multi-scale feature learning ability of the system of the present invention significantly enhances the richness and accuracy of the fusion effect. In the multimodal image fusion task, it is very important to efficiently extract and fuse multi-scale features. The present invention designs a dual-modal spatial adaptive feature fusion module, which can effectively learn feature information at different scales by performing layer normalization, channel separation and adaptive maximum pooling on the input features. The module first performs deep convolution on the input to extract features, then generates multi-scale features through adaptive maximum pooling, and finally restores to the original resolution through upsampling. At the same time, the channel mixing module is used to integrate spatial context information and enhance the channel feature representation. This method ensures that the key semantic information of the source image is retained during the feature fusion process, and the generated fused image shows good performance at different scales. This integration of multi-scale features not only helps to improve the accuracy of target detection and recognition, but also plays a positive role in promoting the richness and semantic integrity of the image.

[0052] The loss function design mechanism introduced in the present invention targets the balance problem between modality-specific features and shared features, and enhances the generalization ability and robustness of the model. Specifically, a composite loss function consisting of decomposition loss, gradient loss and structural similarity loss is designed during the model training process. The goal of introducing decomposition loss is to guide the fused image generated by the network to maintain the correlation of low-frequency features and the independence between high-frequency features, and prevent information crosstalk. Gradient loss emphasizes the retention of image edges and details, ensuring that the generated image has sufficient detail performance while retaining structural integrity. In addition, structural similarity loss further ensures the overall high-quality performance of the generated image by evaluating the consistency of different images in brightness, contrast and structure. This comprehensive loss function design effectively guides the model to adapt to image features under different conditions, improves the model's adaptability to new data, and makes the final generated fused image visually more robust and overall consistent.

[0053] The present invention realizes an efficient end-to-end training process and simplifies the development and implementation of the model. In the prior art, it is often necessary to rely on a large amount of labeled data for supervised learning, while the present invention significantly reduces the dependence on labeled data through an unsupervised end-to-end training mechanism. The model first performs adaptive feature extraction on inputs of different modalities, and realizes the reconstruction and synthesis of dynamic features in the feature fusion module. This process gradually improves the fusion ability of the model through self-learning and optimization of the algorithm, reducing the need for manual parameter adjustment. In addition, the introduction of focused linear attention mechanism and structural reparameterization greatly reduces the complexity, making the model more efficient in practical applications. The design of this model structure not only improves the learning efficiency of the model, but also facilitates actual deployment, so that the method can be widely used in multiple fields such as military surveillance, medical imaging and intelligent monitoring, and meet the needs of efficient image fusion in different scenarios.

[0054] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0056] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0057] Figure 2 It is a schematic diagram of the overall network model of the fusion model described in the present invention;

[0058] Figure 3 Schematic diagram of the network structure of the spatial adaptive feature fusion module of the present invention;

[0059] Figure 4 It is a schematic diagram of the network structure of the FTB module of the present invention;

[0060] Figure 5 It is a schematic diagram comparing fused images generated by the method of the present invention and other methods. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Example 1

[0063] The infrared and visible light image fusion method based on dual-branch spatial adaptive feature extraction described in this embodiment has the ability to efficiently fuse infrared and visible light images, and includes the following steps:

[0064] Considering that the goal of the multimodal image fusion task is to extract enough semantic information from the source images, in order to generate semantically rich fused images, the source images are first dimensionally expanded. Then, they are input into the single-modal spatial adaptive fusion module, such as Figure 3 As shown, long-range dependencies are learned from multi-scale features. This process helps ensure that the fused image contains key semantic information from the source image. Specifically, the features of the two modalities obtained by the convolution operation are subjected to layer normalization operations to better adapt to the data distribution of different sensor sources. Channel segmentation is then performed to reduce complexity, while focusing the model on capturing specific information of each channel, thereby improving sensitivity and adaptability to different spectral data. One channel group directly undergoes 3×3 deep convolution to extract features, while other channel groups first generate multi-scale features through adaptive maximum pooling. Similarly, they undergo 3×3 deep convolution to extract deep features, which are then upsampled to the original resolution and concatenated in the channel dimension. These multi-scale features are then connected using 1×1 convolutions, and the attention feature map is generated through the GELU activation function. Subsequently, element-by-element multiplication is performed with the input features, and the input of the spatial adaptive feature fusion module is added to the output of the intermediate feature layer. The formula of the spatial adaptive fusion module is expressed as follows:

[0065]

[0066]

[0067]

[0068]

[0069] in represents the features input to the spatial adaptive feature fusion module, LN represents the layer normalization operation, S represents the channel separation operation, and f' i represents the output feature of the middle layer, Dw- represents a 3×3 depth convolution with input channel C / 4 and output channel C / 4, U s represents upsampling to restore the original resolution size of the image, D s represents the maximum pooling with different downsampling rates, C represents the concatenation operation along the channel dimension, ⊙ represents the element-wise multiplication operation, GELU represents the GELU activation function, Represents the output features of the intermediate feature layer.

[0070] In order to further improve the performance and effectiveness of the model, a channel mixing module is designed. This module integrates spatial context information and enhances channel feature representation. It uses 3×3 convolution to encode spatial local context and doubles the number of channels of the input feature for channel mixing. Subsequently, 1×1 convolution reduces the number of channels to the original input dimension, thereby reducing complexity. In this process, the GELU activation function is applied to the hidden layer for nonlinear mapping. The specific process is shown as follows:

[0071]

[0072] in represents a 1×1 convolution of input channel 2C and output channel C, Represents the output features after the unimodal spatial adaptive feature mixing module.

[0073] Specifically, the feature map extracted by the single-modal spatial adaptive feature mixing module is input into the dual-branch feature extraction module. Using the FTB module, Figure 4 As shown in Figure 1, the network's attention is directed to long-range low-frequency dependencies by focusing linear attention. At the same time, the local high-frequency features of the two modalities are extracted through the VAB module with a minimalist structure. The FTB module formula is expressed as follows:

[0074]

[0075] Among them, F SA is the output of the middle layer, is the input of FTB, which contains independent representations of layer normalization LN and multi-head self-attention MSA.

[0076]

[0077] in is the output of FTB, and the MLP includes two layer normalization (LN) layers, between which the GELU nonlinear activation function is used.

[0078] The design of the VAB module uses structural reparameterization to train two 1×1 convolutional layers and activation functions in the initial training stage. The concept of structural reparameterization is used to merge the convolutional layer and the batch normalization layer, so that the activation function between the two convolutional layers gradually degenerates into an identity mapping during the training process. After training, the two convolutions are fused into one, thereby speeding up the inference speed of the model.

[0079] Specifically, the low-frequency texture features and high-frequency detail features obtained by the dual-branch feature extraction module are input into the dual-modal spatial adaptive feature fusion module, and the features of the two modalities are spatially adaptively extracted and channel mixed to reconstruct the fused image.

[0080] Specifically, in order to achieve end-to-end unsupervised training and optimize the fusion model, a loss function consisting of decomposition loss, gradient loss and structural similarity loss is designed. The total loss during the fusion model training process is:

[0081]

[0082] in and denote the decomposition loss, gradient loss and structural similarity loss respectively. α and β are tuning parameters.

[0083] In order to solve the imbalance problem between modality-specific features and modality-shared features, the fused image generated by the guidance network realizes the correlation of low-frequency features and the irrelevance of high-frequency features. The image decomposition loss function is defined as:

[0084]

[0085] in, and are the background and detail feature maps of the visible light image, respectively. and are the background and detail feature maps of the infrared image, respectively. Φ is the tanh function, which is used to constrain the value to the interval (-1, 1), and γ is a hyperparameter.

[0086] In order to enhance the edge and detail information of the image, the gradient loss is introduced as follows:

[0087]

[0088] Among them I ir ,I vi ,I fu They represent infrared image, visible light image and fused image respectively, H and W represent the height and width of the image. MAX{,} represents the maximum selection of elements, and Δ represents the gradient operator.

[0089] To ensure that the generated image is similar to the original image in overall structure and retain the global consistency and structural integrity of low-frequency features, the structural similarity loss is defined as:

[0090]

[0091] Where SSIM stands for the structural similarity index metric, which evaluates distortion based on the similarity of brightness, contrast, and structure.

[0092] Example 2

[0093] In order to further verify the feasibility of this method, the method described in Example 1 is qualitatively and quantitatively compared with the seven most advanced methods, namely DenseFuse, DIDFuse, FusionGAN, U2Fusion, SwinFuse, DATFuse, and CDDFuse. The results are reproduced through the source code, and all comparative experiments are implemented under the requirements of the original paper. The model of this scheme is trained using the MSRS dataset, and experiments are conducted on the mainstream infrared and visible light image fusion dataset of TNO. In the training stage, a preprocessing operation is first performed, and the training samples are cropped into 128×128 pixel blocks with a step size of 200 to obtain enough images to train the model. The training batch is 120 rounds with a batch size of 8. The initial learning rate is set to 0.0001, and a multi-learning rate decay strategy is adopted. The Adam optimizer is used to adaptively adjust the learning rate.

[0094] Specific as Figure 5 As shown. The experimental results show that the scheme proposed in the present invention can generate a fused image that effectively retains low-frequency global features and high-frequency local features. This is due to the dual-branch feature extraction module fully mining the complementary information of the source image, while dynamically extracting representative feature representations through the spatial adaptive feature fusion module. This enables deeper mining of potential features when generating fused images, provides a more comprehensive description of the imaging scene, and generates fused images with rich details, global consistency, high contrast, and comprehensive information integration.

[0095] At the same time, quantitative comparative analysis experiments were conducted, and the fusion indicators commonly used in image fusion tasks were used for evaluation, including entropy (EN), standard deviation (SD), spatial frequency (SF), mutual information (MI), correlation sum of differences (SCD), visual information fidelity (VIF), edge information-based indicators (Qab / f) and structural similarity (SSIM). The above indicators are positive indicators, and the larger the value, the better the fusion image result, as shown in Table 1.

[0096]

[0097] The observation results show that the scheme proposed in the present invention achieves the best performance in most indicators of the TNO dataset. It is worth noting that the scheme of the present invention achieves the best results in both EN and SD, which means that it performs better in preserving image details and contrast. This also shows that the scheme can make the grayscale value distribution of the image more stable or closer to the ideal state. In addition, the best result of the SF metric shows that the scheme is better in preserving image texture, which also means that the fused image is clearer. This advantage is due to the spatial adaptive feature fusion module, which prompts the latent features to reshape the fused image while retaining complete information, ensuring the best preservation of details and texture. The best MI indicates that the fused image better retains the information content of the source image; a higher VIF value indicates that the fused image is visually closer to the source image; and a higher Q ab / f The value indicates that the edge details are clearer. In summary, the fused image generated by the proposed model shows significant advantages in most indicators, thus verifying the feasibility of the proposed dual-branch spatial adaptive feature extraction scheme.

[0098] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for fusion of infrared and visible light images with adaptive feature extraction, characterized in that: The following steps are involved: S1: First, the acquired infrared and visible light image datasets are preprocessed, and then the preprocessed datasets are input into the fusion model for training; S2: The shallow features extracted by convolution operation of the infrared and visible light image pairs in S1 are input into the single-modal spatial adaptive feature mixing module to perform multi-scale spatial adaptive feature extraction; S3: The feature map extracted by S2 is input into the dual-branch feature extraction module, and the low-frequency texture features are extracted by the FTB module, and the high-frequency local detail features are extracted by the VAB module; S4: perform element-wise addition of the low-frequency texture features and high-frequency local features of the two modalities, perform channel concatenation of the features of the two modalities, and then use the dual-modal spatial adaptive feature fusion module to reconstruct the fused image; S5: Calculate the loss value of the fused image obtained in S4 and the original infrared and visible light images, so as to optimize the fusion model; S6: Save the trained fusion model; S7: inputting the infrared and visible light images to be fused into the fusion model to generate a fused image; The single-mode spatial adaptive feature mixing module in step S2 specifically includes: The formula of the spatial adaptive fusion module is as follows: in represents the features input to the spatial adaptive feature fusion module, LN represents the layer normalization operation, S represents the channel separation operation, and f' i represents the output features of the intermediate layer, represents a 3×3 depth convolution with input channel C / 4 and output channel C / 4, U s represents upsampling to restore the original resolution size of the image, D s represents the maximum pooling with different downsampling rates, C represents the concatenation operation along the channel dimension, ⊙ represents the element-wise multiplication operation, GELU represents the GELU activation function, Represents the output features of the intermediate feature layer; It also includes a channel mixing module; this module integrates spatial context information and enhances channel feature representation; the channel mixing module formula is as follows: in represents a 1×1 convolution of input channel 2C and output channel C, Represents the output features after the single-modal spatial adaptive feature mixing module; The dual-branch feature extraction module in step S3 specifically includes: The FTB module is used to guide the network's attention to remote low-frequency dependencies by focusing linear attention. At the same time, the VAB module is used to extract local high-frequency features of the two modalities. The FTB module formula is as follows: where F SA is the output of the middle layer, is the input of FTB, which contains independent representations of layer normalization LN and multi-head self-attention MSA; in is the output of FTB; MLP includes two layer normalization (LN) layers, between which the GELU nonlinear activation function is used; The design of the VAB module uses structural reparameterization. In the initial training stage, two convolutional layers of size 1×1 are trained, and the activation function is trained. The concept of structural reparameterization is used to merge the convolutional layer and the batch normalization layer, so that the activation function between the two convolutional layers gradually degenerates into an identity mapping during the training process. After training, the two convolutions are fused into one, thereby speeding up the inference speed of the model.

2. The method for fusion of infrared and visible light images with adaptive feature extraction according to claim 1, characterized in that: The dual-modal spatial adaptive feature fusion module in step S4 specifically includes: inputting the low-frequency texture features and high-frequency detail features obtained by the dual-branch feature extraction module into the dual-modal spatial adaptive feature fusion module, performing spatial adaptive extraction and channel mixing on the features of the two modalities, and then reconstructing the fused image.

3. The method for fusion of infrared and visible light images with adaptive feature extraction as claimed in claim 1, characterized in that: The total loss during the fusion model training process is defined as follows: in and denote decomposition loss, gradient loss, and structural similarity loss, respectively; α and β are tuning parameters; The image decomposition loss function is defined as follows: in, and are the background and detail feature maps of the visible light image, respectively. and are the background and detail feature maps of the infrared image, respectively; Φ is the tanh function, which is used to constrain the value to the interval (-1, 1), and γ is a hyperparameter; The gradient loss is introduced as follows: Among them I ir ,I vi ,I fu Represent infrared image, visible light image and fused image respectively, H and W represent the height and width of the image; MAX{,} represents the maximum selection of elements, and Δ represents the gradient operator; The structural similarity loss is defined as: Where SSIM represents the structural similarity index metric, which evaluates distortion based on the similarity of brightness, contrast, and structure.

Images