A method and apparatus for infrared and visible light image fusion based on a lightweight model

Abstract

This invention provides a method and apparatus for fusing infrared and visible light images based on a lightweight model, belonging to the field of image information processing technology. It includes constructing a shallow feature extraction layer, a deep feature extraction and fusion layer, and a fused image reconstruction layer. The shallow feature extraction layer extracts shallow features from the infrared and visible light images, filtering out information irrelevant to image fusion. The deep feature extraction and fusion layer further extracts image features from the infrared and visible light images based on the image feature map obtained from the shallow feature extraction layer, capturing the feature information contained in both images, and fusing the obtained information features. The fused image reconstruction layer reconstructs the image based on the fused information features obtained from the deep feature extraction and fusion layer, ultimately obtaining a fused image containing information from both infrared and visible light images. This invention achieves more efficient fusion of information from infrared and visible light images.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, and particularly relates to an infrared and visible light image fusion method and apparatus based on a lightweight model. Background Technology

[0002] Image fusion technology is an enhancement method that integrates complementary information from multiple images to generate higher-quality images, significantly improving the accuracy of subsequent processing. Based on the different principles of imaging equipment, image fusion is mainly divided into three categories: multimodal image fusion, digital photographic image fusion, and remote sensing image fusion. Among them, Infrared and Visible Image Fusion (IVIF) is an important branch of multimodal fusion, attracting much attention due to its unique complementary characteristics. Infrared sensors generate images by detecting the thermal radiation of objects, highlighting prominent targets (such as pedestrians and vehicles) in complex environments, but typically lack detailed texture and are susceptible to noise interference. Visible light sensors, on the other hand, rely on ambient light reflection for imaging, providing rich texture and structural information, but their performance is limited in low light or extreme weather conditions (such as fog and heavy rain). By fusing data from both modalities, IVIF technology can generate enhanced images that combine thermal target prominence with detailed clarity, thus demonstrating significant application value in areas such as target detection, pedestrian recognition, and scene understanding. Existing technologies include a dual-branch Transformer-CNN framework, Cddfuse, for extracting and fusing global and local features to better reflect the different semantic information contained in high / low-frequency features. A unified measurement benchmark is also proposed to demonstrate how infrared and visible light fused images can facilitate downstream object detection and semantic segmentation tasks. Another existing technology designs a Residual Fusion Network (RFN) based on a residual structure to replace traditional fusion methods, proposing detail-preserving loss functions and feature enhancement loss functions for training the RFN. Model learning is achieved through an innovative two-stage training strategy: the first stage trains an autoencoder based on the novel concept of nested connections, and the second stage trains the RFN using the proposed loss function. A further existing technology proposes the UNF algorithm, which uses an encoder-decoder architecture with downsampling operations to learn the contextual features of an image. It fuses the downsampled multi-resolution scale with the full-resolution scale that preserves local details, and passes multi-scale contextual features across different stages through a cross-stage fusion module. Furthermore, an upsampling module is employed to address artifacts and blurring issues. In existing technologies, some researchers have proposed a general image fusion framework based on CNN, IFCNN, which has achieved good results on four datasets and has general applicability, but it has poor preservation of texture details. Other existing technologies have proposed a novel fusion network based on autoencoders, DIDFuse. This network first uses an encoder to decompose the image into background and detail feature maps with low-frequency and high-frequency information, respectively. A loss function is used to constrain the similarity of background feature maps and the difference in detail feature maps between the source images. Then, a fusion module merges the background and detail feature maps separately, and finally, a decoder reconstructs the fused image.In the existing technology, a real-time image fusion network called SeAFusion has been proposed, which establishes a connection with high-level vision tasks. This network cascades the image fusion module and the semantic segmentation module, and uses semantic loss to guide high-level semantic information back to the image fusion module, effectively improving the performance of high-level vision tasks on fused images.

[0003] In recent years, deep learning technology has demonstrated significant advantages in the field of image fusion due to its powerful feature learning capabilities and adaptive optimization characteristics. Deep learning models can automatically adjust network parameters through data-driven approaches, thereby achieving more efficient fusion of complementary information from multimodal images. However, most current research focuses only on optimizing objective evaluation metrics, resulting in problems such as insufficient information extraction in fused images. Furthermore, existing algorithms generally face challenges such as complex network structures and high computational costs, making it difficult to meet the urgent need for efficient and lightweight models in real-world scenarios. Therefore, how to improve the object detection performance of fused images and achieve lightweight network models remains a key problem that urgently needs to be solved. Summary of the Invention

[0004] This invention addresses the problems of insufficient information extraction, complex networks, high computational costs, and poor timeliness in current image fusion methods, and proposes an infrared and visible light image fusion method and device based on a lightweight model. The specific technical solution is as follows:

[0005] An infrared and visible light image fusion method based on a lightweight model is proposed, which constructs a shallow feature extraction layer, a deep feature extraction and fusion layer, and a fused image reconstruction layer, respectively.

[0006] The shallow feature extraction layer is used to extract shallow features from infrared and visible light images, filtering out information that is irrelevant to image fusion;

[0007] The deep feature extraction and fusion layer further extracts image features from infrared and visible light images based on the image feature map obtained from the shallow feature extraction layer, captures the feature information contained in the two images, and fuses the obtained information features.

[0008] The fused image reconstruction layer reconstructs the image based on the fused information features obtained from the deep feature extraction fusion layer, ultimately yielding a fused image that contains both infrared and visible light image information.

[0009] An infrared and visible light image fusion device based on a lightweight model includes a shallow feature extraction layer, a deep feature extraction and fusion layer, and a fused image reconstruction layer.

[0010] The shallow feature extraction layer is used to extract shallow features from infrared and visible light images, filtering out information that is irrelevant to image fusion;

[0011] The deep feature extraction and fusion layer further extracts image features from infrared and visible light images based on the image feature map obtained from the shallow feature extraction layer, captures the feature information contained in the two images, and fuses the obtained information features.

[0012] The fused image reconstruction layer reconstructs the image based on the fused information features obtained from the deep feature extraction fusion layer, ultimately yielding a fused image that contains both infrared and visible light image information.

[0013] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0014] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

[0015] The present invention has the following beneficial effects:

[0016] 1. A lightweight infrared and visible light image fusion network based on depthwise separable convolution and Transformer was designed, which enables more efficient fusion of information from infrared and visible light images.

[0017] 2. In the shallow feature extraction layer of the proposed network, depthwise separable convolution is innovatively used to reduce the computational cost of the model, which significantly reduces the computational cost (computational complexity) of the fusion process.

[0018] 3. In the deep feature extraction and fusion layers of the proposed network, this invention utilizes pooling and upsampling operations to reduce the complexity of the Transformer, thus proposing a lightweight Transformer (MTPU) to extract globally correlated information. Simultaneously, an EMA attention mechanism is employed to preserve important image information, achieving full extraction of image information and improving the effect and accuracy of image fusion. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the main network structure of the fusion method of the present invention;

[0020] Figure 2a This is a schematic diagram of the first depth-separable convolution module of the present invention;

[0021] Figure 2b This is a schematic diagram of the second depthwise separable convolution module of the present invention;

[0022] Figure 3 This is a schematic diagram of two depthwise separable convolutional modules;

[0023] Figure 4 This is a schematic diagram of a multi-head Transformer attention module based on pooling aggregation features;

[0024] Figure 5 A schematic diagram of the EMA attention mechanism;

[0025] Figure 6 The figure shows the experimental results of different fusion algorithms of this invention on the MSRS dataset. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0027] This invention proposes a lightweight model-based infrared and visible light image fusion method, the main structure of which is as follows: Figure 1 As shown, it consists of three parts: a shallow feature extraction layer, a deep feature extraction and fusion layer, and a fused image reconstruction layer.

[0028] The shallow feature extraction layer is used to extract shallow features from infrared and visible light images, initially filtering out information that is not very relevant to image fusion, so as to reduce the computational load of subsequent feature extraction modules.

[0029] The deep feature extraction and fusion layer further extracts image features from infrared and visible light images based on the image feature maps obtained from the shallow feature extraction layer, fully capturing the feature information contained in the two images, and then fusing the obtained information features.

[0030] The fused image reconstruction layer reconstructs the image based on the fused information features obtained from the deep feature extraction fusion layer, ultimately yielding a fused image that contains both infrared and visible light image information.

[0031] This method employs the basic structure of a CNN network and trains it in an end-to-end manner. In the shallow feature extraction layer, this invention uses three lightweight shallow feature extraction modules (Depthwise Separable Convolution, DSC-1) based on a first depthwise separable convolution module to progressively extract shallow features from the image. Depthwise separable convolution is a novel convolution module first proposed for image classification tasks. It can improve network performance and efficiency while reducing the number of network parameters and computational cost, thus maintaining the model's capabilities. In depthwise separable convolution, ordinary convolution modules are divided into pointwise convolution and depthwise convolution. The specific structure of the first depthwise separable convolution module is as follows: Figure 2a As shown.

[0032] Assuming the original feature map size is C1×H×W, and the desired output size is C2×H×W, depthwise separable convolution first divides the feature map into C1 1×H×W feature maps along the channel dimension. Then, it performs depthwise convolution on each feature map using a 1×n×n depthwise convolution kernel to extract deep features. The results are then concatenated along the channels and passed through a C2×1×1 pointwise convolution kernel to obtain the output feature map, where n is the size of the depthwise convolution kernel and is greater than 1. The parameter size of a regular convolution kernel is n×n×C1×C2, while the parameter size of depthwise separable convolution is C1×n×n+C1×C2. For most neural networks, depthwise separable convolution significantly reduces the number of parameters and computational cost, thus improving computational efficiency. This invention also proposes a second depthwise separable convolution module, which performs pointwise convolution followed by depthwise convolution, such as... Figure 2b As shown, assuming the original feature map size is C1×H×W, the required output size is C2×H×W. Depthwise separable convolution first outputs a feature map using a C2×1×1 pointwise convolution kernel. Then, the feature map is layered along the channel dimension, divided into C1 1×H×W feature maps. Next, a 1×n×n depthwise convolution kernel is used on each feature map to extract deep features. Finally, the results are concatenated along the channels to obtain the output feature map, where n is the size of the depthwise convolution kernel and is greater than 1.

[0033] In this invention, the module based on the first depthwise separable convolutional module is referred to as the DSC-1 module, and the module based on the second depthwise separable convolutional module is referred to as the DSC-2 module. The structure is as follows: Figure 3As shown. When performing convolution operations that increase the channel dimension, the DSC-1 module has fewer parameters and lower computational cost; when performing dimensionality reduction operations that decrease the channel dimension, the DSC-2 module is more lightweight. In the method of this invention, the shallow feature extraction layer is an dimensionality increase operation, therefore the DSC-1 module is used; the image reconstruction layer is a dimensionality reduction operation, therefore the DSC-2 module is used. Specifically, the DSC-1 module performs depthwise convolution followed by batch normalization, then pointwise convolution, and finally batch normalization again. The DSC-2 module performs pointwise convolution followed by batch normalization, then depthwise convolution, and finally batch normalization again.

[0034] In this invention, depthwise separable convolution is used instead of ordinary convolution to extract shallow features from infrared and visible light images. This represents a depthwise separable convolutional feature extractor, using... This represents the output feature map. Given the input feature map, the extraction formula is expressed as follows:

[0035] ;

[0036] exist Figure 1 In the deep feature extraction and fusion layer, this invention uses a lightweight multi-head Transformer Attention Mechanism Based on Pooled and Upsampling (MTPU) and an efficient multi-scale attention module with cross-spatial learning (EMA) connected in series.

[0037] The MTPU module is a novel Transformer attention module based on pooling aggregation features proposed in this invention, and its structure diagram is shown below. Figure 4 As shown:

[0038] This module uses average pooling to reduce the spatial resolution of the feature map, then performs a Transformer self-attention mechanism transformation, and finally restores the original resolution through bilinear interpolation. To reduce computational cost, the network employs a multi-head attention mechanism, dividing the original feature map into eight heads based on channels. If all heads are downsampled using the same pooling layer before Transformer transformation, edge information between the sampled regions of the upsampled fused image will be lost, resulting in numerous artifacts. To avoid this, the module divides the eight heads into four parts, sampling the feature map using pooling layers of sizes 2×2, 4×4, 8×8, and 16×16 respectively, with a stride equal to the pooling kernel side length. The feature images obtained after sampling through different pooling layers are then processed by their respective Transformer modules and input into an upsampling layer. Different upsampling rates are used to obtain feature maps with the same resolution as the original feature map. Different heads can compensate for missing edge information, resulting in an output feature map whose size remains unchanged from the original feature map. The formula for the MTPU module can be expressed as follows:

[0039] ;

[0040] in, This represents the output feature map. Indicates the input feature map, This indicates the MTPU module feature extractor.

[0041] The MTPU module reduces the computational cost of the Transformer module through downsampling and multi-head attention mechanisms. It also uses pooling layers with different downsampling rates to downsample different heads to reduce information loss. This module effectively lightweights the Transformer structure, reduces the number of parameters and computational cost of the algorithm, and ensures the quality of the fused image.

[0042] Depthwise separable convolutional modules and MTPU modules can cause some loss of spatial and channel information. To reduce this loss and enhance the ability to focus on important features, this algorithm incorporates an EMA attention mechanism module, the structure of which is as follows: Figure 5 As shown.

[0043] The EMA module divides a given input feature map of size C×H×W into n sub-feature maps along the channel dimension to learn different semantics, where the depth is C, the height is H, and the width is W. First, the input feature map is cut into n blocks along the depth direction. Attention weights for the grouped feature maps are extracted through three parallel paths: two parallel paths on a 1×1 branch and the third path on a 3×3 branch. To capture the dependencies between all channels and reduce computational cost, for the 1×1 branch, channels are encoded along both the X and Y spatial directions and two average pooling operations are performed; for the 3×3 branch, only one 3×3 convolutional kernel is stacked. The features extracted from the two spatial directions of the 1×1 branch are then concatenated and subjected to a 1×1 convolution. The output is split into two vectors, and two non-linear sigmoid functions are used to fit the two-dimensional binomial distribution after convolution. A re-weighting module restores the input feature map patch to a size of C×H×W. A cross-spatial learning module then integrates the outputs of the 3×3 branches, and finally passes them through a sigmoid function and a re-weighting module to obtain the output result. EMA reduces computational cost while preserving information in each channel. It also groups the channel dimension into multiple sub-features, ensuring that spatial semantic features are evenly distributed within each feature group and highlighting the global vertical connections between all pixels in the feature map.

[0044] The cross-spatial learning module in the EMA module structure establishes the interdependence between channels and spatial locations, embedding accurate location information into the EMA module. First, two-dimensional global average pooling is performed on the outputs of the 1×1 and 3×3 branches, and a softmax function is used to fit a linear transformation at the average pooling output. By multiplying the output of each branch by a matrix dot product, the spatial attention feature maps for each branch are obtained. Finally, the output feature maps within each group are calculated into a set of two generated spatial attention weight values, and then a sigmoid function is used to capture pixel-level pairwise relationships.

[0045] In the fused image reconstruction layer, the feature map obtained by the deep feature extraction fusion layer is passed through three cascaded DSC-2 modules and then through the Sigmoid function to obtain the fused image.

[0046] To verify the fusion performance and lightweight nature of this invention, a comparative experiment with similar methods was designed. The experiment used the PyTorch deep learning framework and trained the model on a single NVIDIA GeForce GPU 3090. The publicly available remote sensing dataset MSRS was selected as the training data to train the proposed network method. This dataset contains 1444 high-quality image pairs, most of which have low signal-to-noise ratios and low contrast, and include both daytime and nighttime scenes. 1083 image pairs were selected as the training dataset, and 33 image pairs were selected as the test dataset. To fully demonstrate the superiority of the fusion algorithm, the experiment compared it with six other state-of-the-art methods: Cddfuse, RFN, UNF, IFCNN, DID, and Seafusion. The implementations of these six methods are publicly available. The experiment fused the images according to the parameters set in the authors' source code and then compared the fused images with those fused by the proposed algorithm.

[0047] For the evaluation metrics, five different evaluation methods were selected. These include: information entropy-based metrics such as information entropy (EN), mutual information (MI), and edge information-based metrics (QAB / F); image feature-based metrics such as standard deviation (SD) and spatial frequency (SF); correlation-based metrics such as differential correlation sum (SCD); human perception-based metrics such as visual information fidelity (VIF); and image structure-based metrics such as structural similarity measure (SSIM). Furthermore, for lightweight evaluation criteria, floating-point computation time (FLOPS), model parameter count, and model size were used.

[0048] Figure 6 To visually represent the fusion results of the MSRS dataset and avoid data randomness, this invention uses five sets of images for comparison and analysis, including three daytime scene images and two nighttime scene images. The green boxes in the images highlight detailed texture information. From a qualitative perspective, it can be seen that there are certain differences among the seven image fusion methods, exhibiting different fusion focuses and effect characteristics.

[0049] In daytime scenes, the fused image primarily utilizes texture details from the visible light image, supplemented by prominent target information from the infrared image. Comparing the first three images, it's evident that the fusion results of IFCNN and RFN-Nest methods are generally visually poor, with low environmental contrast and loss of some intensity and detail information from the source images. The green box exhibits artifacts due to the fusion of excessive infrared information, indicating a lack of balance between infrared and visible light information, resulting in redundant information in the image. UNF, SeaFusion, Cddfuse, and DID methods preserve overall image intensity information well, with similar fusion information. Regarding the texture details within the green box, comparing the first and second images, SeaFusion and UNF contain more visible light information, while Cddfuse and DID contain more infrared image information. In the third image, all four algorithms exhibit noise, with white artifacts appearing above the person's head, indicating the fusion of redundant infrared information. Only the algorithm proposed in this invention maintains a good balance between infrared and visible light image information, effectively preserving texture information in daytime scenes.

[0050] In nighttime scenes, image fusion needs to highlight not only salient targets in the infrared image but also texture details in the visible light image with light sources. This requires striking a balance between the two. As shown in the fourth and fifth images of nighttime scenes, the overall contrast of RFN-Nest and IFCNN images is still relatively low, and the images are too dark. Areas containing light source information are not obvious in dark scenes, which is not conducive to human visual observation of salient targets. UNF, SeaFusion, CddFuse, and DID highlight salient targets in the infrared image and fuse more visible light information in the light source areas of dark scenes. Comparing the green box areas in the fourth and fifth images, UNF and DID have a slight loss of light intensity, while CddFuse and SeaFusion have brighter lights compared to the visible light image and some information redundancy. Only the algorithm of this invention not only retains the salient target information in the infrared image but also retains the visible light image information well in the light source areas.

[0051] In summary, the method proposed in this invention strikes a balance between different aspects of image fusion. It preserves as much visible light image information as possible in daytime scenes, highlights important target information in nighttime scenes, and ensures that information near light sources is closer to the visible light image. All images remain clear while maintaining high brightness for infrared targets.

[0052] Network lightweighting refers to reducing the number of parameters and computational cost of neural networks, thereby reducing the complexity of the model and the computational resource requirements, while the performance and functionality loss are relatively low. In order to verify the lightweighting effect and fusion quality of the proposed network, this section uses the lightweighting evaluation criteria and fusion quality evaluation indicators described in the experimental settings for comprehensive evaluation. The comparison algorithm is the six classic algorithms mentioned above, and the results are shown in Tables 1 and 2.

[0053] Table 1. Comparison of lightweight fusion methods: experimental results

[0054]

[0055] Table 2 Comparative experimental results of different fusion methods

[0056]

[0057] As shown in Table 1, the method of this invention has the smallest number of model parameters and the smallest floating-point computation, and also the smallest memory footprint. This indicates that compared to other comparative algorithms, this invention achieves the best lightweight performance. Furthermore, Table 2 shows that compared to other algorithms, this invention achieves more uniform evaluation results across various metrics, exceeding half of similar algorithms in most metrics. Moreover, for other non-optimal metrics, this invention, despite significantly reducing the number of parameters, still achieves optimal and near-optimal values ​​in most evaluation results. This demonstrates that this invention maintains lightweight performance while better ensuring the quality of the fused image.

[0058] The present invention also provides an infrared and visible light image fusion method based on a lightweight model, which constructs a shallow feature extraction layer, a deep feature extraction and fusion layer and a fused image reconstruction layer, respectively.

[0059] The shallow feature extraction layer is used to extract shallow features from infrared and visible light images, filtering out information that is irrelevant to image fusion;

[0060] The deep feature extraction and fusion layer further extracts image features from infrared and visible light images based on the image feature map obtained from the shallow feature extraction layer, captures the feature information contained in the two images, and fuses the obtained information features.

[0061] The fused image reconstruction layer reconstructs the image based on the fused information features obtained from the deep feature extraction fusion layer, ultimately yielding a fused image that contains both infrared and visible light image information.

[0062] The present invention also provides an infrared and visible light image fusion device based on a lightweight model, including a shallow feature extraction layer, a deep feature extraction and fusion layer, and a fused image reconstruction layer;

[0063] The shallow feature extraction layer is used to extract shallow features from infrared and visible light images, filtering out information that is irrelevant to image fusion;

[0064] The deep feature extraction and fusion layer further extracts image features from infrared and visible light images based on the image feature map obtained from the shallow feature extraction layer, captures the feature information contained in the two images, and fuses the obtained information features.

[0065] The fused image reconstruction layer reconstructs the image based on the fused information features obtained from the deep feature extraction fusion layer, ultimately yielding a fused image that contains both infrared and visible light image information.

[0066] The present invention also provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0067] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

Claims

1. An infrared and visible light image fusion method based on a lightweight model, characterized in that, The shallow feature extraction layer, the deep feature extraction fusion layer and the fusion image reconstruction layer are respectively constructed; The shallow feature extraction layer is used for extracting the shallow features of the infrared and visible light images and filtering out the information irrelevant to image fusion; The deep feature extraction fusion layer further extracts the image features of the infrared and visible light images on the basis of the image feature maps obtained by the shallow feature extraction layer, captures the feature information contained in the two kinds of images, and fuses the obtained information features; The fusion image reconstruction layer reconstructs the image on the basis of the fusion information features obtained by the deep feature extraction fusion layer, and finally obtains the fusion image containing the infrared image and visible light image information; In the deep feature extraction fusion layer, a lightweight multi-head attention mechanism module and an efficient multi-scale attention module are connected in series; the lightweight multi-head attention mechanism module uses average pooling to reduce the spatial resolution of the feature map and then performs self-attention mechanism transformation, and then restores the original resolution through bilinear interpolation; The efficient multi-scale attention module groups the channel dimension into multiple sub-features based on the information retained on each channel, so that the spatial semantic features are uniformly distributed in each feature group, and the global upper and lower connections of all pixels in the feature map are highlighted; The lightweight multi-head attention mechanism module divides the 8 heads into four parts, respectively uses the pooling layers with sizes of 2x2, 4x4, 8x8 and 16x16 to sample the feature map, the step length is the same as the size of the pooling kernel, and the feature images obtained after sampling through different pooling layers are then input into the attention mechanism, and then input into the upsampling layer to obtain the feature images with the same resolution as the original feature map through different upsampling rates, the missing edge information between different heads is made up, and finally the output is obtained, the output feature map size remains unchanged; The efficient multi-scale attention module divides the given input feature map with a size of CxHxW into n sub-feature maps along the channel dimension direction to learn different semantics, where the depth is C, the height is H, and the width is W; the input feature map is first cut into n blocks along the depth direction; three parallel paths are used to extract the attention weights of the grouped feature maps, wherein two parallel paths are on the 1x1 branch, and the third path is on the 3x3 branch; for the 1x1 branch, the channels are encoded along the X and Y spatial directions and two average pooling operations are performed; for the 3x3 branch, only a 3x3 convolution kernel is stacked, then the features extracted from the two spatial directions of the 1x1 branch are connected, and the output is passed through a 1x1 convolution, and the output is split into two vectors, two nonlinear Sigmoid functions are used to fit the two-dimensional binomial distribution after convolution, the input feature map block is restored to CxHxW size by using the Re-weight module, and the output of the 3x3 branch is integrated by using the cross-space learning module, and then the output result is obtained through the Sigmoid function and the Re-weight module; The cross-space learning module performs two-dimensional global average pooling on the outputs of the 1x1 branch and the 3x3 branch, and simultaneously adopts a Softmax function at the output of the average pooling to fit a linear transformation, thereby obtaining respective spatial attention feature maps of the two branches by multiplying the outputs of the branches with a matrix dot product operation, and finally calculating the output features in each group as a set of generated spatial attention weight values, and then using a Sigmoid function to capture the pixel-level pairwise relationship.

2. The infrared and visible image fusion method based on a lightweight model according to claim 1, characterized in that, The shallow feature extraction layer uses a plurality of lightweight shallow feature extraction modules based on a first depth separable convolution module to gradually extract shallow features of the image, the first depth separable convolution module is divided into point-by-point convolution and depth convolution, the original feature map size is C1xHxW, and the output size is C2xHxW feature map, the first depth separable convolution module first divides the feature map in the channel dimension into C1 1xHxW feature maps, then uses a 1xnxn depth convolution kernel on each feature map to extract deep features, and then splices the obtained results in the channel to obtain the output feature map through a C2x1x1 point-by-point convolution kernel, and n is the size of the depth convolution kernel and is greater than 1. 3.The infrared and visible image fusion method based on a lightweight model according to claim 1, characterized in that, In the fusion image reconstruction layer, the feature map obtained through the deep feature extraction fusion layer is processed through three lightweight shallow feature extraction modules based on a second depth separable convolution module in series, and then a Sigmoid function is used to obtain a fusion image; in the second depth separable convolution module, the original feature map size is C1xHxW, and the output size is C2xHxW; first, a C2x1x1 point-by-point convolution kernel is used to output the feature map, then the feature map is divided in the channel dimension into C1 1xHxW feature maps, then a 1xnxn depth convolution kernel is used on each feature map to extract deep features, and then the obtained results are spliced in the channel to obtain the output feature map, and n is the size of the depth convolution kernel and is greater than 1.

4. An infrared and visible image fusion device based on a lightweight model, which implements the method of claim 1, characterized in that, It comprises a shallow feature extraction layer, a deep feature extraction fusion layer, and a fusion image reconstruction layer. The shallow feature extraction layer is used to extract shallow features of infrared and visible light images, and filter out information irrelevant to image fusion. The deep feature extraction fusion layer further extracts image features of infrared and visible light images on the basis of the image feature maps obtained by the shallow feature extraction layer, captures feature information contained in the two images, and fuses the obtained information features; The fusion image reconstruction layer reconstructs an image on the basis of the fusion information features obtained by the deep feature extraction fusion layer, and finally obtains a fusion image containing infrared image and visible light image information.

5. An electronic device, comprising: It comprises: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 3.

6. A computer readable storage medium characterized by, a processor, and a memory having stored thereon executable instructions that, as a result of being executed by the processor, cause the processor to carry out the method of any one of claims 1 to 3.

Images