A visible light and infrared image fusion method based on semantic perception

Abstract

The present invention relates to a semantically aware visible light and infrared image fusion method, comprising the following steps: acquiring a paired visible light image and infrared image; converting the infrared image into a single-channel grayscale image; converting the visible light image into a YCbCr format and using only the Y channel to obtain a Y-channel image; and inputting the grayscale image and the Y-channel image into a fusion network model to obtain a single-channel fused image. The present invention can effectively aggregate local and global information, giving the fusion network semantic awareness capabilities.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a visible light and infrared image fusion method based on semantic perception. Background Art

[0002] In outdoor scenes, targets are easily obscured by complex backgrounds, making it difficult to effectively identify them using a single source of information. The introduction of multimodal information is a key trend in the development of detection technology. Visible light images typically have high resolution and can capture rich texture details and background information in sufficient lighting conditions. However, their imaging performance is significantly affected by lighting conditions and occlusion. Extreme weather conditions such as wind, sand, rain, and snow are common in outdoor scenes. Furthermore, dense forests, complex and diverse terrain, and significantly insufficient nighttime illumination make it difficult to capture effective target information using visible light images alone. Infrared thermal imaging, unaffected by lighting and occlusion, can compensate for the limitations of visible light imaging, but infrared images suffer from a loss of texture detail. In recent years, visible light images have been the most widely used in target detection, and infrared thermal imaging technology has also matured. Fusion of visible light and infrared images can produce fused images with rich texture and high contrast, better meeting the needs of outdoor applications. Fusion of infrared and visible light images is currently widely used in target detection, image segmentation, video surveillance, and other fields.

[0003] The fusion of visible light and infrared images is essentially an image enhancement technique, whose purpose is to provide better data support for downstream visual tasks such as subsequent target detection and semantic segmentation. However, most existing visible light and infrared image fusion methods focus on improving the model's ability to extract and preserve details in order to obtain fused images with rich details and obvious contrast, but often ignore the model's semantic perception ability. Semantic perception enables the model to understand the overall information of the scene presented by the image and distinguish between foreground and background. Models that lack semantic perception can present more complex textures locally, but may ignore the maintenance of complete semantic targets, resulting in incomplete targets or discontinuous target edges, which is not conducive to human perception and machine perception of downstream tasks such as target detection and semantic segmentation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a visible light and infrared image fusion method based on semantic perception, which can effectively aggregate local and global information and give the fusion network the ability of semantic perception.

[0005] The technical solution adopted by the present invention to solve the technical problem is to provide a visible light and infrared image fusion method based on semantic perception, comprising the following steps:

[0006] Acquire paired visible light and infrared images;

[0007] Convert the infrared image into a single-channel grayscale image, convert the visible light image into the YCbCr format and use the Y channel alone to obtain the Y channel image;

[0008] Inputting the grayscale image and the y-channel image into a fusion network model to obtain a single-channel fused image;

[0009] The fusion network model includes:

[0010] A first multi-scale feature extraction module is used to extract multi-scale feature information of the y channel image;

[0011] A second multi-scale feature extraction module, configured to extract multi-scale feature information of the grayscale image;

[0012] a feature fusion module, configured to perform intra-modal spatial interaction and inter-modal interaction on the multi-scale feature information of the y-channel image and the multi-scale feature information of the grayscale image to obtain multi-layer fusion features, wherein all fusion features except the fusion features of the last layer of the multi-layer fusion features are returned to the first multi-scale feature extraction module and the second multi-scale feature extraction module for feature extraction;

[0013] A fusion head module, configured to perform image fusion based on the first N layers of fusion features in the multi-layer fusion features;

[0014] The semantic segmentation head module is used to perform semantic segmentation based on the last N layers of fusion features in the multi-layer fusion features.

[0015] The first multi-scale feature extraction module and the second multi-scale feature extraction module have the same structure and both include:

[0016] Convolution operation layer, used to map the input image into high-dimensional features;

[0017] Five Mamba feature extraction layers constitute a Unet structure for multi-scale feature extraction. The output of each Mamba feature extraction layer is input into the feature fusion module for feature fusion. The feature fusion results of the first four layers are added to the output of the Mamba feature extraction layer through a feature addition operation and then serve as the input of the next Mamba feature extraction layer. The first Mamba feature extraction layer is connected to the input of the fifth Mamba feature extraction layer through a skip connection, and the second Mamba feature extraction layer is connected to the input of the fourth Mamba feature extraction layer through a skip connection. A downsampling layer is set after the first and second Mamba feature extraction layers; and an upsampling layer is set after the third and fourth Mamba feature extraction layers.

[0018] The feature fusion module includes:

[0019] The spatial-first scanning Mamba feature extraction layer processes input features using a spatial-first scanning strategy that first scans within the current modality and then scans in the same direction in the next modality.

[0020] The modality-first scanning Mamba feature extraction layer processes input features using a modality-first scanning strategy that scans the corresponding positions of the two modalities and then obtains the next position in the order of the spatial dimensions.

[0021] The fusion output layer is used to add the output of the Mamba feature extraction layer of the space-first scanning and the output of the Mamba feature extraction layer of the modality-first scanning, and then perform a linear operation of the convolution layer to obtain the fusion feature.

[0022] The Mamba feature extraction layer includes:

[0023] The layer normalization unit is used to perform layer normalization on the input feature map;

[0024] The first convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map z;

[0025] The second convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map x;

[0026] A SiLU operation unit, configured to perform a SiLU operation on the feature map z to obtain a first feature;

[0027] a feature processing unit, configured to flatten the feature map x into a one-dimensional sequence in a horizontal forward direction, a horizontal reverse direction, a vertical forward direction, and a vertical reverse direction, and perform a separate state space model processing on each one-dimensional sequence, then cancel the flattening operation to obtain four outputs, and combine the four outputs to obtain a second feature;

[0028] a dot product unit, configured to perform a dot product of the first feature and the second feature;

[0029] The output unit is used to perform a convolutional layer linear operation on the output of the point multiplication unit, and then add it to the input feature map to obtain the output feature.

[0030] The fusion head module includes a first processing unit, a second processing unit and a third processing unit connected in sequence, wherein:

[0031] The first processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence;

[0032] The second processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence;

[0033] The third processing unit includes a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer and a Tanh activation layer connected in sequence.

[0034] The semantic segmentation head module includes two fourth processing units, a fifth processing unit and a sixth processing unit connected in sequence, wherein:

[0035] The fourth processing unit includes a 3×3 convolution layer, a batch normalization operation layer and a bilinear interpolation layer connected in sequence;

[0036] The fifth processing unit includes three processing blocks connected in sequence, each processing block including a 3×3 convolutional layer, a batch normalization operation layer, and a ReLU activation layer connected in sequence;

[0037] The sixth processing unit includes a 1×1 convolution layer, a batch normalization operation layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence.

[0038] The loss function of the fusion network model is expressed as: L all =L fuse +αL seg , where L all is the loss of the fusion network model, L fuse is the image fusion loss, L seg is the semantic segmentation loss, and α is the balance coefficient.

[0039] The semantic segmentation loss is expressed as: L seg =OhemCELoss(I seg ,I label ), where OhmCELoss() is the image segmentation loss function, I seg is the output of the semantic segmentation head module, I label is the semantic segmentation label.

[0040] The image fusion loss is expressed as: in, is the gradient loss, expressed as: is the content loss, expressed as: s1 and s2 are weighted coefficients, I fuse is the output of the fusion head module, I vis is a visible light image, I inf For infrared images, Indicates the Sobel operator for gradient operation, max indicates the maximum value, || ||1 indicates the L1 norm, || ||2 indicates the L2 norm, I inf (x,y) represents the brightness of the pixel with coordinates (x,y) in the infrared image, I vis (x,y) represents the brightness of the pixel at coordinates (x,y) in the visible light image.

[0041] The method for fusing visible light and infrared images based on semantic perception further includes:

[0042] Extract the cb channel and cr channel of the visible light image in ycbcr format to obtain the cb channel image and cr channel image;

[0043] The single-channel fused image is spliced ​​with the cb channel image and the cr channel image to obtain a ycbcr fused image;

[0044] The YCbCr fused image is converted into RGB format to obtain an RGB fused image.

[0045] Beneficial effects

[0046] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention uses multi-scale feature extraction to extract the multi-scale features of each modality of infrared images and visible light images respectively, and uses U-shaped networks and jump connections to avoid forgetting key spatial information, and enhances the model's ability to extract long-distance dependencies with linear time complexity, and uses semantic segmentation tasks to drive the feature extraction of the entire Unet, thereby improving the model's detail extraction and semantic perception capabilities, and promoting the fusion head module to obtain features with sufficient spatial details and more conducive to semantic perception. The fusion module used in the present invention adopts two strategies of joint space-first scanning and modality-first scanning to achieve more sufficient and efficient feature aggregation of the two modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a method for fusion of visible light and infrared images based on semantic perception according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the structure of the fusion network model in an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of the Mamba feature extraction layer in an embodiment of the present invention;

[0050] Figure 4 Schematic diagram of a feature fusion module (horizontal forward scanning) in an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a fusion head module in an embodiment of the present invention;

[0052] Figure 6 Schematic diagram of a semantic segmentation head module in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0054] The embodiments of the present invention relate to a method for fusing visible light and infrared images based on semantic perception. In order to solve the problem of optimizing the performance of two tasks in two independent networks, the present embodiment performs joint training in an end-to-end network, and drives the feature extraction of the entire Unet with the semantic segmentation task, thereby promoting the fusion head module to obtain features with sufficient spatial details and more conducive to semantic perception. Similarly, the fusion task promotes the full extraction of shallow detail features, thereby providing favorable support for subsequent semantic segmentation tasks. In order to solve the problem of simplifying the network structure and computational complexity, the Cnn-Mamba Unet of the present embodiment uses the most typical mamba block in the state space model to extract features, thereby enhancing the global perception capability of the model with linear complexity and achieving efficient feature extraction.

[0055] like Figure 1 As shown, the visible light and infrared image fusion method based on semantic perception of this embodiment includes the following steps:

[0056] Step 1: Acquire a pair of visible light images and infrared images;

[0057] Step 2: Convert the infrared image into a single-channel grayscale image, convert the visible light image into the YCbCr format and use the Y channel alone to obtain a Y channel image;

[0058] Step 3: input the grayscale image and the y-channel image into a fusion network model to obtain a single-channel fused image;

[0059] Step 4, extract the cb channel and cr channel of the visible light image in ycbcr format to obtain a cb channel image and a cr channel image;

[0060] Step 5: splice the single-channel fused image with the cb channel image and the cr channel image to obtain a ycbcr fused image;

[0061] Step 6: Convert the YCbCr fused image into RGB format to obtain an RGB fused image.

[0062] like Figure 2 As shown, the fusion network model in this embodiment includes:

[0063] A first multi-scale feature extraction module is used to extract multi-scale feature information of the y channel image;

[0064] A second multi-scale feature extraction module, configured to extract multi-scale feature information of the grayscale image;

[0065] a feature fusion module, configured to perform intra-modal spatial interaction and inter-modal interaction on the multi-scale feature information of the y-channel image and the multi-scale feature information of the grayscale image to obtain multi-layer fusion features, wherein all fusion features except the fusion features of the last layer of the multi-layer fusion features are returned to the first multi-scale feature extraction module and the second multi-scale feature extraction module for feature extraction;

[0066] A fusion head module, configured to perform image fusion based on the first N layers of fusion features in the multi-layer fusion features;

[0067] The semantic segmentation head module is used to perform semantic segmentation based on the last N layers of fusion features in the multi-layer fusion features.

[0068] In this embodiment, the first multi-scale feature extraction module serves as a visible light branch to extract multi-scale features from a visible light image, and the second multi-scale feature extraction module serves as an infrared branch to extract multi-scale features from an infrared image.

[0069] The first multi-scale feature extraction module and the second multi-scale feature extraction module have the same structure and both include:

[0070] Convolution operation layer, used to map the input image into high-dimensional features;

[0071] Five Mamba feature extraction layers constitute a Unet structure for multi-scale feature extraction. The output of each Mamba feature extraction layer is input into the feature fusion module for feature fusion. The feature fusion results of the first four layers are added to the output of the Mamba feature extraction layer through a feature addition operation and then serve as the input of the next Mamba feature extraction layer. The first Mamba feature extraction layer is connected to the input of the fifth Mamba feature extraction layer through a skip connection, and the second Mamba feature extraction layer is connected to the input of the fourth Mamba feature extraction layer through a skip connection. A downsampling layer is set after the first and second Mamba feature extraction layers; and an upsampling layer is set after the third and fourth Mamba feature extraction layers.

[0072] Taking the visible light branch as an example, the input visible light source image is first mapped into high-dimensional features through a 3×3 convolution kernel, a convolution operation with a padding of 1 and a stride of 1, and a PreLU activation function. Features are then extracted through five layers of Mamba blocks (i.e., the Mamba feature extraction layer). The outputs of each layer are sent to the feature fusion module FMamba for feature fusion. The fused features of the first four layers are returned to the Cnn-Mamba Unet through the element-add operation for subsequent feature extraction. The first and second layers of the Mamba blocks are respectively connected to the Down block for double downsampling, and the third and fourth layers of the Mamba blocks are respectively connected to the Up block for double upsampling. The Down block consists of a 3×3 convolution kernel, a convolution operation with a padding of 0 and a stride of 1, and a PreLU activation function. The Up block consists of a 3×3 convolution kernel, a deconvolution operation with a padding of 0 and a stride of 1, and a PreLU activation function.

[0073] like Figure 3 As shown in the figure, the Mamba block adopts the four-direction (Scan 1: horizontal forward scan, Scan 2: horizontal reverse scan, Scan 3: vertical forward scan, Scan 4: vertical reverse scan) sequence modeling method proposed in VMamba to enhance spatial perception capabilities. It includes:

[0074] The layer normalization unit is used to perform layer normalization on the input feature map;

[0075] The first convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map z;

[0076] The second convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map x;

[0077] A SiLU operation unit, configured to perform a SiLU operation on the feature map z to obtain a first feature;

[0078] a feature processing unit, configured to flatten the feature map x into a one-dimensional sequence in a horizontal forward direction, a horizontal reverse direction, a vertical forward direction, and a vertical reverse direction, and perform a separate state space model processing on each one-dimensional sequence, then cancel the flattening operation to obtain four outputs, and combine the four outputs to obtain a second feature;

[0079] a dot product unit, configured to perform a dot product of the first feature and the second feature;

[0080] The output unit is used to perform a convolutional layer linear operation on the output of the point multiplication unit, and then add it to the input feature map to obtain the output feature.

[0081] The feature maps of the input Mamba block (size H×W×C) are first layer-normalized (LN) and then linearly mapped into different feature maps x and z through two parallel 1×1 convolutional layers. The feature map labeled x is flattened in four directions into a one-dimensional sequence (size HW×C) using a flatten operation. These one-dimensional sequences are processed by separate SSM blocks, followed by unflattening to obtain four outputs, which are combined through an element-add operation to obtain a new output. This is then dot-multiplied with the feature map z after SiLU, and finally added to the original features of the input module after a linear operation to obtain the final output (size H×W×C).

[0082] It is not difficult to find that in this embodiment, the first multi-scale feature extraction module and the second multi-scale feature extraction module both adopt Cnn-Mamba Unet, which uses the Cnn-Mamba architecture to extract multi-scale information from the source image and uses U-shaped network and jump connection to avoid forgetting key spatial information.

[0083] The features extracted from the five-layer Mamba block have different scales, which are 1, 1 / 2, 1 / 4, 1 / 2, and 1 times the source image input size. The infrared and visible light features are channel-wise concatenated and then input into the FMamba block (i.e., the feature fusion module) for feature aggregation.

[0084] The Mamba block is only suitable for single input and cannot process input from two modalities. Figure 4As shown) consists of two symmetrical Mamba blocks, one of which is a Mamba feature extraction layer for space-first scanning and the other is a Mamba feature extraction layer for modality-first scanning. The two Mamba blocks are different from Figure 3 The Mamba block in the feature fusion module expands the original four-directional 2D scanning to four-directional 3D scanning to accommodate dual-modal input features and interact between features. The Mamba feature extraction layer with spatial-first scanning uses a spatial-first scanning strategy to process input features. During processing, it first scans within the current modality and then scans in the same direction in the next modality. The Mamba feature extraction layer with modality-first scanning uses a modality-first scanning strategy to process input features. During processing, it scans the corresponding positions of the two modalities and then obtains the next position in the order of scanning in the spatial dimension. Finally, the output of the Mamba feature extraction layer with spatial-first scanning is added to the output of the Mamba feature extraction layer with modality-first scanning, and then a linear operation of the convolution layer is performed to obtain the fused feature.

[0085] It's easy to see that the feature fusion module in this implementation extends the typical Vision Mamba module to include inputs from both modalities, extracting long-range dependencies with linear time complexity while simultaneously performing spatial interactions within each modality and between modalities. Compared to existing fusion modules, this implementation's feature fusion module achieves more comprehensive and efficient feature aggregation across both modalities.

[0086] After obtaining the five levels of features (F_in1, F_in2, F_in3, F_in4, F_in5) output from the feature fusion module, the features of the 1st, 2nd, and 3rd layers with 1, 1 / 2, and 1 / 4 sizes are sent to the fusion head module to obtain the fused image, and the features of the 3rd, 4th, and 5th layers with 1 / 4, 1 / 2, and 1 sizes are sent to the semantic segmentation head module to obtain the semantic segmentation results.

[0087] like Figure 5As shown, the fusion head module includes a first processing unit, a second processing unit, and a third processing unit connected in sequence. The first processing unit includes a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, a 3×3 convolutional layer, and a bilinear interpolation layer connected in sequence. The second processing unit includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a bilinear interpolation layer connected in sequence. The third processing unit includes a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer, and a Tanh activation layer connected in sequence. The input of the first processing unit is feature F_in3, the input of the second processing unit is the channel-wise concatenation of the output of the first processing unit and feature F_in2, and the input of the third processing unit is the channel-wise concatenation of the output of the second processing unit and feature F_in1.

[0088] like Figure 6 As shown, the semantic segmentation head module includes two fourth processing units, a fifth processing unit, and a sixth processing unit connected in sequence. The fourth processing unit includes a 3×3 convolutional layer, a batch normalization operation layer, and a bilinear interpolation layer connected in sequence. The fifth processing unit includes three processing blocks connected in sequence, each including a 3×3 convolutional layer, a batch normalization operation layer, and a ReLU activation layer connected in sequence. The sixth processing unit includes a 1×1 convolutional layer, a batch normalization operation layer, a ReLU activation layer, and a 3×3 convolutional layer connected in sequence. The input of the first fourth processing unit is feature F_in3, the input of the second fourth processing unit is the channel-wise concatenation of the output of the first fourth processing unit and feature F_in4, the input of the fifth processing unit is the channel-wise concatenation of the output of the second fourth processing unit and feature F_in5, and the input of the sixth processing unit is the output of the fifth processing unit.

[0089] During the training phase, the fusion network model in this embodiment is an end-to-end model. The training input is a pair of infrared images, visible light images, and semantic segmentation labels. The fusion task and the semantic segmentation task are optimized simultaneously. The overall loss function is as follows:

[0090] L all =L fuse +αL seg

[0091] Among them, L all is the loss of the fusion network model, L fuse is the image fusion loss, L seg is the semantic segmentation loss, and α is the balance coefficient.

[0092] The semantic segmentation loss is calculated using the common image segmentation loss function in semantic segmentation tasks (Onlinehardexample miningCrossEntropyLoss, OhemCELoss), which is expressed as follows:

[0093] L seg =OhemCELoss(I seg ,I label )

[0094] Among them, I seg is the output of the semantic segmentation head module, I label is the semantic segmentation label.

[0095] Image fusion loss includes gradient loss and content loss The fusion image generated by the gradient loss guidance captures all the texture details in the two source images. The fusion image generated by the content loss guidance maintains content consistency with the source images. In particular, based on the characteristics of the visible light source image having background texture details and the infrared source image having a more prominent foreground contrast, the fusion image is constrained to be consistent with the visible light source image in terms of pixels, and the areas with higher brightness in the infrared source image tend to be consistent with the infrared source image pixels. fuse , and is defined as follows:

[0096]

[0097]

[0098]

[0099]

[0100] Among them, I fuse is the output of the fusion head module, I vis is a visible light image, I inf For infrared images. Indicates the Sobel operator to find the gradient operation, max means to take the maximum value, || ||1 means L1 norm, || ||2 means L2 norm. s1 and s2 are gradient loss respectively. and content loss The weighting coefficient of .

[0101] It is not difficult to find that the present invention uses multi-scale feature extraction to extract multi-scale features of each modality for infrared images and visible light images respectively, and uses U-shaped networks and jump connections to avoid forgetting key spatial information, and enhances the model's ability to extract long-distance dependencies with linear time complexity. The semantic segmentation task drives the feature extraction of the entire Unet, thereby improving the model's detail extraction and semantic perception capabilities, and promoting the fusion head module to obtain features with sufficient spatial details and more conducive to semantic perception.

Claims

1. A visible light and infrared image fusion method based on semantic perception, characterized in that: The following steps are involved: Acquire paired visible light and infrared images; Convert the infrared image into a single-channel grayscale image, convert the visible light image into the YCbCr format and use the Y channel alone to obtain the Y channel image; Inputting the grayscale image and the y-channel image into a fusion network model to obtain a single-channel fused image; The fusion network model includes: A first multi-scale feature extraction module is used to extract multi-scale feature information of the y channel image; A second multi-scale feature extraction module, configured to extract multi-scale feature information of the grayscale image; a feature fusion module, configured to perform intra-modal spatial interaction and inter-modal interaction on the multi-scale feature information of the y-channel image and the multi-scale feature information of the grayscale image to obtain multi-layer fusion features, wherein all fusion features except the fusion features of the last layer of the multi-layer fusion features are returned to the first multi-scale feature extraction module and the second multi-scale feature extraction module for feature extraction; A fusion head module, configured to perform image fusion based on the first N layers of fusion features in the multi-layer fusion features; The semantic segmentation head module is used to perform semantic segmentation based on the last N layers of fusion features in the multi-layer fusion features.

2. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The first multi-scale feature extraction module and the second multi-scale feature extraction module have the same structure and both include: Convolution operation layer, used to map the input image into high-dimensional features; Five Mamba feature extraction layers constitute a Unet structure for multi-scale feature extraction. The output of each Mamba feature extraction layer is input into the feature fusion module for feature fusion. The feature fusion results of the first four layers are added to the output of the Mamba feature extraction layer through a feature addition operation and then serve as the input of the next Mamba feature extraction layer. The first Mamba feature extraction layer is connected to the input of the fifth Mamba feature extraction layer through a skip connection, and the second Mamba feature extraction layer is connected to the input of the fourth Mamba feature extraction layer through a skip connection. A downsampling layer is set after the first and second Mamba feature extraction layers; and an upsampling layer is set after the third and fourth Mamba feature extraction layers.

3. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The feature fusion module includes: The Mamba feature extraction layer with spatial priority scanning processes input features using a spatial priority scanning strategy that scans within the current modality first and then scans in the same direction in the next modality. The Mamba feature extraction layer with modality priority scanning processes input features using a modality priority scanning strategy that scans the corresponding positions of the two modalities and then obtains the next position in the order of the spatial dimensions. The fusion output layer is used to add the output of the Mamba feature extraction layer of the space-first scanning and the output of the Mamba feature extraction layer of the modality-first scanning, and then perform a linear operation of the convolution layer to obtain the fusion feature.

4. The method for fusion of visible light and infrared images based on semantic perception according to claim 2 or 3, characterized in that: The Mamba feature extraction layer includes: The layer normalization unit is used to perform layer normalization on the input feature map; The first convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map z; The second convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map x; A SiLU operation unit, configured to perform a SiLU operation on the feature map z to obtain a first feature; a feature processing unit, configured to flatten the feature map x into a one-dimensional sequence in a horizontal forward direction, a horizontal reverse direction, a vertical forward direction, and a vertical reverse direction, and perform a separate state space model processing on each one-dimensional sequence, then cancel the flattening operation to obtain four outputs, and combine the four outputs to obtain a second feature; a dot product unit, configured to perform a dot product of the first feature and the second feature; The output unit is used to perform a convolutional layer linear operation on the output of the point multiplication unit, and then add it to the input feature map to obtain the output feature.

5. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The fusion head module includes a first processing unit, a second processing unit, and a third processing unit connected in sequence, wherein the first processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence; The second processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence; The third processing unit includes a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer and a Tanh activation layer connected in sequence.

6. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The semantic segmentation head module includes two fourth processing units, a fifth processing unit and a sixth processing unit connected in sequence, in, The fourth processing unit includes a 3×3 convolution layer, a batch normalization operation layer and a bilinear interpolation layer connected in sequence; The fifth processing unit includes three processing blocks connected in sequence, each processing block including a 3×3 convolutional layer, a batch normalization operation layer, and a ReLU activation layer connected in sequence; The sixth processing unit includes a 1×1 convolution layer, a batch normalization operation layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence.

7. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The loss function of the fusion network model is expressed as: L all =L fuse +αL seg , where L all is the loss of the fusion network model, L fuse is the image fusion loss, L seg is the semantic segmentation loss, and α is the balance coefficient.

8. The method for fusion of visible light and infrared images based on semantic perception according to claim 7, characterized in that: The semantic segmentation loss is expressed as: L seg =OhemCELoss(I seg ,I label ), where OhmCELoss() is the image segmentation loss function, I seg is the output of the semantic segmentation head module, I label is the semantic segmentation label.

9. The method for fusion of visible light and infrared images based on semantic perception according to claim 7, characterized in that: The image fusion loss is expressed as: in, is the gradient loss, expressed as: is the content loss, expressed as: s1 and s2 are weighted coefficients, I fuse is the output of the fusion head module, I vis is a visible light image, I inf For infrared images, Indicates the Sobel operator for gradient operation, max indicates the maximum value, ||||1 indicates the L1 norm, ||||2 indicates the L2 norm, I inf (x,y) represents the brightness of the pixel with coordinates (x,y) in the infrared image, I vis (x,y) represents the brightness of the pixel at coordinates (x,y) in the visible light image.

10. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: Also includes: Extract the cb channel and cr channel of the visible light image in ycbcr format to obtain the cb channel image and cr channel image; The single-channel fused image is spliced ​​with the cb channel image and the cr channel image to obtain a ycbcr fused image; The YCbCr fused image is converted into RGB format to obtain an RGB fused image.

Images