Visible light and infrared image fusion method based on semantic perception

Abstract

The invention relates to a visible light and infrared image fusion method based on semantic perception. The method comprises the following steps: obtaining a visible light image and an infrared image which are paired; converting the infrared image into a single-channel grayscale image, converting the visible light image into a ycbcr format, and independently using a y channel to obtain a y channel image; and inputting the grayscale image and the y-channel image into a fusion network model to obtain a single-channel fusion image. According to the method, local and global information can be effectively aggregated, and the semantic perception capability of the fusion network is endowed.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a visible light and infrared image fusion method based on semantic perception. Background Art

[0002] In outdoor scenes, targets are easily disturbed by complex backgrounds, and a single source of information is difficult to effectively identify targets. The introduction of multimodal information is an important trend in the development of current detection technology. Visible light images usually have high resolution and can obtain rich texture details and background information under sufficient lighting conditions, but their imaging methods are seriously affected by lighting conditions and occlusion. Extreme weather such as wind, sand, rain and snow often occur in outdoor scenes. In addition, there are dense trees, complex and diverse terrain, and the night light is obviously insufficient. It is often difficult to collect effective information about the target by visible light images alone. Infrared thermal imaging is not affected by lighting and occlusion, and can make up for the limitations of visible light imaging, but infrared images lack texture details. In recent years, visible light images have been most widely used in the field of target detection, and the development of infrared thermal imaging technology has also been relatively mature. Fusion of visible light images with infrared images can simultaneously obtain fused images with rich textures and obvious contrast, which can better meet the application needs in outdoor scenes. Infrared and visible light image fusion technology has been widely used in target detection, image segmentation, video surveillance and other fields.

[0003] The fusion of visible light and infrared images is essentially a technical means of image enhancement, and its purpose is to provide better data support for downstream visual tasks such as subsequent target detection and semantic segmentation. However, most of the existing visible light infrared image fusion methods mainly focus on improving the model's ability to extract and retain details in order to obtain a fused image with rich details and obvious contrast, but often ignore the model's semantic perception ability. Semantic perception enables the model to understand the overall information of the scene presented by the image and distinguish between foreground and background; models that lack semantic perception can present more complex textures locally, but may ignore the maintenance of complete semantic targets, resulting in incomplete targets or discontinuous target edges, which is not conducive to human eye perception and machine perception of downstream tasks such as target detection and semantic segmentation. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a visible light and infrared image fusion method based on semantic perception, which can effectively aggregate local and global information and give the fusion network the ability of semantic perception.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a visible light and infrared image fusion method based on semantic perception, comprising the following steps:

[0006] Acquire paired visible light and infrared images;

[0007] Convert the infrared image into a single-channel grayscale image, convert the visible light image into the ycbcr format and use the y channel alone to obtain the y channel image;

[0008] Input the grayscale image and the y channel image into a fusion network model to obtain a single channel fused image;

[0009] The fusion network model includes:

[0010] A first multi-scale feature extraction module, used to extract multi-scale feature information of the y channel image;

[0011] A second multi-scale feature extraction module, used to extract multi-scale feature information of the grayscale image;

[0012] A feature fusion module, used for performing intra-modal spatial interaction and inter-modal interaction on the multi-scale feature information of the y channel image and the multi-scale feature information of the grayscale image to obtain multi-layer fusion features, wherein all fusion features except the fusion features of the last layer of the multi-layer fusion features are returned to the first multi-scale feature extraction module and the second multi-scale feature extraction module for feature extraction;

[0013] A fusion head module, used for performing image fusion according to the first N layers of fusion features in the multi-layer fusion features;

[0014] The semantic segmentation head module is used to perform semantic segmentation according to the last N layers of fusion features in the multi-layer fusion features.

[0015] The first multi-scale feature extraction module and the second multi-scale feature extraction module have the same structure and both include:

[0016] Convolution operation layer, used to map the input image into high-dimensional features;

[0017] Five Mamba feature extraction layers constitute a Unet structure for multi-scale feature extraction, wherein the output of each Mamba feature extraction layer is input into the feature fusion module for feature fusion, and the feature fusion results of the first four layers are added to the output of the Mamba feature extraction layer through a feature addition operation and then used as the input of the next Mamba feature extraction layer; wherein the first Mamba feature extraction layer is connected to the input of the fifth Mamba feature extraction layer through a skip connection, and the second Mamba feature extraction layer is connected to the input of the fourth Mamba feature extraction layer through a skip connection; a downsampling layer is provided after the first Mamba feature extraction layer and the second Mamba feature extraction layer; an upsampling layer is provided after the third Mamba feature extraction layer and the fourth Mamba feature extraction layer.

[0018] The feature fusion module includes:

[0019] The Mamba feature extraction layer of space-first scanning processes the input features using a space-first scanning strategy, where the space-first scanning strategy is: first scanning in the current modality, and then scanning in the same direction in the next modality;

[0020] The Mamba feature extraction layer of modality-first scanning processes the input features using a modality-first scanning strategy, wherein the modality-first scanning strategy is: after scanning the corresponding positions of the two modalities, the next position is obtained in the scanning order of the spatial dimension;

[0021] The fusion output layer is used to add the output of the Mamba feature extraction layer of the space-first scanning and the output of the Mamba feature extraction layer of the modality-first scanning, and then perform a linear operation of the convolution layer to obtain the fusion feature.

[0022] The Mamba feature extraction layer includes:

[0023] A layer normalization unit is used to perform layer normalization on the input feature map;

[0024] The first convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map z;

[0025] The second convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map x;

[0026] A SiLU operation unit, used for performing a SiLU operation on the feature map z to obtain a first feature;

[0027] A feature processing unit, used for flattening the feature graph x into a one-dimensional sequence in a horizontal forward direction, a horizontal reverse direction, a vertical forward direction and a vertical reverse direction by flattening operations, and performing a separate state space model processing on each one-dimensional sequence, and then canceling the flattening operation to obtain four outputs, and combining the four outputs to obtain a second feature;

[0028] A dot product unit, used for performing a dot product of the first feature and the second feature;

[0029] The output unit is used to perform a convolutional layer linear operation on the output of the point multiplication unit, and then add it to the input feature map to obtain the output feature.

[0030] The fusion head module includes a first processing unit, a second processing unit and a third processing unit connected in sequence, wherein:

[0031] The first processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence;

[0032] The second processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence;

[0033] The third processing unit includes a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer and a Tanh activation layer connected in sequence.

[0034] The semantic segmentation head module includes two fourth processing units, a fifth processing unit and a sixth processing unit connected in sequence, wherein:

[0035] The fourth processing unit includes a 3×3 convolution layer, a batch normalization operation layer and a bilinear interpolation layer connected in sequence;

[0036] The fifth processing unit includes three processing blocks connected in sequence, each of which includes a 3×3 convolution layer, a batch normalization operation layer, and a ReLU activation layer connected in sequence;

[0037] The sixth processing unit includes a 1×1 convolution layer, a batch normalization operation layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence.

[0038] The loss function of the fusion network model is expressed as: L all =L fuse +αL seg , where L all is the loss of the fusion network model, L fuse is the image fusion loss, L seg is the semantic segmentation loss, and α is the balance coefficient.

[0039] The semantic segmentation loss is expressed as: L seg =OhemCELoss(I seg ,I label ), where OhemCELoss() is the image segmentation loss function, I seg is the output of the semantic segmentation head module, I label is the semantic segmentation label.

[0040] The image fusion loss is expressed as: in, is the gradient loss, expressed as: is the content loss, expressed as: s1 and s2 are weighting coefficients, I fuse is the output of the fusion head module, I vis is the visible light image, I inf For infrared images, represents the Sobel operator to find the gradient operation, max represents the maximum value, || ||1 represents the L1 norm, || ||2 represents the L2 norm, I inf (x, y) represents the brightness of the pixel with coordinates (x, y) in the infrared image. vis (x,y) represents the brightness of the pixel with coordinates (x,y) in the visible light image.

[0041] The method for fusing visible light and infrared images based on semantic perception further includes:

[0042] Extract the cb channel and cr channel of the visible light image in ycbcr format to obtain the cb channel image and the cr channel image;

[0043] The single-channel fused image is spliced ​​with the cb channel image and the cr channel image to obtain a ycbcr fused image;

[0044] The YCbCr fused image is converted into RGB format to obtain an RGB fused image.

[0045] Beneficial Effects

[0046] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention uses multi-scale feature extraction to extract multi-scale features of each modality for infrared images and visible light images respectively, and uses U-shaped networks and jump connections to avoid forgetting key spatial information, and enhances the model's ability to extract long-distance dependencies with linear time complexity, and drives the feature extraction of the entire Unet with semantic segmentation tasks, thereby improving the model's detail extraction and semantic perception capabilities, and promoting the fusion head module to obtain features with sufficient spatial details and more conducive to semantic perception. The fusion module used in the present invention adopts two strategies of joint space priority scanning and modality priority scanning to achieve more sufficient and efficient feature aggregation of the two modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a method for fusion of visible light and infrared images based on semantic perception according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the structure of the fusion network model in an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of a Mamba feature extraction layer in an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a feature fusion module (horizontal forward scanning) in an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a fusion head module in an embodiment of the present invention;

[0052] Figure 6 Schematic diagram of a semantic segmentation head module in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0054] The embodiments of the present invention relate to a method for fusing visible light and infrared images based on semantic perception. In order to solve the problem of optimizing the performance of two tasks in two independent networks, the present embodiment performs joint training in an end-to-end network, and drives the feature extraction of the entire Unet with the semantic segmentation task, thereby promoting the fusion head module to obtain features with sufficient spatial details and more conducive to semantic perception. Similarly, the fusion task promotes the full extraction of shallow detail features, thereby providing favorable support for subsequent semantic segmentation tasks. In order to solve the problem of simplifying the network structure and computational complexity, the Cnn-Mamba Unet of the present embodiment uses the most typical mamba block in the state space model to extract features, thereby enhancing the global perception ability of the model with linear complexity and achieving efficient feature extraction.

[0055] like Figure 1 As shown, the visible light and infrared image fusion method based on semantic perception of this embodiment includes the following steps:

[0056] Step 1, obtaining a pair of visible light images and infrared images;

[0057] Step 2, convert the infrared image into a single-channel grayscale image, convert the visible light image into the ycbcr format and use the y channel alone to obtain a y channel image;

[0058] Step 3, inputting the grayscale image and the y channel image into a fusion network model to obtain a single channel fused image;

[0059] Step 4, extracting the cb channel and cr channel of the visible light image in the ycbcr format to obtain a cb channel image and a cr channel image;

[0060] Step 5, splicing the single-channel fused image with the cb channel image and the cr channel image to obtain a ycbcr fused image;

[0061] Step 6: Convert the YCbCr fused image into RGB format to obtain an RGB fused image.

[0062] like Figure 2 As shown, the fusion network model in this embodiment includes:

[0063] A first multi-scale feature extraction module, used to extract multi-scale feature information of the y channel image;

[0064] A second multi-scale feature extraction module, used to extract multi-scale feature information of the grayscale image;

[0065] A feature fusion module, used for performing intra-modal spatial interaction and inter-modal interaction on the multi-scale feature information of the y channel image and the multi-scale feature information of the grayscale image to obtain multi-layer fusion features, wherein all fusion features except the fusion features of the last layer of the multi-layer fusion features are returned to the first multi-scale feature extraction module and the second multi-scale feature extraction module for feature extraction;

[0066] A fusion head module, used for performing image fusion according to the first N layers of fusion features in the multi-layer fusion features;

[0067] The semantic segmentation head module is used to perform semantic segmentation according to the last N layers of fusion features in the multi-layer fusion features.

[0068] In this embodiment, the first multi-scale feature extraction module serves as a visible light branch to extract multi-scale features from a visible light image (Visible Image), and the second multi-scale feature extraction module serves as an infrared branch to extract multi-scale features from an infrared image (Infrared Image).

[0069] The first multi-scale feature extraction module and the second multi-scale feature extraction module have the same structure and both include:

[0070] Convolution operation layer, used to map the input image into high-dimensional features;

[0071] Five Mamba feature extraction layers constitute a Unet structure for multi-scale feature extraction, wherein the output of each Mamba feature extraction layer is input into the feature fusion module for feature fusion, and the feature fusion results of the first four layers are added to the output of the Mamba feature extraction layer through a feature addition operation and then used as the input of the next Mamba feature extraction layer; wherein the first Mamba feature extraction layer is connected to the input of the fifth Mamba feature extraction layer through a skip connection, and the second Mamba feature extraction layer is connected to the input of the fourth Mamba feature extraction layer through a skip connection; a downsampling layer is provided after the first Mamba feature extraction layer and the second Mamba feature extraction layer; an upsampling layer is provided after the third Mamba feature extraction layer and the fourth Mamba feature extraction layer.

[0072] Taking the visible light branch as an example, the input visible light source image is first mapped to high-dimensional features through a 3×3 convolution kernel, a convolution operation with padding of 1 and a stride of 1, and a PreLU activation function. Then, the features are extracted through a 5-layer Mamba block (i.e., the Mamba feature extraction layer). The output of each layer is sent to the feature fusion module FMamba for feature fusion. The fused features of the first 4 layers are returned to the Cnn-Mamba Unet through the element-add operation to participate in subsequent feature extraction. Among them, the 1st and 2nd layers of Mamba blocks are respectively connected to the Down block for double downsampling, and the 3rd and 4th layers of Mamba blocks are respectively connected to the Up block for double upsampling. The Down block consists of a 3×3 convolution kernel, a convolution operation with padding of 0 and a stride of 1, and a PreLU activation function. The Up block consists of a 3×3 convolution kernel, a deconvolution operation with padding of 0 and a stride of 1, and a PreLU activation function.

[0073] like Figure 3 As shown in the figure, the Mamba block adopts the four-direction (Scan 1: horizontal forward scan, Scan 2: horizontal reverse scan, Scan 3: vertical forward scan, Scan 4: vertical reverse scan) sequence modeling method proposed in VMamba to increase spatial perception capabilities. It includes:

[0074] A layer normalization unit is used to perform layer normalization on the input feature map;

[0075] The first convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map z;

[0076] The second convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map x;

[0077] A SiLU operation unit, used for performing a SiLU operation on the feature map z to obtain a first feature;

[0078] A feature processing unit, used for flattening the feature graph x into a one-dimensional sequence in a horizontal forward direction, a horizontal reverse direction, a vertical forward direction and a vertical reverse direction by flattening operations, and performing a separate state space model processing on each one-dimensional sequence, and then canceling the flattening operation to obtain four outputs, and combining the four outputs to obtain a second feature;

[0079] A dot product unit, used for performing a dot product of the first feature and the second feature;

[0080] The output unit is used to perform a convolutional layer linear operation on the output of the point multiplication unit, and then add it to the input feature map to obtain the output feature.

[0081] For the feature map of the input Mamba block (size H×W×C), layer normalization (LN) is first performed, and then linearly mapped to different feature maps x and feature maps z through two parallel 1×1 convolutional layers. The feature map labeled x is flattened into a one-dimensional sequence (size HW×C) in four directions through the flatten operation. These one-dimensional sequences are processed by separate SSM blocks, and then unflattened to obtain four outputs, which are combined through the element-add operation to obtain a new output. Then, it is dot-multiplied with the feature map z after SiLU, and finally added to the original features of the input module after Linear to obtain the final output (size H×W×C).

[0082] It is not difficult to find that in this embodiment, the first multi-scale feature extraction module and the second multi-scale feature extraction module both use Cnn-Mamba Unet, which uses the Cnn-Mamba architecture to extract multi-scale information from the source image, and uses a U-type network and jump connections to avoid forgetting key spatial information.

[0083] The features extracted from the 5-layer Mamba block have different scales, which are 1, 1 / 2, 1 / 4, 1 / 2, and 1 times the source image input size. The infrared and visible light features are spliced ​​through channels and input into the FMamba block (i.e., feature fusion module) for feature aggregation.

[0084] The Mamba block is only suitable for single input and cannot process input of two modalities. Figure 4As shown in FIG. 1 , the image is composed of two symmetrical Mamba blocks, one of which is a Mamba feature extraction layer for space-first scanning and the other is a Mamba feature extraction layer for modality-first scanning. The two Mamba blocks are distinguished by Figure 3 The Mamba used in the feature fusion module expands the original four-directional 2D scanning to four-directional 3D scanning in the Mamba block to adapt to the dual-modal input features and interact between features. Among them, the Mamba feature extraction layer with spatial priority scanning adopts a spatial priority scanning strategy to process the input features. During processing, it first scans within the current modality and then scans in the same direction in the next modality. The Mamba feature extraction layer with modality priority scanning adopts a modality priority scanning strategy to process the input features. During processing, it scans the corresponding positions of the two modalities and then obtains the next position in the scanning order of the spatial dimension. Finally, the output of the Mamba feature extraction layer with spatial priority scanning is added to the output of the Mamba feature extraction layer with modality priority scanning, and then a linear operation of the convolution layer is performed to obtain the fused feature.

[0085] It is not difficult to find that the feature fusion module of this embodiment is expanded to the input of two modalities based on the typical vision mamba module, extracts long-distance dependencies with linear time complexity, and simultaneously performs spatial interactions within each modality and interactions between modalities. Compared with the existing fusion module, the feature fusion module of this embodiment performs more comprehensive and efficient feature aggregation of the two modalities.

[0086] After obtaining the five levels of features (F_in1, F_in2, F_in3, F_in4, F_in5) output from the feature fusion module, the features of the 1st, 2nd, and 3rd layers with 1, 1 / 2, and 1 / 4 sizes are sent to the fusion head module to obtain the fused image, and the features of the 3rd, 4th, and 5th layers with 1 / 4, 1 / 2, and 1 sizes are sent to the semantic segmentation head module to obtain the semantic segmentation results.

[0087] like Figure 5As shown, the fusion head module includes a first processing unit, a second processing unit and a third processing unit connected in sequence, wherein the first processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a 3×3 convolution layer and a bilinear interpolation layer connected in sequence; the second processing unit includes a 1×1 convolution layer, a 3×3 convolution layer and a bilinear interpolation layer connected in sequence; the third processing unit includes a 3×3 convolution layer, a PReLU activation layer, a 3×3 convolution layer, a PReLU activation layer, a 3×3 convolution layer and a Tanh activation layer connected in sequence. The input of the first processing unit is feature F_in3, the input of the second processing unit is the feature after the output of the first processing unit and feature F_in2 are spliced ​​in the channel dimension, and the input of the third processing unit is the feature after the output of the second processing unit and feature F_in1 are spliced ​​in the channel dimension.

[0088] like Figure 6 As shown, the semantic segmentation head module includes two fourth processing units, a fifth processing unit and a sixth processing unit connected in sequence, wherein the fourth processing unit includes a 3×3 convolution layer, a batch normalization operation layer and a bilinear interpolation layer connected in sequence; the fifth processing unit includes three processing blocks connected in sequence, and the processing block includes a 3×3 convolution layer, a batch normalization operation layer and a ReLU activation layer connected in sequence; the sixth processing unit includes a 1×1 convolution layer, a batch normalization operation layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence. The input of the first fourth processing unit is feature F_in3, the input of the second fourth processing unit is the feature after the output of the first fourth processing unit and the feature F_in4 are spliced ​​in the channel dimension, the input of the fifth processing unit is the feature after the output of the second fourth processing unit and the feature F_in5 are spliced ​​in the channel dimension, and the input of the sixth processing unit is the output of the fifth processing unit.

[0089] In the training phase, the fusion network model in this embodiment is an end-to-end model, and the training input is a pair of infrared images, visible light images and semantic segmentation labels, while optimizing the fusion task and semantic segmentation task. The overall loss function is as follows:

[0090] L all =L fuse +αL seg

[0091] Among them, L all is the loss of the fusion network model, L fuse is the image fusion loss, L seg is the semantic segmentation loss, and α is the balance coefficient.

[0092] The semantic segmentation loss calculation uses the common image segmentation loss function in semantic segmentation tasks (Onlinehardexample miningCrossEntropyLoss, OhemCELoss), which is expressed as follows:

[0093] L seg =OhemCELoss(I seg ,I label )

[0094] Among them, I seg is the output of the semantic segmentation head module, I label is the semantic segmentation label.

[0095] Image fusion loss includes gradient loss and content loss The fusion image generated by the gradient loss guidance obtains all the texture details in the two source images. The fusion image generated by the content loss guidance maintains the consistency of content with the source image. According to the characteristics that the visible light source image has background texture details and the infrared source image has a more prominent foreground contrast, the fusion image is constrained to be consistent with the visible light source image in terms of pixels, and the areas with higher brightness in the infrared source image tend to be consistent with the infrared source image pixels. fuse , and is defined as follows:

[0096]

[0097]

[0098]

[0099]

[0100] Among them, I fuse is the output of the fusion head module, I vis is the visible light image, I inf For infrared images. represents the Sobel operator to find the gradient operation, max represents the maximum value, || ||1 represents the L1 norm, and || ||2 represents the L2 norm. s1 and s2 are the gradient loss respectively. and content loss The weighting coefficient of .

[0101] It is not difficult to find that the present invention uses multi-scale feature extraction to extract multi-scale features of each modality for infrared images and visible light images respectively, and uses U-type networks and jump connections to avoid forgetting key spatial information, and enhances the model's ability to extract long-distance dependencies with linear time complexity, and uses semantic segmentation tasks to drive the feature extraction of the entire Unet, thereby improving the model's detail extraction and semantic perception capabilities, and promoting the fusion head module to obtain features with sufficient spatial details and more conducive to semantic perception.

Claims

1. A visible light and infrared image fusion method based on semantic perception, characterized in that: The following steps are involved: Acquire paired visible light and infrared images; Convert the infrared image into a single-channel grayscale image, convert the visible light image into the ycbcr format and use the y channel alone to obtain the y channel image; Input the grayscale image and the y channel image into a fusion network model to obtain a single channel fused image; The fusion network model includes: A first multi-scale feature extraction module, used to extract multi-scale feature information of the y channel image; A second multi-scale feature extraction module, used to extract multi-scale feature information of the grayscale image; A feature fusion module, used for performing intra-modal spatial interaction and inter-modal interaction on the multi-scale feature information of the y channel image and the multi-scale feature information of the grayscale image to obtain multi-layer fusion features, wherein all fusion features except the fusion features of the last layer of the multi-layer fusion features are returned to the first multi-scale feature extraction module and the second multi-scale feature extraction module for feature extraction; A fusion head module, used for performing image fusion according to the first N layers of fusion features in the multi-layer fusion features; The semantic segmentation head module is used to perform semantic segmentation according to the last N layers of fusion features in the multi-layer fusion features.

2. The visible light and infrared image fusion method based on semantic perception according to claim 1 is characterized in that: The first multi-scale feature extraction module and the second multi-scale feature extraction module have the same structure and both include: Convolution operation layer, used to map the input image into high-dimensional features; Five Mamba feature extraction layers constitute a Unet structure for multi-scale feature extraction, wherein the output of each Mamba feature extraction layer is input into the feature fusion module for feature fusion, and the feature fusion results of the first four layers are added to the output of the Mamba feature extraction layer through a feature addition operation and then used as the input of the next Mamba feature extraction layer; wherein the first Mamba feature extraction layer is connected to the input of the fifth Mamba feature extraction layer through a skip connection, and the second Mamba feature extraction layer is connected to the input of the fourth Mamba feature extraction layer through a skip connection; a downsampling layer is provided after the first Mamba feature extraction layer and the second Mamba feature extraction layer; an upsampling layer is provided after the third Mamba feature extraction layer and the fourth Mamba feature extraction layer.

3. The visible light and infrared image fusion method based on semantic perception according to claim 1 is characterized in that: The feature fusion module includes: The Mamba feature extraction layer of space-first scanning uses a space-first scanning strategy to process input features. The space-first scanning strategy is: first scan in the current modality, and then scan in the same direction in the next modality; the Mamba feature extraction layer of modality-first scanning uses a modality-first scanning strategy to process input features. The modality-first scanning strategy is: scan the corresponding positions of the two modalities and then obtain the next position in the scanning order of the spatial dimension; The fusion output layer is used to add the output of the Mamba feature extraction layer of the space-first scanning and the output of the Mamba feature extraction layer of the modality-first scanning, and then perform a linear operation of the convolution layer to obtain the fusion feature.

4. The visible light and infrared image fusion method based on semantic perception according to claim 2 or 3, characterized in that: The Mamba feature extraction layer includes: A layer normalization unit is used to perform layer normalization on the input feature map; The first convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map z; The second convolutional layer linear unit is used to map the feature map after the layer normalization operation to the feature map x; A SiLU operation unit, used for performing a SiLU operation on the feature map z to obtain a first feature; A feature processing unit, used for flattening the feature graph x into a one-dimensional sequence in a horizontal forward direction, a horizontal reverse direction, a vertical forward direction and a vertical reverse direction by flattening operations, and performing a separate state space model processing on each one-dimensional sequence, and then canceling the flattening operation to obtain four outputs, and combining the four outputs to obtain a second feature; A dot product unit, used for performing a dot product of the first feature and the second feature; The output unit is used to perform a convolutional layer linear operation on the output of the point multiplication unit, and then add it to the input feature map to obtain the output feature.

5. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The fusion head module includes a first processing unit, a second processing unit, and a third processing unit connected in sequence, wherein the first processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence; The second processing unit includes a 1×1 convolution layer, a 3×3 convolution layer, and a bilinear interpolation layer connected in sequence; The third processing unit includes a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer, a PReLU activation layer, a 3×3 convolutional layer and a Tanh activation layer connected in sequence.

6. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: The semantic segmentation head module includes two fourth processing units, a fifth processing unit and a sixth processing unit connected in sequence, in, The fourth processing unit includes a 3×3 convolution layer, a batch normalization operation layer and a bilinear interpolation layer connected in sequence; The fifth processing unit includes three processing blocks connected in sequence, each of which includes a 3×3 convolution layer, a batch normalization operation layer, and a ReLU activation layer connected in sequence; The sixth processing unit includes a 1×1 convolution layer, a batch normalization operation layer, a ReLU activation layer and a 3×3 convolution layer connected in sequence.

7. The visible light and infrared image fusion method based on semantic perception according to claim 1 is characterized in that: The loss function of the fusion network model is expressed as: L all =L fuse +αL seg , where L all is the loss of the fusion network model, L fuse is the image fusion loss, L seg is the semantic segmentation loss, and α is the balance coefficient.

8. The visible light and infrared image fusion method based on semantic perception according to claim 7 is characterized in that: The semantic segmentation loss is expressed as: L seg =OhemCELoss(I seg ,I label ), where OhemCELoss() is the image segmentation loss function, I seg is the output of the semantic segmentation head module, I label is the semantic segmentation label.

9. The method for fusion of visible light and infrared images based on semantic perception according to claim 7, characterized in that: The image fusion loss is expressed as: in, is the gradient loss, expressed as: is the content loss, expressed as: s1 and s2 are weighting coefficients, I fuse is the output of the fusion head module, I vis is the visible light image, I inf For infrared images, represents the Sobel operator to find the gradient operation, max represents the maximum value, ||||1 represents the L1 norm, || ||2 represents the L2 norm, I inf (x, y) represents the brightness of the pixel with coordinates (x, y) in the infrared image. vis (x,y) represents the brightness of the pixel with coordinates (x,y) in the visible light image.

10. The method for fusion of visible light and infrared images based on semantic perception according to claim 1, characterized in that: Also includes: Extract the cb channel and cr channel of the visible light image in ycbcr format to obtain the cb channel image and the cr channel image; The single-channel fused image is spliced ​​with the cb channel image and the cr channel image to obtain a ycbcr fused image; The YCbCr fused image is converted into RGB format to obtain an RGB fused image.

Images