Cross-modal semantic segmentation method based on double-branch multi-scale feature fusion

Abstract

The invention provides a cross-modal semantic segmentation method based on double-branch multi-scale feature fusion, and the method comprises the steps: employing a Segform as a backbone network, and carrying out the feature extraction of an infrared image and a visible light RGB image, the features are enhanced and fused by using a feature enhancement module, an effective attention enhancement module and a cross-modal feature fusion module designed in the invention. Firstly, shallow layer features are enhanced through a feature enhancement module, and a coordinate attention mechanism is introduced to enhance expression of detail features. And then, an effective attention enhancement module and a cross-modal feature fusion module are utilized to measure weights in the aspects of row and column vectors, complementary feature information of two modals is enhanced, and cross-modal features are fused. And finally, the image is reconstructed by using an all-ML (Multiplayer Perceptron) decoder of a lightweight class, and a prediction segmentation mask is obtained. According to the method, the segmentation precision at night or in a light change environment can be improved, the problems of inaccurate target edge contour and missing segmentation are avoided, and the method has more advantages in the aspect of processing complex scenes and details.

Description

Technical Field

[0001] The present invention belongs to the field of multispectral semantic segmentation, and specifically is a cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion. Background Art

[0002] Image semantic segmentation refers to classifying pixels according to their attributes to obtain information such as the size, shape, and position of the target, thereby decomposing the visual scene into entities of different semantic categories and achieving fine-grained understanding and analysis of the image. Currently, semantic segmentation is widely used in many fields, such as autonomous driving, geological monitoring, road safety detection, military reconnaissance, etc.

[0003] Traditional image semantic segmentation methods divide regions based on the characteristics of the image itself through artificially pre-set rules, and divide pixels with similar characteristics in the image into one category, mainly including threshold-based, edge-detection-based, and region-based segmentation methods. This type of algorithm has few parameters and low complexity, but only uses the surface information of the image and cannot achieve end-to-end image semantic segmentation. When faced with more complex image segmentation tasks, in some details, the manually set segmentation rules cannot be automatically adjusted according to the image features, resulting in unsatisfactory segmentation results and poor robustness.

[0004] With the development of deep learning technology, its powerful automatic feature learning and data fitting capabilities have shown better performance and generalization capabilities in the field of image semantic segmentation. Image semantic segmentation based on deep learning mainly includes three steps: feature encoding and enhancement, pixel or region classification, and boundary optimization and fine processing. It mainly includes region-based image semantic segmentation methods, fully supervised learning image semantic segmentation methods, and weakly supervised learning image semantic segmentation methods. Image semantic segmentation methods based on deep learning can deeply mine the pixel features of images, automatically learn the high-level semantic information of images, and infer the information expressed by the image based on the scene of the image itself. With the continuous optimization of the network model, it is superior to traditional methods in segmentation accuracy and efficiency.

[0005] At present, all image semantic segmentation methods are based on single-modal visible light RGB images. Limited by its imaging mechanism, visible light cameras are difficult to capture sufficient and effective scene information under poor lighting conditions and bad weather conditions, which makes it difficult to distinguish different categories of objects with similar appearance or shape during segmentation. For objects that are too small, it is difficult to identify their specific contours. In order to overcome the above defects, scholars have gradually paid attention to cross-modal semantic segmentation using infrared and visible light RGB images in recent years. Infrared imaging sensors are sensitive to thermal radiation and can obtain target information in low-light and bad weather environments, while visible light imaging sensors can capture rich color and detail information. Most of the existing cross-modal semantic segmentation algorithms use convolutional neural networks (CNNs) as the main framework of the network model. Due to the limited receptive field of CNNs, only local feature information can be extracted, which affects the distinguishability of subsequent feature extraction, resulting in limited segmentation performance. By stacking a large number of convolutional layers, a larger receptive field and global information can be obtained, but it also increases the complexity and computational complexity of the network. Therefore, this field urgently needs a cross-modal semantic segmentation algorithm that can overcome the above defects. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] Semantic segmentation of single-modal visible light images has problems such as poor segmentation effect and low precision at night or in light-changing environments, inaccurate target edge contours, and easy missed segmentation of small targets. The present invention provides a cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion, which solves the problems raised in the above background technology.

[0008] (II) Technical solution

[0009] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0010] A cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion, the method comprising the following steps: Step A: obtaining a cross-modal sample image pair annotated with a true label, the cross-modal sample image pair consisting of an RGB image and a Thermal image. Step B: inputting the RGB image and the Thermal image into a dual-branch segformer structure for feature extraction, and using a feature enhancement module to enhance the shallow feature map of the first scale, the enhanced feature map and the feature maps of other scales are uniformly represented as F i j, j∈{1,2,3,4} is the scale index, i∈{rgb,thermal} is the image modality index. Step C: Construct an effective attention enhancement module, use the channel attention mechanism to capture feature relationships across modalities, generate channel weight vectors, enhance the representation of significant information of different modalities, and obtain the features of each scale after feature enhancement Step D: Construct a cross-modal feature fusion module to fuse the feature maps of each scale and obtain the fused feature maps of each scale. Step E: Fusion features Input into the lightweight MLP decoder to obtain the semantic segmentation result.

[0011] (III) Beneficial effects

[0012] Compared with the prior art, the present invention provides a cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion, which has the following beneficial effects:

[0013] (1) The Segformer model is used to deeply extract infrared and visible light image features, realize multi-scale image semantic representation, and express the features of infrared and visible light images more comprehensively and accurately. At the same time, the feature enhancement module is used to improve the spatial information expression ability of shallow features and solve the problem of blurred target edge contours.

[0014] (2) The effective attention enhancement module and the cross-modal fusion module are used to model the relationship between the pixels of the two modal feature maps and generate the corresponding attention matrix to obtain complementary information between the two modalities. The weights are measured from the row and column vector dimensions to better fuse the cross-modal features, improve the segmentation accuracy of the model in complex scenes, reduce the probability of missed detection, and improve the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion according to an embodiment of the present invention;

[0016] Figure 2 This is an overall block diagram of a cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion according to an embodiment of the present invention;

[0017] Figure 3 The semantic segmentation results of the comparison method of the embodiment of the present invention and the method of the present invention are shown in FIG. Figure 3 (a) is the infrared image, Figure 3 (b) is the visible light image, Figure 3 (c) is the true label value, Figure 3 (d) is the semantic segmentation result image of comparison method 1. Figure 3 (e) is the semantic segmentation result image of comparison method 2; Figure 3(f) is the semantic segmentation result image of comparison method three; Figure 3 (g) is the semantic segmentation result image of the method of the present invention;

[0018] Figure 4 It is objective evaluation index data of the semantic segmentation results of the comparison method of the embodiment of the present invention and the method of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Example

[0021] like Figure 1 As shown, an embodiment of the present invention proposes a cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion, and the method includes the following steps:

[0022] Step A: Obtain a pair of cross-modal sample images annotated with real labels, which consists of an RGB image and a Thermal image. Step B: Input the RGB image and the Thermal image into the dual-branch segformer structure for feature extraction, and use the feature enhancement module to enhance the shallow feature map of the first scale. The enhanced feature map and the feature maps of other scales are uniformly represented as F i j , j∈{1,2,3,4} is the scale index, i∈{rgb,thermal} is the image modality index. Step C: Construct an effective attention enhancement module, use the channel attention mechanism to capture feature relationships across modalities, generate channel weight vectors, enhance the representation of significant information of different modalities, and obtain the features of each scale after feature enhancement Step D: Construct a cross-modal feature fusion module to fuse the feature maps of each scale and obtain the fused feature maps of each scale. Step E: Fusion features Input into the lightweight MLP decoder to obtain the semantic segmentation result.

[0023] In this embodiment, the specific operation process of feature extraction in Segformer is as follows: Segformer is a hierarchical Transformer structure, which includes four layers of Transformer modules. The Transformer module in the encoder first uses an overlapping block embedding module to decompose the input image into overlapping image blocks; then the global context information and local detail information are captured through an efficient self-attention layer and a hybrid feedforward neural network layer. Finally, the overlapping block merging layer is used to merge into a complete feature map without overlap. The RGB image and the Thermal image are respectively input into the dual-branch segformer encoder to obtain RGB multi-scale features and Thermal multi-scale features with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image.

[0024] In this embodiment, the specific operation process of feature enhancement in the feature enhancement module is as follows: the input feature map is subjected to 1 layer of 2D fast Fourier transform, 1 1×1 convolution layer, ReLu activation function, 1 1×1 convolution layer, inverse fast Fourier transform layer and coordinate attention mechanism to obtain the enhanced first scale feature map:

[0025] FFT conv =Conv 1×1 (RELU(Conv 1×1 (BN(FFT2D(X)))))

[0026] IFFT = BN(InvFFT2D(FFT conv ))

[0027] FFT res =RES(IFFT,X)

[0028] FFT ca =CA(FFT res )

[0029] F i 1 =RES(FFT ca ,FFT res )

[0030] Where X is the shallow feature map output by the first layer transformer module, FFT2D(·) is the two-dimensional fast Fourier transform, Conv 1×1 (·) is a 1×1 convolutional layer, RELU (·) is a ReLu activation function layer, FFT convis the output after fast Fourier transform and convolution operation, InvFFT2D(·) is the two-dimensional inverse fast Fourier transform, BN(·) is the standard normalization layer, IFFT is the output of inverse fast Fourier transform, RES(·) is the residual connection operation, FFT res is the output obtained by residual connection of the result of fast Fourier transform and input features. CA(·) is the coordinate attention mechanism, FFT ca is the output after the coordinate attention operation, F i 1 It is the first scale feature map output by the feature enhancement module.

[0031] In this embodiment, the specific operation process of feature enhancement in the effective attention enhancement module is as follows: the effective attention enhancement module uses the channel attention mechanism to capture feature relationships across modalities, generate channel weight vectors, and enhance the representation of significant information of different modalities. This module is mainly composed of three branches, namely the attention relationship branch, the common feature information branch, and the complementary feature information branch. The attention relationship branch uses the global pooling layer to aggregate the infrared feature map and the visible light feature map into the channel vector of a specific category, and then uses a multi-layer perceptron composed of a fully connected layer and a sigmoid activation function to obtain the channel weight vectors of the two modalities, and finally obtains the cross-modal attention relationship weight vector by element-by-element multiplication:

[0032]

[0033]

[0034]

[0035] Where R j is the channel weight vector of the visible light RGB image, T j is the channel weight vector of infrared features, S j is the cross-modal attention relation vector, j∈{1,2,3,4} is the scale index, MLP(·) is the fully connected layer operation, Pooling(·) is the global pooling operation, It is an element-wise multiplication operation.

[0036] The common feature information branch first multiplies the input infrared feature map and visible light feature map by the attention relationship weight vector according to the channel to obtain the feature map Then the two modal feature maps are cascaded, passed through a 7×7 deep separable convolution layer, channel attention mechanism (global average pooling layer, fully connected layer, sigmoid layer), and then the split operation is used to obtain the channel weight vectors of the two modalities, which are compared with the corresponding Multiply to get the feature map with enhanced cross-modal common feature information

[0037]

[0038]

[0039]

[0040] Where a is a constant, which is a scaling factor for adjusting the weight value distribution of cross-modal attention relationship; σ(·) is the Sigmoid activation function layer, Cat(·) is the cascade operation, and DM 7×7 (·) is a 7×7 depthwise separable convolutional layer, CAM(·) is a channel attention mechanism operation, split(·) is a channel separation operation, It is an element-wise multiplication operation.

[0041] The complementary information branch first inverts the attention relationship weight vector value to obtain the channel weight vector reflecting the cross-modal complementary information, and then multiplies it with the input infrared feature map and visible light feature map by channel to obtain a feature map containing cross-modal complementary feature information. Finally, the feature map containing cross-modal complementary feature information and feature maps containing shared feature information Add pixel by pixel to get the enhanced feature map

[0042]

[0043] in represents a feature map containing cross-modal complementary feature information, Represents the enhanced feature map.

[0044] In this embodiment, the specific operation process of feature fusion in the cross-modal feature fusion module is as follows: first, the feature maps of the two modalities are and Calculate L of row (column) vector 1 norm, obtain the activity measure in the row (column) direction through softmax and k∈{row,col} represents the direction index, row represents the row direction, and col represents the column direction; then the activity measure is used as the weight value to perform weighted fusion on the feature map. Then, the activity measure is multiplied by the corresponding input feature to obtain the fused feature from the row (column) vector direction Will and The multi-dimensional feature map is obtained by cascading, and the number of channels is adjusted by using a 1×1 convolution layer. Then, a residual block consisting of a 3×3 depth-separable convolution layer with a step size of 1, a Relu activation function layer, a 1×1 convolution layer with a step size of 1, and a 1×1 convolution layer with a jump connection is passed to obtain fused feature maps at different scales.

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] where ||·|| 1 For L 1 Norm calculation, Cat(·) is a cascade operation, DW 3×3 (·) is a 1×1 convolutional layer, RELU(·) is a 3×3 depth-wise separable convolutional layer, and RELU(·) is a Relu activation function layer. and represents the activity measure, k∈{row,col} represents the direction index, row represents the row direction, col represents the column direction, Represented as the fused feature map.

[0051] In this embodiment, the specific operation process of feature reconstruction in the decoder is as follows: first, the fusion feature maps of each scale are Through an MLP layer, the number of channels is adjusted to a uniform value, and then the fused feature maps of each scale are upsampled, and the feature map size is unified to Then cascade and use another MLP layer to get the predicted segmentation result F out :

[0052]

[0053] Where MLP(·) is a fully connected layer, Upsample(·) is an upsampling operation, Cat(·) is a cascade operation, and F out Represented as semantic segmentation result.

[0054] The following further illustrates an application example of the present invention: In order to verify the effectiveness of the cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion proposed in the present invention, the public dataset MFNet was used to train and test the cross-modal semantic segmentation network with dual-branch multi-scale feature fusion. Figure 3The two daytime images and two nighttime images shown in the figure were visually compared, and three other methods were selected for comparative experimental tests. The corresponding test results are shown in Figure 3 As shown in (d)(e)(f)(g).

[0055] The MFNet dataset is a pair of visible light RGB images and infrared images collected by the InfReCR500 camera for urban street scenes. The dataset contains a total of 1569 pairs of images, including 820 pairs of daytime images and 749 pairs of nighttime images. In addition to the background category, there are 8 target categories, namely color cone, car stop, pedestrian, curve, bicycle, guardrail, vehicle and bump, and the resolution of the images is 640×320. The dataset is divided into: the training set is 784 pairs of images, including 410 pairs of daytime images and 374 pairs of nighttime images; the validation set is 394 pairs of images, including 205 pairs of daytime images and 189 pairs of nighttime images; the test set is 391 pairs of images, including 205 pairs of daytime images and 186 pairs of nighttime images.

[0056] The first comparison method is Segformer (B2) (Xie, Enze et al. "SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers." Neural Information Processing Systems (2021).), an efficient semantic segmentation method based on Transformer. Transformer blocks are used to split the input image into overlapping image blocks, with partial overlap between each image block to ensure that important image information is not lost. Then, an efficient self-attention layer and a hybrid feedforward neural network layer are used to capture features, and finally a lightweight decoder is used to reconstruct the image to obtain the semantic segmentation result. This method belongs to a unimodal semantic segmentation method.

[0057] The second comparison method is MFNet (Q.Ha, K.Watanabe, T.Karasawa, Y.Ushiku and T.Harada, "MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes." 2017IEEE / RSJInternational Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 2017), which uses two encoders with the same structure to process RGB images and infrared images respectively, and concatenates the processed features into the decoder. In the encoder, feature fusion is performed by feature concatenation, and a skip connection is added between the encoder and the decoder to finally obtain the semantic segmentation result.

[0058] The third comparison method is an edge-aware guided semantic segmentation method for multispectral scenes (Zhou, Wujie et al. "Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing." ArXivabs / 2112.05144 (2021)), which uses dilated convolutions with different expansion rates to expand the receptive field, and adds an edge label supervision network to enhance the segmentation accuracy of the target edge.

[0059] Figure 3 (d) (e) (f) (g) are respectively comparative method 1, comparative method 2, comparative method 3 and the method of the present invention. Figure 3 (a) (b) shows the fusion results of a set of infrared and visible light images. Figure 3 (c) represents the true label value of semantic segmentation; Figure 3 In the fusion results shown, in the segmentation results of the comparison method one, there are a lot of missed segmentations in the segmentation results of the night images; in the segmentation results of the comparison method two, the feature information is seriously lost and the segmentation results are incomplete; in the segmentation results of the comparison method three, the edge detail information is not fully extracted and the target contour segmentation is rough; in the segmentation results of the method of the present invention, the structural information of the segmented image is clear and complete, the complementary information of the infrared and visible light images is well preserved, the detail information is rich, and the segmentation accuracy is high.

[0060] In order to better evaluate the semantic segmentation effect, in addition to the above subjective evaluation, objective evaluation indicators are used to objectively analyze the segmentation results of different semantic segmentation methods. Two evaluation indicators are set, namely mean Intersection over Union (mIoU) and mean accuracy (mAcc). The larger the evaluation indicator value, the better the segmentation effect, and vice versa. The objective evaluation indicator data is as follows: Figure 4 As shown, from Figure 4 It can be seen from the objective evaluation data in that the result image obtained by the semantic segmentation method of the present invention is better than comparison method 1, comparison method 2 and comparison method 3 in terms of various objective index evaluations, which is consistent with the above subjective evaluation results. This shows that the method of the present invention can stably obtain high-quality semantic segmentation results.

[0061] So far, the embodiments of the present invention have been described in detail with reference to the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of the cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion of the present invention.

[0062] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion, characterized in that: The method comprises the following steps: Step A: obtaining a pair of cross-modal sample images annotated with real labels, wherein the cross-modal sample image pair consists of an RGB image and a Thermal image. Step B: inputting the RGB image and the Thermal image into a dual-branch segformer structure for feature extraction, and using a feature enhancement module to enhance the shallow feature map of the first scale, and the enhanced feature map and the feature maps of other scales are uniformly represented as j∈{1,2,3,4} is the scale index, and i∈{rgb,thermal} is the image modality index. Step C: Construct an effective attention enhancement module, use the channel attention mechanism to capture feature relationships across modalities, generate channel weight vectors, enhance the representation of significant information of different modalities, and obtain the features of each scale after feature enhancement. Step D: Construct a cross-modal feature fusion module to fuse the feature maps of each scale to obtain the fusion features of each scale. Step E: Fusion features Input into the lightweight MLP decoder to obtain the semantic segmentation result.

2. The cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion as claimed in claim 1, characterized in that: The step B includes: the Segformer is a layered Transformer structure, including four layers of Transformer modules. The Transformer module in the encoder first uses an overlapping block embedding module to decompose the input image into overlapping image blocks; then the global context information and local detail information are captured through an efficient self-attention layer and a hybrid feedforward neural network layer. Finally, the overlapping block merging layer is used to merge into a complete feature map without overlap. The RGB image and the Thermal image are respectively input into the dual-branch segformer encoder to obtain RGB multi-scale features and Thermal multi-scale features with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. The efficient self-attention layer can capture the relationship between any two elements in the sequence, and at the same time adds a scaling factor R to reduce the computational complexity. The hybrid feedforward neural network layer obtains the position information in the image space by adding a 3×3 two-dimensional convolution layer between the two linear transformation layers of the feedforward neural network layer. The overlapping block merging layer obtains feature maps of different scales by setting two different sets of hyperparameters. The hyperparameters are K=7, S=4, P=3 and K=3, S=2, P=1, where K is the overlapping image block size, S is the step size between adjacent blocks, and P is the padding size. The feature enhancement module is composed of a fast Fourier transform module with a residual structure and a coordinate attention mechanism module. The input feature map is subjected to a two-dimensional fast Fourier transform, a 1×1 convolution layer, a ReLu activation function, a 1×1 convolution layer, an inverse fast Fourier transform layer and a coordinate attention mechanism to obtain the enhanced first scale feature map, which is implemented by the following formula: FFT conv =Conv 1×1 (RELU(Conv 1×1 (BN(FFT2D(X))))) IFFT=BN(InvFFT2D(FFT conv )) FFT res =RES(IFFT,X) FFT ca =CA(FFT res ) F i 1 =RES(FFT ca ,FFT res ) Where X is the shallow feature map output by the first layer transformer module, FFT2D() is the two-dimensional fast Fourier transform, Conv 1×1 () is a 1×1 convolution layer, RELU() is a ReLu activation function layer, FFTconv is the output after fast Fourier transform and convolution operation, InvFFT2D() is a two-dimensional inverse fast Fourier transform, BN() is a standard normalization layer, IFFT is the output of the inverse fast Fourier transform, RES() is a residual connection operation, FFT res The output obtained by performing a residual connection between the result of the fast Fourier transform and the input feature. CA() is the coordinate attention mechanism, FFT ca is the output after the coordinate attention operation, F i 1 It is the first scale feature map output by the feature enhancement module.

3. The cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion as claimed in claim 1, characterized in that: The step C includes: the effective attention enhancement module uses the channel attention mechanism to capture feature relationships across modalities, generates channel weight vectors, and enhances the representation of significant information of different modalities. The module is mainly composed of three branches, namely, the attention relationship branch, the common feature information branch, and the complementary feature information branch. The attention relationship branch aims to obtain the channel weights of the two modal features using the channel attention mechanism. First, the infrared feature map and the visible light feature map are aggregated into the channel vector of a specific category using the global pooling layer, and then the channel weight vectors of the two modalities are obtained using a multi-layer perceptron composed of a fully connected layer and a sigmoid activation function. Finally, the cross-modal attention relationship weight vector is obtained by element-by-element multiplication, which is implemented by the following formula: Where R j is the channel weight vector of the visible light RGB image, T j is the channel weight vector of infrared features, S j is the cross-modal attention relationship vector, j∈{1,2,3,4} is the scale index, MLP() is the fully connected layer operation, Pooling() is the global pooling operation, It is an element-wise multiplication operation. The common feature information branch aims to capture the common feature information between the two modalities. First, the input infrared feature map and visible light feature map are multiplied by the attention relationship weight vector by channel to obtain the feature map Then the two modal feature maps are cascaded, passed through a 7×7 deep separable convolution layer, channel attention mechanism (global average pooling layer, fully connected layer, sigmoid layer), and then the split operation is used to obtain the channel weight vectors of the two modalities, which are compared with the corresponding Multiply to get the feature map with enhanced cross-modal common feature information This is achieved through the following formula: Where a is a constant, which is a scaling factor for adjusting the weight value distribution of cross-modal attention relationship; σ() is the Sigmoid activation function layer, Cat() is the cascade operation, and DM 7×7 () is a 7×7 depth-wise separable convolutional layer, CAM() is a channel attention mechanism operation, split() is a channel separation operation, It is an element-wise multiplication operation. The complementary information branch aims to obtain complementary feature information between the two modalities. First, the attention relationship weight vector value is inverted to obtain the channel weight vector reflecting the cross-modal complementary information, and then it is multiplied with the input infrared feature map and visible light feature map by channel to obtain a feature map containing cross-modal complementary feature information. Finally, the feature map containing cross-modal complementary feature information and feature maps containing shared feature information Add pixel by pixel to get the enhanced feature map This is achieved through the following formula: in Represented as a feature map containing cross-modal complementary feature information, Represented as the enhanced feature map.

4. The cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion as claimed in claim 1, characterized in that: The step D comprises: the cross-modal feature fusion module performs cross-modal feature fusion on the feature maps of the two modalities. and Calculate the L1 norm of the row (column) vector and obtain the activity measure in the row (column) direction through softmax and k∈{row,col} represents the direction index, row represents the row direction, and col represents the column direction; then the activity measure is used as the weight value to perform weighted fusion on the feature map. Then, the activity measure is multiplied by the corresponding input feature to obtain the fused feature from the row (column) vector direction Will and The multi-dimensional feature map is obtained by cascading, and the number of channels is adjusted by using a 1×1 convolution layer. Then, a residual block consisting of a 3×3 depth-separable convolution layer with a step size of 1, a Relu activation function layer, a 1×1 convolution layer with a step size of 1, and a 1×1 convolution layer with a jump connection is passed to obtain fused feature maps at different scales. This is achieved through the following formula: Where 1 is the L1 norm calculation, Cat() is the cascade operation, and DW 3×3 () is a 1×1 convolutional layer, RELU() is a 3×3 depth-wise separable convolutional layer, and RELU() is a Relu activation function layer.

5. The cross-modal semantic segmentation method based on dual-branch multi-scale feature fusion as claimed in claim 1, characterized in that: The step E includes: the decoder uses a lightweight All-MLP decoder to avoid redundant calculations. First, the fusion feature maps of each scale are Through an MLP layer, the number of channels is adjusted to a uniform value, and then the fused feature maps of each scale are upsampled, and the feature map size is unified to Then cascade and use another MLP layer to get the predicted segmentation result F out . This is achieved through the following formula: Among them, MLP() is a fully connected layer, Upsample() is an upsampling operation, Cat() is a cascade operation, and F out Represented as the predicted segmentation result.

Images