A low-illumination multi-modal scene understanding method based on dynamic feature selection complementation

Abstract

The application discloses a low-illumination multi-modal scene understanding method based on dynamic feature selection complementarity, and belongs to the technical field of artificial intelligence, wherein according to the complementary characteristics of multi-modal feature information, a parallel feature extraction network based on a Transformer is constructed; a short-distance selective attention module is constructed to reduce information interference between the same modalities and ensure the reservation of complementary information under different modalities; a channel selective self-attention module is constructed to enhance the information integrity of the feature channel dimension; and a lightweight fully-connected multi-layer perception decoding module is constructed to receive the feature information of the encoding network and perform feature reconstruction on feature maps of different scales; the application can fully utilize the complementary advantages of multi-modal feature information, avoid the insufficient low-illumination scene understanding ability caused by single-modal information, and be applied to scene analysis of unmanned systems and unmanned navigation systems under complex low-illumination scenes.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for understanding low-light multimodal scenes based on dynamic feature selection complementarity. Background Technology

[0002] With the rapid development of artificial intelligence technology, the requirements for unmanned systems in perceiving and understanding complex scenes are becoming increasingly stringent. This is especially true for autonomous vehicles, intelligent robots, and drones, which face challenges in complex low-light environments. Single visible light image sensors are easily affected by complex low-light conditions. Therefore, combining them with thermal image sensors can compensate for the shortcomings of visible light sensors, ensuring that unmanned systems maintain consistent perception capabilities in daylight, nighttime, and adverse weather conditions. Visible light images provide rich semantic information, while thermal images provide stable scene images. Researching efficient fusion methods for visible light and thermal images can effectively improve the stability of unmanned systems in understanding scenes under complex low-light conditions.

[0003] Xie et al., in their paper "SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers," proposed a novel semantic segmentation framework. The authors first proposed a hierarchical Transformer encoder that can acquire multi-scale features without positional encoding, adapting to inputs of various resolutions. Then, they proposed a lightweight, fully connected multilayer perceptron decoder that generates a segmentation mask by fusing information from different levels and combining local and global attention. While this method achieves excellent semantic segmentation performance, it neglects the complementary nature of multimodal feature information. The proposed fusion method cannot effectively utilize the complementary information between modalities and is susceptible to noise interference from different intra- and intermodal information, thus limiting the performance of multimodal semantic segmentation to some extent. Furthermore, it suffers from parameter redundancy and increased computational cost, making it unsuitable for deployment on unmanned systems. Summary of the Invention

[0004] The purpose of this invention is to provide a low-light multimodal scene understanding method based on dynamic feature selection complementarity. This method efficiently utilizes the complementary characteristics of multimodal feature information and employs different fusion strategies to construct an efficient multimodal joint representation network. It solves the problems of insufficient utilization of multimodal feature information and low fusion efficiency, while avoiding the redundancy of parameters and increased computational load caused by inefficient network modules. This invention provides a low-light multimodal scene understanding method based on dynamic feature selection complementarity, which can be applied to autonomous driving on urban roads at night and unmanned navigation systems in dimly lit underground spaces.

[0005] The technical solution to achieve the purpose of this invention is: a low-light multimodal scene understanding method based on dynamic feature selection complementarity, comprising the following steps:

[0006] Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; divide the unified images into training and testing datasets according to a ratio of 784 / 393, perform data augmentation on the training dataset to form the network training dataset, and proceed to Step 2.

[0007] Step 2: Construct a multimodal selective attention semantic segmentation network:

[0008] The multimodal selective attention semantic segmentation network includes a visible light encoding network, a thermal image encoding network, a short-range selective attention module, a channel-selective self-attention module, and a lightweight decoding network. The visible light encoding network and the thermal image encoding network are both based on Transformer networks pre-trained on the ImageNet dataset, serving as encoding networks for feature extraction. The short-range selective attention module is used to preserve multimodal complementary feature information. The channel-selective self-attention module is used to enhance the information integrity of feature channel dimensions. The decoding network consists of a lightweight fully connected multilayer perceptron, used to receive feature information from the encoding network and reconstruct features from feature maps at different scales, proceeding to step 3.

[0009] Step 3: Input the network training dataset into the multimodal selective attention semantic segmentation network, and train the multimodal selective attention semantic segmentation network using the network training dataset to obtain the trained multimodal selective attention semantic segmentation network model:

[0010] S31. The visible light coding network feature extraction is divided into four stages, and four visible light features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32); Correspondingly, the thermal image coding network feature extraction is divided into four stages, and four thermal image features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), then proceed to S32;

[0011] S32. Input the visible light features and thermal image features extracted from each stage into the short-range selective attention module to obtain short-range fusion features. Then input the short-range fusion features into the channel selective self-attention module to obtain the visible light and thermal image interactive fusion features of this stage, and proceed to S33.

[0012] S33. The visible light and thermal image fusion feature with a scale of (H / 4)×(W / 4) obtained in the first stage of S32 and the visible light and thermal image fusion feature with a scale of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the lightweight fully connected multilayer perceptron decoding module to obtain four decoding features with a pixel size of (H / 4)×(W / 4). The four decoding features are then concatenated to obtain the aggregated feature, and the process is transferred to S34.

[0013] S34. The aggregated features output in S33 are processed through a multilayer perceptron to perform channel dimensionality reduction, resulting in dimensionality-reduced features. The cross-entropy loss between the dimensionality-reduced features and the corresponding image labels in the MFNet dataset is calculated, and the network parameters of the multimodal selective attention semantic segmentation network are updated accordingly to obtain the trained multimodal selective semantic segmentation model. Proceed to step 4.

[0014] Step 4: Input the test dataset into the trained multimodal selective attention semantic segmentation model, output the prediction result corresponding to each sample in the test set, and test the accuracy of the trained multimodal selective attention semantic segmentation model.

[0015] Compared with the prior art, the advantages of this invention are:

[0016] (1) This invention proposes a low-light multimodal scene understanding method based on dynamic feature selection complementarity. It can efficiently utilize the complementary characteristics of multimodal feature information and construct an efficient multimodal fusion network by adopting different fusion strategies. It solves the problems of insufficient utilization of multimodal feature information and low fusion efficiency, and significantly reduces the number of parameters and computation.

[0017] (2) This invention proposes a short-range selective attention module that integrates intramodal and intermodal selective attention to reduce information interference in the same modality and ensure the preservation of complementary information and the suppression of noise information in different modalities.

[0018] (3) This invention proposes a channel-selective self-attention module, which enhances the information integrity of the feature channel dimension and improves generalization and robustness. Attached Figure Description

[0019] Figure 1 This is a model diagram of a low-light multimodal scene understanding method based on dynamic feature selection complementarity.

[0020] Figure 2 The image shows the experimental results of an urban road scene on the MFNet dataset.

[0021] Figure 3 The image shows the experimental results of urban road scenes from the FMB dataset. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below:

[0023] Combination Figure 1 A low-light multimodal scene understanding method based on dynamic feature selection complementarity includes the following steps:

[0024] Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; divide the uniformly sized images into training and testing datasets according to a ratio of 784 / 393, perform data augmentation on the training dataset to form the network training dataset, and proceed to Step 2.

[0025] Step 2: Construct a multimodal selective attention semantic segmentation network:

[0026] The multimodal selective attention semantic segmentation network includes a visible light encoding network, a thermal image encoding network, a short-range selective attention module, a channel-selective self-attention module, and a lightweight decoding network. The visible light encoding network and the thermal image encoding network are both based on Transformer networks pre-trained on the ImageNet dataset, serving as encoding networks for feature extraction. The short-range selective attention module is used to preserve multimodal complementary feature information. The channel-selective self-attention module is used to enhance the information integrity of feature channel dimensions. The decoding network consists of a lightweight fully connected multilayer perceptron, used to receive feature information from the encoding network and reconstruct features from feature maps at different scales, proceeding to step 3.

[0027] Step 3: Input the network training dataset into the multimodal selective attention semantic segmentation network, and train the multimodal selective attention semantic segmentation network using the network training dataset to obtain the trained multimodal selective attention semantic segmentation network model:

[0028] S31. The visible light coding network feature extraction is divided into four stages, and four visible light features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32); Correspondingly, the thermal image coding network feature extraction is divided into four stages, and four thermal image features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), and then proceed to S32.

[0029] S32. Input the visible light features and thermal image features extracted at each stage into the short-range selective attention module to obtain short-range fusion features. Then input the short-range fusion features into the channel-selective self-attention module to obtain the features after interactive fusion of visible light and thermal images at this stage.

[0030] The visible light and thermal image features extracted in the first stage of S31 are input into the short-range selective attention module to obtain fused features. The fused features are then input into the channel-selective self-attention module to obtain features with a scale of (H / 4)×(W / 4) after interactive fusion of visible light and thermal images. The visible light and thermal image features extracted in the second stage are input into the short-range selective attention module to obtain fused features. The fused features are then input into the channel-selective self-attention module to obtain features with a scale of (H / 8)×(W / 8). The visible light and thermal image features extracted in the third stage are input into the short-range selective attention module to obtain fused features. The fused features are then input into the channel-selective self-attention module to obtain features with a scale of (H / 16)×(W / 16). The visible light and thermal image features extracted in the fourth stage are input into the short-range selective attention module to obtain fused features. The fused features are then input into the channel-selective self-attention module to obtain features with a scale of (H / 32)×(W / 32).

[0031] Specifically, visible light features and thermal image features are input into a short-range selective attention module to obtain short-range fused features, as follows:

[0032] S32-1 defines the query vector mapped from visible light features as Q. r The key vector is defined as K r The value vector is defined as V r Similarly, the query vector mapped from the thermal image features is defined as Q. t The key vector is defined as K t The value vector is defined as V tThe query vector, key vector, and value vector are segmented separately, and the mean and standard deviation of the segmented query vector and key vector are calculated. Then, the segmented query vector and key vector are normalized based on their mean and standard deviation.

[0033]

[0034] In the formula, mode x∈{r,t}; r is the visible light feature; t is the thermal image feature; The segmented query vector; The key vector after segmentation; The vector of values ​​after partitioning; Partition is the partitioning function; Q' x The normalized query vector; K' x is the normalized key vector; mean is the mean function; std is the standard deviation function.

[0035] S32-2, calculate the visible light self-attention A based on the normalized query vector and the normalized key vector, respectively. rr And thermal image self-attention A tt According to visible light self-attention A rr And thermal image self-attention A tt The selective attention A' within the visible light mode was calculated respectively. rr Intramodal selective attention A' of thermal images tt :

[0036]

[0037] A' rr =Select(σ(A) rr )>α,A rr ,-∞)

[0038] A' tt =Select(σ(A) tt )>α,A tt ,-∞)

[0039] In the formula, Q' r K' is the query vector after normalization of visible light features; r The key vector is the normalized visible light feature vector; Q' t K' is the query vector after normalizing the thermal image features. t The key vector is the normalized feature vector of the thermal image; T represents the transpose; τ1 and τ2 are learnable parameters; Select(σ(A rr )>α,A rr (,-∞) is a selection mechanism, indicating that when σ(A) rrWhen )>α, return A. rr Otherwise, it returns -∞; Select(σ(A) tt )>α,A tt Similarly, σ is the sigmoid function; α is a hyperparameter, set to 0.2.

[0040] S32-3, Based on the query vector Q' normalized from visible light features r The bond vector K' after normalization of thermal image features t Calculate the attention A between visible light and thermal image modes. rt The query vector Q' is normalized based on the features of the thermal image. t The bond vector K' after normalization of visible light features r Calculate the attention A between the thermal image and the visible light mode. tr And based on the attention A between visible light and thermal image modalities rt Attention A between thermal images and visible light modes tr The selective attention A' between the visible light and thermal image modes was calculated respectively. rt Selective attention between thermal images and visible light modes A' tr :

[0041]

[0042] A' rt =Select(σ(A) rt )>α,A rt ,-∞)

[0043] A' tr =Select(σ(A) tr )>α,A tr ,-∞)

[0044] In the formula, T represents transpose; τ3 and τ4 are both learnable parameters; Select(σ(A rt )>α,A rt (,-∞) is a selection mechanism, indicating that when σ(A) rt When )>α, return A. rt Otherwise, it returns -∞; Select(σ(A) tr )>α,A tr Similarly, σ is the sigmoid function; α is a hyperparameter set to 0.2.

[0045] Based on visible light modal selective attention A' rt Intramodal selective attention A' of thermal images tt Selective attention between visible light and thermal image modalities (A') rtSelective attention between thermal images and visible light modes A' tr The short-range selective attention score S was calculated. s :

[0046] S s =Softmax(A′ rr +A′ tt +A′ rt +A′ tr )

[0047] In the formula, Softmax is the normalized exponential function.

[0048] S32-4, using the segmented value vector and short-distance selective attention score S s Calculate short-range preliminary fusion features Short-range preliminary fusion features Perform a flipping operation, then compare it with the visible light feature F. r and thermal image features F t After addition, layer normalization is performed to obtain feature F. s ":

[0049]

[0050] In the formula, This is the value vector after visible light feature segmentation; is the value vector after thermal image feature segmentation; LN is the layer normalization operation; Linear is the linear mapping; Reverse is the flipping operation.

[0051] Feature F s "Short-range fusion features F are obtained by using a multilayer perceptron and performing layer normalization." s :

[0052] F s =LN(MLP(F s ")+F s ")

[0053] In the formula, MLP represents a multilayer perceptron.

[0054] Specifically, the short-range fusion features are input into the channel-selective self-attention module to obtain the features after interactive fusion of visible light and thermal images at this stage, as follows:

[0055] S32-A, which uses the short-range fusion features F output by the short-range selective attention module. s After linear mapping, the query vector Q is obtained. s Key vector K s Sum vector V sand query vector Q s and bond vector K s After normalization, the normalized query vector Q' is obtained. s and bond vector K' s Then, based on the normalized query vector Q' s and bond vector K' s Computational Channel Self-Attention A cc and channel-selective self-attention A' cc The channel-selective self-attention was normalized to obtain the channel-selective self-attention score S. c :

[0056]

[0057] A' cc =Select(σ(A) cc )>β,A cc ,-∞)

[0058] S c =Softmax(A' cc )

[0059] In the formula, τ5 is a learnable parameter; Select(σ(A) cc )>β,A cc (,-∞) is a selection mechanism, indicating that when σ(A) cc When )>β, return A. cc Otherwise, it returns -∞; σ is the sigmoid function; β is a hyperparameter set to 0.05; Softmax is the normalization function.

[0060] S32-B, using value vector V s Channel-selective self-attention score S c The preliminary fusion feature F of the channels was calculated. c ', to initially fuse channel features F c 'With short-range fusion features F s The features F are obtained by summing the results and then performing linear mapping and layer normalization operations. c ":

[0061] F′ c =V s S c

[0062] F″ c =LN(Linear(F c ′)+F s )

[0063] In the formula, Linear represents a linear mapping; LN is the layer normalization operation.

[0064] Finally, feature F c "The channel fusion feature F is obtained by using a multilayer perceptron and performing layer normalization." c :

[0065] F c =LN(MLP(F c ")+F c ")

[0066] In the formula, MLP stands for Multilayer Perceptron.

[0067] Switch to S33;

[0068] S33. The visible light and thermal image fusion feature with a scale of (H / 4)×(W / 4) obtained in the first stage of S32, and the visible light and thermal image fusion features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the lightweight fully connected multilayer perceptron decoding module to obtain four decoded features with a pixel size of (H / 4)×(W / 4). The four decoded features are then concatenated to obtain the aggregated feature, as follows:

[0069] The visible light and thermal image features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the latter three stages are upsampled to obtain three decoding features with a scale of (H / 4)×(W / 4). These three decoding features with a scale of (H / 4)×(W / 4) are then concatenated with the features with a scale of (H / 4)×(W / 4) obtained from the first stage of visible light and thermal image interaction fusion to obtain the aggregated feature X. C :

[0070] X C =Concat(H 1 H 2 H 3 H 4 )

[0071] In the formula, Concat represents the concatenation operation; H 1 H 2 H 3 H 4 These represent the decoding features of the four stages of the encoder.

[0072] Switch to S34.

[0073] S34. The aggregated features output in S33 are processed through a multilayer perceptron network for channel dimensionality reduction to obtain dimensionality-reduced features. The cross-entropy loss between the dimensionality-reduced features and the corresponding image labels in the MFNet dataset is calculated, and the network parameters of the multimodal selective attention semantic segmentation network are updated accordingly to obtain the trained multimodal selective semantic segmentation model, as follows:

[0074]

[0075] In the formula, Loss is the cross-entropy loss; For label values; y i The model outputs the predicted values ​​for the samples, where i is the class index of each pixel.

[0076] Proceed to step 4.

[0077] Step 4: Input the test dataset into the trained multimodal selective attention semantic segmentation model, output the prediction result corresponding to each sample in the test set, and test the accuracy of the trained multimodal selective attention semantic segmentation model.

[0078] Example 1

[0079] The present invention discloses a low-light night vision scene understanding method based on multimodal image fusion, comprising the following steps:

[0080] Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; divide the uniformly sized images into training and testing datasets according to a ratio of 784 / 393, perform data augmentation on the training dataset to form the network training dataset, and proceed to Step 2.

[0081] Step 2: Construct a multimodal selective attention semantic segmentation network:

[0082] The multimodal selective attention semantic segmentation network includes a visible light encoding network, a thermal image encoding network, a short-range selective attention module, a channel-selective self-attention module, and a lightweight decoding network. The visible light encoding network and the thermal image encoding network are both based on Transformer networks pre-trained on the ImageNet dataset, serving as encoding networks for feature extraction. The short-range selective attention module is used to preserve multimodal complementary feature information. The channel-selective self-attention module is used to enhance the information integrity of feature channel dimensions. The decoding network consists of a lightweight fully connected multilayer perceptron, used to receive feature information from the encoding network and reconstruct features from feature maps at different scales, proceeding to step 3.

[0083] Step 3: Train the selective attention multimodal semantic segmentation network using the network training dataset to obtain the trained multimodal selective attention semantic segmentation network model:

[0084] S31. The visible light coding network feature extraction is divided into four stages, and four visible light features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32); Correspondingly, the thermal image coding network feature extraction is divided into four stages, and four thermal image features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), and then proceed to S32.

[0085] S32. Input the visible light features and thermal image features extracted from each stage into the short-range selective attention module to obtain short-range fusion features. Then input the short-range fusion features into the channel selective self-attention module to obtain the features after interactive fusion of visible light and thermal images in this stage, and proceed to S33.

[0086] S33. The visible light and thermal image fusion feature with a scale of (H / 4)×(W / 4) obtained in the first stage of S32, and the visible light and thermal image fusion feature with a scale of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the lightweight fully connected multilayer perceptron decoding module to obtain four decoding features with a pixel size of (H / 4)×(W / 4). The four decoding features are then concatenated to obtain the aggregated feature, and then transferred to S34.

[0087] S34. The aggregated features output in S33 are processed through a multilayer perceptron to perform channel dimensionality reduction, resulting in dimensionality-reduced features. The cross-entropy loss between the dimensionality-reduced features and the corresponding image labels in the MFNet dataset is calculated, and the network parameters of the multimodal selective attention semantic segmentation network are updated accordingly to obtain the trained multimodal selective semantic segmentation model. Proceed to step 4.

[0088] Step 4: Input the test dataset into the trained multimodal selective attention semantic segmentation model, output the prediction result corresponding to each sample in the test set, and test the accuracy of the trained multimodal selective attention semantic segmentation model.

[0089] The method of this invention was tested on a computer configured with an Intel Xeon CPU and a Tesla V100 GPU, using the Python programming language and the PyTorch deep learning framework. During training, the batch size was set to 8, the AdamW optimizer with weight decay of 0.01 was used, and the learning rate was adjusted using the Warm-up Polymer learning rate adjustment method with an initial learning rate of 0.00006. Multiple batches were trained on the training sample set to obtain the low-light multimodal scene understanding method based on dynamic feature selection complementarity described in this invention. Visualization of the experimental results is as follows. Figure 2 , Figure 3 As shown.

[0090] To demonstrate the superior performance of this invention, a recently advanced multimodal semantic segmentation method was selected as a comparison model. The comparison results are shown in Table 1. The accuracy, crossover ratio, number of parameters, and computational cost of the model were evaluated on the MFNet dataset, with the input being a visible light image (480×640×3) and a thermal image (480×640×3).

[0091] Table 1 shows the comparative experimental results of different methods on the MFNet database.

[0092]

[0093] The experimental results show that the method of the present invention achieved a segmentation accuracy of 60.7%, with the computational cost (FLOPs) reduced to 74.86G and the number of parameters (Params) reduced to 41.08M. Furthermore, the low-light multimodal scene understanding method based on dynamic feature selection complementarity can be applied to autonomous driving on urban roads at night and unmanned navigation systems in dimly lit underground spaces.

Claims

1. A method for understanding low-light multimodal scenes based on dynamic feature selection complementarity, characterized in that, The steps are as follows: Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; The images after being standardized are divided into training and testing datasets according to a ratio of 784 / 393. Data augmentation is performed on the training dataset to form the network training dataset. Proceed to step 2. Step 2: Construct a multimodal selective attention semantic segmentation network: The multimodal selective attention semantic segmentation network includes a visible light encoding network, a thermal image encoding network, a short-range selective attention module, a channel-selective self-attention module, and a lightweight decoding network. The visible light encoding network and the thermal image encoding network are both based on Transformer networks pre-trained on the ImageNet dataset, serving as encoding networks for feature extraction. The short-range selective attention module is used to preserve multimodal complementary feature information. The channel-selective self-attention module is used to enhance the information integrity of feature channel dimensions. The decoding network consists of a lightweight fully connected multilayer perceptron, used to receive feature information from the encoding network and reconstruct features from feature maps at different scales, proceeding to step 3. Step 3: Input the network training dataset into the multimodal selective attention semantic segmentation network, and train the multimodal selective attention semantic segmentation network using the network training dataset to obtain the trained multimodal selective attention semantic segmentation network model, as follows: S31. The visible light coding network feature extraction is divided into four stages, and four visible light features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32); Correspondingly, the thermal image coding network feature extraction is divided into four stages, and four thermal image features of different scales are extracted for each stage, namely (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), then proceed to S32; S32. Input the visible light features and thermal image features extracted at each stage into the short-range selective attention module to obtain short-range fusion features, and then input the short-range fusion features into the channel selective self-attention module to obtain the features after interactive fusion of visible light and thermal images at this stage. Specifically, visible light features and thermal image features are input into a short-range selective attention module to obtain short-range fused features, as follows: S32-1 defines the query vector mapped from visible light features as... The key vector is defined as follows: Value vector is defined as Similarly, the query vector mapped from the thermal image features is defined as The key vector is defined as follows: Value vector is defined as The query vector, key vector, and value vector are segmented separately, and the mean and standard deviation of the segmented query vector and key vector are calculated. Then, the segmented query vector and key vector are normalized based on their mean and standard deviation. ， ， In the formula, mode ; It has visible light characteristics; Features of thermal images; The segmented query vector; The key vector after segmentation; The resulting value vector; For the segmentation function; This is the normalized query vector; The normalized key vector; It is a mean function; It is a function of standard deviation; S32-2, calculate the visible light self-attention based on the normalized query vector and the normalized key vector, respectively. Self-attention for thermal images According to visible light self-attention Self-attention for thermal images Selective attention within the visible light mode was calculated respectively. Intramodal selective attention of thermal images : ， ， ， ， In the formula, This is the query vector after normalization of visible light features; The key vector is the normalized visible light feature; The query vector is the normalized feature vector of the thermal image. The key vector is the normalized feature vector of the thermal image; T represents the transpose. , These are learnable parameters; It is a selection mechanism, indicating when When, return Conversely, return ; Similarly; It is the sigmoid function; This is a hyperparameter, set to 0.2; S32-3, Based on the query vector normalized from visible light features The key vector after normalization of thermal image features Calculate the attention between visible light and thermal image modes The query vector is normalized based on the features of the thermal image. and the bond vector after normalization of visible light features Calculate the attention between thermal images and visible light modes. And based on the attention between visible light and thermal image modes Attention between thermal images and visible light modes Selective attention between visible light and thermal image modes was calculated respectively. Selective attention between thermal images and visible light modes : ， ， ， ， In the formula, T represents transpose; , All are learnable parameters; It is a selection mechanism, indicating when When, return Conversely, return ; Similarly; It is the sigmoid function; It is a hyperparameter, set to 0.2; Selective attention within the visible light mode Intramodal selective attention in thermal images Selective attention between visible light and thermal image modalities Selective attention between thermal images and visible light modes Short-range selective attention score is calculated : ， In the formula, Softmax is the normalized exponential function; S32-4, using the segmented value vector and short-distance selective attention score Calculate short-range preliminary fusion features To initially fuse short-range features Perform a flipping operation, then compare with visible light features and thermal image features After addition, layer normalization is performed to obtain the features. : ， ， In the formula, This is the value vector after visible light feature segmentation; This is the value vector after feature segmentation of the thermal image; For layer normalization operation; It is a linear mapping; This is a flip operation; Features Short-range fused features are obtained by using a multilayer perceptron and performing layer normalization. : ， In the formula, MLP represents a multilayer perceptron; Switch to S33; S33. The visible light and thermal image fusion feature with a scale of (H / 4)×(W / 4) obtained in the first stage of S32 and the visible light and thermal image fusion feature with a scale of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the lightweight fully connected multilayer perceptron decoding module to obtain four decoding features with a pixel size of (H / 4)×(W / 4). The four decoding features are then concatenated to obtain the aggregated feature, and the process is transferred to S34. S34. The aggregated features output in S33 are processed through a multilayer perceptron to perform channel dimensionality reduction, resulting in dimensionality-reduced features. The cross-entropy loss between the dimensionality-reduced features and the corresponding image labels in the MFNet dataset is calculated, and the network parameters of the multimodal selective attention semantic segmentation network are updated accordingly to obtain the trained multimodal selective semantic segmentation model. Proceed to step 4; Step 4: Input the test dataset into the trained multimodal selective attention semantic segmentation model, output the prediction result corresponding to each sample in the test set, and test the accuracy of the trained multimodal selective attention semantic segmentation model.

2. The low-light multimodal scene understanding method based on dynamic feature selection complementarity as described in claim 1, characterized in that, In S32, the short-range fusion features are input into the channel-selective self-attention module to obtain the features after interactive fusion of visible light and thermal images at this stage, as follows: S32-A, which integrates the short-range fusion features output by the short-range selective attention module. After linear mapping, the query vector is obtained. Key vector Sum value vector and query vector and key vector Normalization is performed to obtain the normalized query vector. and key vector Then, based on the normalized query vector and key vector Computational Channel Self-Attention and channel-selective self-attention The channel-selective self-attention was then normalized to obtain the channel-selective self-attention score. : ， ， ， In the formula, These are learnable parameters; It is a selection mechanism, indicating when When, return Conversely, return ; It is the sigmoid function; is a hyperparameter, set to 0.05; Softmax is the normalization function; S32-B, using value vectors Channel-selective self-attention score Preliminary channel fusion characteristics were calculated. The channel features were initially integrated. With short-range fusion features The features are obtained by addition and through linear mapping and layer normalization operations. : ， ， In the formula, LN represents a linear mapping; LN is the layer normalization operation. Finally, the features Channel fusion features are obtained by using a multilayer perceptron and performing layer normalization. : ， In the formula, MLP stands for Multilayer Perceptron.

3. The low-light multimodal scene understanding method based on dynamic feature selection complementarity according to claim 2, characterized in that, In S33, the visible light and thermal image fusion features with a scale of (H / 4) × (W / 4) obtained in the first stage of S32, and the visible light and thermal image fusion features with scales of (H / 8) × (W / 8), (H / 16) × (W / 16), and (H / 32) × (W / 32) obtained in the following three stages are respectively transmitted to the lightweight fully connected multilayer perceptron decoding module, resulting in four decoded features with a pixel size of (H / 4) × (W / 4). The four decoded features are then concatenated to obtain the aggregated feature, as follows: The visible light and thermal image features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the latter three stages are upsampled to obtain three decoding features with a scale of (H / 4)×(W / 4). These three decoding features with a scale of (H / 4)×(W / 4) are then concatenated with the features with a scale of (H / 4)×(W / 4) obtained in the first stage of the visible light and thermal image interaction fusion to obtain the aggregated features. : ， In the formula, Indicates a splicing operation; , , , These represent the decoding features of the four stages of the encoder.

4. The low-light multimodal scene understanding method based on dynamic feature selection complementarity according to claim 3, characterized in that, In S34, the aggregated features output from S33 are processed by a multilayer perceptron network for channel dimensionality reduction to obtain dimensionality-reduced features. The cross-entropy loss between the dimensionality-reduced features and the corresponding image labels in the MFNet dataset is calculated, and the network parameters of the multimodal selective attention semantic segmentation network are updated accordingly to obtain the trained multimodal selective semantic segmentation model, as detailed below: ， In the formula, Cross-entropy loss; For tag values; The sample prediction values ​​output by the model. Category index for each pixel.

Images