Low-illumination scene analysis method based on multi-modal feature fusion and clustering

Abstract

The invention discloses a low-illumination scene analysis method based on multi-modal feature fusion and clustering, which belongs to the artificial intelligence technology and comprises the following steps: constructing a single-branch feature extraction network based on Transform; constructing a multi-modal feature interaction and fusion module to realize multi-modal feature interaction and fusion; a multi-modal fusion feature clustering module is constructed to perform feature clustering on features after multi-modal interaction fusion, and down-sampling of the features is realized while clustering is completed by using semantic distances and spatial distances among different features, so that the problem of loss of detail edge information in structured down-sampling is relieved; and constructing a multi-scale feature aggregation decoding module which is used for receiving the coding network feature information and dividing the category of each feature pixel according to the semantic distance of the multi-scale features. According to the method, visible light and thermal image information can be fully utilized, and the method can be applied to unmanned system scene analysis and navigation in various complex low-illumination scenes.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a low-illumination scene analysis method based on multimodal feature fusion and clustering. Background Art

[0002] With the rapid development of artificial intelligence technology, unmanned systems have higher and higher requirements for the perception and understanding of complex scenes, especially for unmanned systems such as self-driving cars, intelligent robots, and drones, which face challenges in complex low-light environments. In diverse and complex environments, a single visible light image visual sensor is easily affected by diverse low-light environments, such as night, rainy days, foggy days, smoke and dust, etc. Therefore, a thermal image sensor is combined to make up for the shortcomings of visible light image sensors, ensuring that unmanned systems have the same perception capabilities during the day and night and in severe weather conditions. In summary, studying the efficient fusion method of visible light images and thermal images can effectively improve the stability of scene understanding of unmanned systems under low-light night vision conditions.

[0003] Wang et al. proposed a novel semantic-guided multimodal fusion network in SGFNet: Semantic-Guided Fusion Network for RGB-ThermalSemantic Segmentation. The authors first proposed a semantic-guided head to extract semantic information in the thermal image branch. In the RGB branch, the authors proposed a multimodal coordination and distillation unit to first fuse multimodal features. Then, the authors proposed a cross-level and semantic-guided enhancement unit to enhance the feature information that integrates cross-level information and semantics. For the decoder, in order to obtain a larger receptive field and edge details, the authors proposed an edge-aware LawinASPP decoder. Although this method has achieved excellent semantic segmentation performance, the introduction of additional semantic information reduces the generalization application of different data sets. There is also an interactive overload between multimodalities, which leads to parameter redundancy and increased computational complexity, but limited accuracy improvement, which is not conducive to deployment on unmanned system equipment. Summary of the invention

[0004] The purpose of the present invention is to provide a low-light scene analysis method based on multimodal feature fusion and clustering for the first time, which efficiently utilizes the complementary characteristics of multimodal feature information; constructs a multimodal feature interaction fusion module to realize early multimodal feature interaction and fusion; constructs a multimodal fusion feature clustering module to perform feature clustering on the features after multimodal interaction fusion, and uses the semantic distance and spatial distance between different features to complete clustering while realizing feature downsampling, thereby alleviating the problem of loss of detail edge information in structured downsampling. The present invention provides a low-light night vision scene understanding method based on multimodal image fusion, which can be used in nighttime urban road automatic driving and dark unmanned navigation systems in underground spaces.

[0005] The technical solution to achieve the purpose of the present invention is: a low-light scene analysis method based on multimodal feature fusion and clustering, comprising the following steps:

[0006] Step 1: Normalize the 1569 images in the MFNet dataset and unify the pixel size to H×W, where H represents length and W represents width; divide the images after unification into training dataset and test dataset in a ratio of 784 / 393, and perform data enhancement on the training dataset to form a network training dataset; proceed to step 2.

[0007] Step 2: Construct multimodal feature fusion and clustering network:

[0008] The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multiscale feature aggregation decoding module; wherein the multimodal feature encoding network is based on a Transformer network pre-trained on the ImageNet data set, and serves as an encoding network for extracting features; the multimodal interaction fusion module is used for multimodal image feature interaction and fusion; the multimodal fusion feature clustering module is used for feature clustering of fusion features obtained after multimodal interaction fusion; the multiscale feature aggregation decoding module is used to receive the multiscale feature information obtained by the encoding network, and classify each feature pixel according to the semantic distance of the multiscale feature; proceed to step 3.

[0009] Step 3: Use the network training data set to train the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model:

[0010] S31. The feature extraction of the multimodal feature encoding network is divided into four stages. The first stage is used to extract the features of visible light and thermal images, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features of the corresponding scale in turn, and the scales of the features are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Go to S32.

[0011] S32, input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interactive fusion module to obtain features with a scale of (H / 4)×(W / 4) after the interactive fusion of visible light and thermal images, and transmit them to the multimodal fusion feature clustering module to obtain clustering features and input them into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8) in the second stage; upsample the features with a scale of (H / 8)×(W / 8) obtained in the second stage and perform clustering with the features with a scale of (H / 4)×(W / 4) obtained in the first stage after the interactive fusion of visible light and thermal images. The rows are added, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the third feature extraction stage to obtain the feature with a scale of (H / 16)×(W / 16) in the third stage; the feature with a scale of (H / 16)×(W / 16) obtained in the third stage is upsampled and added with the feature with a scale of (H / 8)×(W / 8) obtained in the second stage, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the fourth feature extraction stage to obtain the feature with a scale of (H / 32)×(W / 32) in the fourth stage; transfer to S33.

[0012] S33. The features with a scale of (H / 4)×(W / 4) after interactive fusion of visible light and thermal images obtained in the first stage of S32 and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the multi-scale feature aggregation decoding module through skip connections to obtain decoding features with a pixel size of (H / 4)×(W / 4), and then transferred to S34.

[0013] S34, calculate the Euclidean distance between the decoding features and the category features output in S33, negate the calculated distance value to obtain the output feature, construct a cross entropy loss between the output feature and the label, and use this to update the network parameters of the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model, and proceed to step 4.

[0014] Step 4: Input the test data set into the trained multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test data set, and test the accuracy of the trained multimodal feature fusion and clustering model.

[0015] Compared with the prior art, the present invention has the following significant advantages:

[0016] (1) The present invention proposes a low-light scene analysis method based on multimodal feature fusion and clustering, and for the first time proposes a single-branch feature extraction network constructed based on Transformer and unstructured clustering downsampling method, which significantly reduces the number of parameters and calculation amount.

[0017] (2) The present invention proposes a multimodal feature interactive fusion module, which enables the two modal information to be shared and establishes interactive attention from the spatial context, thereby achieving effective interaction and fusion between different modal feature information to reduce the differences between modal information.

[0018] (3) The present invention proposes a feature clustering module based on spatial semantic distance, which effectively utilizes the semantic distance and position distance of features and alleviates the problem of loss of feature information and edge detail information caused by structured downsampling.

[0019] (4) The present invention proposes a lightweight and efficient multi-scale feature aggregation decoding module, which classifies the multi-scale coding features extracted by the feature extraction network according to the semantic distance between the features and each category feature, and makes full use of the potential semantic distance information of the features to achieve more accurate category generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a model diagram of a low-light scene analysis method based on multimodal feature fusion and clustering.

[0021] Figure 2 This is the experimental result of urban road scene of MFNet dataset.

[0022] Figure 3 Visualization of structured and unstructured downsampling results for features at different stages of the network. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention clearer, the embodiments of the present invention are further described in detail below:

[0024] Combination Figure 1 , a low-light scene parsing method based on multimodal feature fusion and clustering, comprising the following steps:

[0025] Step 1: Normalize the 1569 images in the MFNet dataset and unify the pixel size to H×W, where H represents length and W represents width; divide the images after unification into training dataset and test dataset in a ratio of 784 / 393, and perform data enhancement on the training dataset to form a network training dataset; proceed to step 2.

[0026] Step 2: Construct multimodal feature fusion and clustering network:

[0027] The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multiscale feature aggregation decoding module; wherein the multimodal feature encoding network is based on a Transformer network pre-trained on the ImageNet data set, and serves as an encoding network for extracting features; the multimodal interaction fusion module is used for multimodal image feature interaction and fusion; the multimodal fusion feature clustering module is used for feature clustering of fusion features obtained after multimodal interaction fusion; the multiscale feature aggregation decoding module is used to receive the multiscale feature information obtained by the encoding network, and classify each feature pixel according to the semantic distance of the multiscale feature, and then proceed to step 3.

[0028] Step 3: Use the network training data set to train the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model:

[0029] S31. The feature extraction of the multimodal feature encoding network is divided into four stages. The first stage is used to extract the features of visible light and thermal images, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features of the corresponding scale in turn, and the scales of the features are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Go to S32.

[0030] S32, input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interactive fusion module to obtain features with a scale of (H / 4)×(W / 4) after interactive fusion of visible light and thermal images, and transmit them to the multimodal fusion feature clustering module to obtain clustering features and input them into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8) in the second stage; upsample the features with a scale of (H / 8)×(W / 8) obtained in the second stage and interactively fuse them with the visible light and thermal images obtained in the first stage to obtain features with a scale of (H / 4)×(W / 4) The addition is performed, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the third feature extraction stage to obtain the feature with a scale of (H / 16)×(W / 16) in the third stage; the feature with a scale of (H / 16)×(W / 16) obtained in the third stage is upsampled and added to the feature with a scale of (H / 8)×(W / 8) obtained in the second stage, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the fourth feature extraction stage to obtain the feature with a scale of (H / 32)×(W / 32) in the fourth stage, as follows:

[0031] The visible light feature is defined as F vi , the thermal image feature is defined as F ir ; Then the visible light feature F vi Obtain visible light feature interaction vector through convolution and the visible light eigenvalue vector Similarly, the thermal image feature F ir Mapping thermal image feature interaction vector and thermal image eigenvalue vector Interactive Attention Score A I It is calculated by the interaction vector of the two modes:

[0032]

[0033] In the formula, represents matrix multiplication and T represents the transpose operation.

[0034] The weighted sum of the value vectors of the two modes is performed by the score of the interactive attention to obtain the weighted visible light feature X vi and thermal image feature X ir The weighted features of the obtained visible light and thermal images are convolved and combined with the input visible light feature F vi and thermal image feature F ir Add together to get the visible light feature F' vi and thermal image feature F' ir , the process is as follows:

[0035]

[0036] F′ ir =Conv(X ir )+F ir ,F′ vi =Conv(X vi )+F vi

[0037] In the formula, Softmax is the normalized exponential function and Conv is the convolution operation.

[0038] The channel attention value is calculated by global feature statistics such as average, maximum and variance, and the channel attention score is obtained through multi-layer perceptron mapping, so as to achieve weighting of interactive features and obtain the final output visible light feature F″. vi and thermal image feature F″ ir , the process is as follows:

[0039] F″ vi =F′ vi ⊙MLP(Concat(mean(F′ vi ),max(F′ vi ),var(F′ vi )))

[0040] F″ ir =F′ ir ⊙MLP(Concat(mean(F′ ir ),max(F′ ir ),var(F′ ir )))

[0041] F fused =F vi +F″ ir

[0042] In the formula, ⊙ represents the Hadamard product, MLP is a multi-layer perceptron, Concat is a concatenation operation, mean is the mean function, max is the maximum function, var is the variance function, and the final fusion output result is the interactive fusion feature F fused .

[0043] The multimodal interaction fusion features obtained above are input into the multimodal fusion feature clustering module, as follows:

[0044] First, in the feature clustering process, the feature map is divided into N Token features, and the improved k-nearest neighbor density peak clustering algorithm is used.

[0045] The Token set required by the feature clustering module is X = {x 1 ,x2 ,…,x n ,…x m ,…,x N}, the corresponding Token space position set Y = {y 1 ,y 2 ,…,y n ,…y m ,…y N}, N is the total number of sets, and the distance d between any two Tokens n,m for:

[0046] d n,m =τ||x n -x m || 2 +(1-τ)||y n -y m || 2

[0047] Where τ is the weighting factor of semantic distance and spatial distance, ||·|| 2 represents the Euclidean distance, n and m represent the nth and mth Token features respectively.

[0048] Calculate the local density of each Token based on its k nearest neighbors:

[0049] Among them, the local density ρ calculated by the k nearest neighbors of the nth Token n as follows:

[0050]

[0051] In the formula, KNN(x n ) represents the k nearest neighbors of the nth Token, x n With x m They correspond to the nth and mth Token features respectively.

[0052] Then, the minimum distance value between each Token and other Tokens with higher local density is calculated as the distance indicator. For the Token with the highest local density, the maximum distance between it and any other Token is calculated as the distance indicator. The formula is as follows:

[0053]

[0054] Where δ n is the distance indicator. Combining the local density and distance indicators, the score of each Token is ρ n ×δ n The higher the score, the higher the token with the highest score will be determined as the cluster center.

[0055] Finally, other tokens are assigned to the nearest cluster center according to the feature distance to complete feature clustering;

[0056] For Token feature merging, we first define an importance score P to clearly indicate the importance of each Token. P is calculated by the Token feature through the fully connected layer, expressed as: P = FC(X), where FC represents the linear mapping of the fully connected layer; the importance score P is weighted and averaged by the Softmax function:

[0057]

[0058] In the formula, C n is the set of n-th clusters, p m Is the Token feature x m The importance score of It is the new Token feature after Token clustering and merging. After feature clustering is completed, the features after clustering and merging are Input to the second stage encoding module, j represents the jth Token feature.

[0059] S33. The features with a scale of (H / 4)×(W / 4) after interactive fusion of visible light and thermal images obtained in the first stage of S32 and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the multi-scale feature aggregation decoding module through skip connections to obtain decoding features with a pixel size of (H / 4)×(W / 4), and then transferred to S34.

[0060] S34, calculate the Euclidean distance between the decoding features and the category features output in S33, negate the calculated distance value to obtain the output feature, construct a cross entropy loss between the output feature and the label, and use this to update the network parameters of the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model, and proceed to step 4.

[0061] Step 4: Input the test data set into the multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test data set, and test the accuracy of the multimodal feature fusion and clustering model.

[0062] Example 1

[0063] The present invention provides a low-light scene analysis method based on multimodal feature fusion and clustering, comprising the following steps:

[0064] Step 1: Normalize the 1569 images in the MFNet dataset and unify the pixel size to H×W, where H represents length and W represents width; divide the images after unification into training dataset and test dataset in a ratio of 784 / 393, and perform data enhancement on the training dataset to form a network training dataset; proceed to step 2.

[0065] Step 2: Construct multimodal feature fusion and clustering network:

[0066] The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multiscale feature aggregation decoding module; wherein the multimodal feature encoding network is based on a Transformer network pre-trained on the ImageNet data set, and serves as an encoding network for extracting features; the multimodal interaction fusion module is used for multimodal image feature interaction and fusion; the multimodal fusion feature clustering module is used for feature clustering of fusion features obtained after multimodal interaction fusion; the multiscale feature aggregation decoding module is used to receive the multiscale feature information obtained by the encoding network, and classify each feature pixel according to the semantic distance of the multiscale feature; proceed to step 3.

[0067] Step 3: Use the network training data set to train the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model:

[0068] S31. The feature extraction of the multimodal feature encoding network is divided into four stages. The first stage is used to extract the features of visible light and thermal images, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features of the corresponding scale in turn, and the scales of the features are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Go to S32.

[0069] S32, input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interactive fusion module to obtain features with a scale of (H / 4)×(W / 4) after interactive fusion of visible light and thermal images, and transmit them to the multimodal fusion feature clustering module to obtain clustering features and input them into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8) in the second stage; upsample the features with a scale of (H / 8)×(W / 8) obtained in the second stage and interactively fuse them with the visible light and thermal images obtained in the first stage to obtain features with a scale of (H / 4)×(W / 4) The addition is performed, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the third feature extraction stage to obtain the feature with a scale of (H / 16)×(W / 16) in the third stage; the feature with a scale of (H / 16)×(W / 16) obtained in the third stage is upsampled and added to the feature with a scale of (H / 8)×(W / 8) obtained in the second stage, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the fourth feature extraction stage to obtain the feature with a scale of (H / 32)×(W / 32) in the fourth stage; the details are as follows:

[0070] The visible light feature is defined as F vi , the thermal image feature is defined as F ir ; Then the visible light feature F vi Obtain visible light feature interaction vector through convolution and the visible light eigenvalue vector Similarly, the thermal image feature F ir Mapping thermal image feature interaction vector With the thermal image eigenvalue vector Interactive Attention Score A I It is calculated by the interaction vector of the two modes:

[0071]

[0072] In the formula, represents matrix multiplication and T represents the transpose operation.

[0073] The weighted sum of the value vectors of the two modes is performed by the score of the interactive attention to obtain the weighted visible light feature X vi and thermal image feature X ir The weighted features of the obtained visible light and thermal images are convolved and combined with the input visible light feature F vi and thermal image feature F ir Add together to get the visible light feature F' vi and thermal image feature F' ir , the process is as follows:

[0074]

[0075] F′ ir =Conv(X ir )+F ir ,F′ vi =Conv(X vi )+F vi

[0076] In the formula, Softmax is the normalized exponential function and Conv is the convolution operation.

[0077] Construct a channel attention value calculated by global feature statistics such as average, maximum and variance, and obtain the channel attention score through multi-layer perceptron mapping, so as to achieve weighting of interactive features and obtain the final output visible light feature F' v ' i and thermal image feature F' i ' r , the process is as follows:

[0078] F″ vi =F′ vi ⊙MLP(Concat(mean(F′ vi ),max(F′ vi ),var(F′ vi )))

[0079] F″ ir =F′ ir ⊙MLP(Concat(mean(F′ ir ),max(F′ ir ),var(F′ ir )))

[0080] F fused =F″ vi +F″ ir

[0081] In the formula, ⊙ represents the Hadamard product, MLP is a multi-layer perceptron, Concat is a concatenation operation, mean is the mean function, max is the maximum function, var is the variance function, and the final fusion output result is the interactive fusion feature F fused .

[0082] The multimodal interaction fusion features obtained above are input into the multimodal fusion feature clustering module, as follows:

[0083] First, in the feature clustering process, the feature map is divided into N Token features, and the improved k-nearest neighbor density peak clustering algorithm is used.

[0084] The Token set required by the feature clustering module is X = {x 1 ,x 2 ,…,x n ,…x m ,…,x N}, the corresponding Token space position set Y = {y 1 ,y 2 ,…,y n ,…y m ,…y N}, N is the total number of sets, and the distance d between any two Tokens n,m for:

[0085] d n,m =τ||x n -x m || 2 +(1-τ)||y n -y m || 2

[0086] Where τ is the weighting factor of semantic distance and spatial distance, ||·|| 2 represents the Euclidean distance, n and m represent the nth and mth Token features respectively.

[0087] The local density of each Token is calculated based on its k nearest neighbors.

[0088] Among them, the local density ρ calculated by the k nearest neighbors of the nth Token n as follows:

[0089]

[0090] In the formula, KNN(x n ) represents the k nearest neighbors of the nth Token, x n With x m They correspond to the nth and mth Token features respectively.

[0091] Then, the minimum distance value between each Token and other Tokens with higher local density is calculated as the distance indicator. For the Token with the highest local density, the maximum distance between it and any other Token is calculated as the distance indicator. The formula is as follows:

[0092]

[0093] Where δ n is the distance indicator. Combining the local density and distance indicators, the score of each Token is ρ n ×δ nThe higher the score, the higher the token with the highest score will be determined as the cluster center.

[0094] Finally, other tokens are assigned to the nearest cluster center according to the feature distance to complete feature clustering.

[0095] For Token feature merging, we first define an importance score P to clearly indicate the importance of each Token. P is calculated by the Token feature through the fully connected layer, expressed as: P = FC(X), where FC represents the linear mapping of the fully connected layer; the importance score P is weighted and averaged by the Softmax function:

[0096]

[0097] In the formula, C n is the set of n-th clusters, p m Is the Token feature x m The importance score of It is the new Token feature after Token clustering and merging. After feature clustering is completed, the features after clustering and merging are Input to the second stage encoding module, j represents the jth Token feature.

[0098] S33. The multimodal interactive fusion features with a scale of (H / 4)×(W / 4) obtained in the first stage of S32 and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the multi-scale feature aggregation decoding module through jump connections to obtain decoding features with a pixel size of (H / 4)×(W / 4), and then transferred to S34.

[0099] S34, calculating the Euclidean distance between the decoding feature output in S33 and the category feature, negating the distance value to obtain the output feature, constructing a cross entropy loss between the output feature and the label, and updating the network parameters of the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model, and proceeding to step 4;

[0100] Step 4: Input the test data set into the multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test data set, and test the accuracy of the multimodal feature fusion and clustering model.

[0101] The method of the present invention conducts relevant experiments on a network built with Python programming language and Pytorch deep learning framework on a computer configured with i7 CPU and NVIDIAV100 GPU. During the training process, the batch size is set to 2, the optimizer uses the AdamW optimizer with a weight decay of 0.01, the learning rate adjustment method uses the Warm up ploy learning rate adjustment method, and the initial learning rate is 0.00006. Multiple batches are trained on the training sample set to obtain a low-light scene analysis method based on multimodal feature fusion and clustering. The visual experimental results are shown in Figure 2 shown.

[0102] In order to demonstrate the superior performance of the present invention, the present invention selects the most advanced multimodal semantic segmentation method as a comparison model. The comparison experimental results are shown in Table 1:

[0103] Table 1 Comparative experimental results of different methods on the MFNet database

[0104]

[0105]

[0106] The parameter amount and segmentation accuracy of the model are evaluated on the MFNet dataset, and its input is a visible light image (480×640×3) and a thermal image (480×640×3). From the experimental results, it can be seen that the parameter amount of the method of the present invention is 29.54M, the calculation amount is 36.6G, and the segmentation accuracy rate is 57.9%, which is better than other algorithm models.

Claims

1. A low-light scene analysis method based on multimodal feature fusion and clustering, characterized in that: Here are the steps: Step 1: Normalize the 1569 images in the MFNet dataset and unify the pixel size to H×W, where H represents length and W represents width; The images after uniform size are divided into training data set and test data set according to the ratio of 784 / 393, and the training data set is enhanced to form the network training data set; go to step 2; Step 2: Construct multimodal feature fusion and clustering network: The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multiscale feature aggregation decoding module; wherein the multimodal feature encoding network is based on a Transformer network pre-trained on an ImageNet data set, and serves as an encoding network for extracting features; the multimodal interaction fusion module is used for multimodal image feature interaction and fusion; the multimodal fusion feature clustering module is used for feature clustering of fusion features obtained after multimodal interaction fusion; the multiscale feature aggregation decoding module is used to receive the multiscale feature information obtained by the encoding network, and classify each feature pixel according to the semantic distance of the multiscale feature; proceed to step 3; Step 3: Use the network training data set to train the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model: S31, the feature extraction of the multimodal feature encoding network is divided into four stages. The first stage is used to extract the features of visible light and thermal images, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts the image features of the corresponding scale in turn, and the scales of the features are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), (H / 32)×(W / 32), respectively. Go to S32; S32, input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interactive fusion module to obtain features with a scale of (H / 4)×(W / 4) after the interactive fusion of visible light and thermal images, and transmit them to the multimodal fusion feature clustering module to obtain clustering features and input them into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8) in the second stage; upsample the features with a scale of (H / 8)×(W / 8) obtained in the second stage and perform clustering with the features with a scale of (H / 4)×(W / 4) obtained in the first stage after the interactive fusion of visible light and thermal images. The rows are added, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the third feature extraction stage to obtain the feature with a scale of (H / 16)×(W / 16) in the third stage; the feature with a scale of (H / 16)×(W / 16) obtained in the third stage is upsampled and added with the feature with a scale of (H / 8)×(W / 8) obtained in the second stage, and the addition result is input into the multimodal fusion feature clustering module to obtain the clustering feature and input into the fourth feature extraction stage to obtain the feature with a scale of (H / 32)×(W / 32) in the fourth stage; transfer to S33; S33, the features with a scale of (H / 4)×(W / 4) obtained after interactive fusion of visible light and thermal images in the first stage of S32 and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) respectively obtained in the following three stages are transmitted to the multi-scale feature aggregation decoding module through skip connections, and the decoding features with a pixel size of (H / 4)×(W / 4) are obtained, and then transferred to S34; S34, calculating the Euclidean distance between the decoding feature and the category feature output in S33, negating the calculated distance value to obtain the output feature, constructing a cross entropy loss between the output feature and the label, and updating the network parameters of the multimodal feature fusion and clustering network to obtain a trained multimodal feature fusion and clustering model, and proceeding to step 4; Step 4: Input the test data set into the multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test data set, and test the accuracy of the multimodal feature fusion and clustering model.

2. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 1, characterized in that: In S32, the visible light and thermal image features extracted in the first stage of S31 are input into the multimodal feature interactive fusion module to obtain features with a scale of (H / 4)×(W / 4) after interactive fusion of visible light and thermal images, and transmitted to the multimodal fusion feature clustering module to obtain clustering features and input into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8) in the second stage. The specific process of the multimodal feature interactive fusion module is as follows: The visible light feature is defined as F vi , the thermal image feature is defined as F ir ; Then the visible light feature F vi Obtain visible light feature interaction vector through convolution and the visible light eigenvalue vector Similarly, the thermal image feature F ir Mapping thermal image feature interaction vector and thermal image eigenvalue vector Interactive Attention Score A I It is calculated by the interaction vector of the two modes: In the formula, represents matrix multiplication, and T represents the transpose operation; The weighted sum of the value vectors of the two modes is performed by the score of the interactive attention to obtain the weighted visible light feature X vi and thermal image feature X ir The weighted features of the obtained visible light and thermal images are convolved and combined with the input visible light feature F vi and thermal image feature F ir Add together to get the visible light feature F′ vi and thermal image feature F′ ir , the process is as follows: F′ ir =Conv(X ir )+F ir ,F′ vi =Conv(X vi )+F vi Where Softmax is the normalized exponential function, and Conv is the convolution operation; Construct the channel attention value calculated by the global feature statistics of the average, maximum and variance, and obtain the channel attention score through multi-layer perceptron mapping, so as to achieve the weighting of the interactive features and obtain the final output visible light feature F″ vi and thermal image feature F″ ir .

3. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 2, characterized in that: The visible light and thermal image features extracted in the first stage in S32 are input to the multimodal feature interactive fusion module to obtain features with a scale of (H / 4)×(W / 4) after the interactive fusion of visible light and thermal images, and are transmitted to the multimodal fusion feature clustering module to obtain clustering features and input to the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8) in the second stage. The feature clustering operation is specifically as follows: First, in the feature clustering process, the feature map is divided into N token features, and the improved k-nearest neighbor density peak clustering algorithm is used; The Token set required by the feature clustering module is X = {x1, x2, ..., x n ,…x m ,…,x N }, the corresponding Token space position set Y = {y1,y2,…,y n ,…y m ,…y N }, N is the total number of sets, and the distance d between any two Tokens n,m for: d n,m =τ||x n -x m ||2+(1-τ))||y n -y m ||2 Where τ is the weighting factor of semantic distance and spatial distance, ||·||2 represents the Euclidean distance, and n and m represent the nth and mth Token features respectively; Calculate the local density of each Token based on its k nearest neighbors; Among them, the local density ρn calculated by the k nearest neighbors of the nth Token is as follows: In the formula, KNN(xn) represents the k nearest neighbors of the nth Token, and xn and xm represent the features of the nth and mth Token respectively; Then, the minimum distance value between each Token and other Tokens with higher local density is calculated as the distance indicator. For the Token with the highest local density, the maximum distance between it and any other Token is calculated as the distance indicator. The formula is as follows: In the formula, δ n is the distance indicator. Combining the local density and distance indicators, the score of each Token is ρ n ×δ n , the higher the score, the higher the token with the highest score will be determined as the cluster center; ρ m Represents the local density calculated by the k nearest neighbors of the mth Token; Finally, other tokens are assigned to the nearest cluster center according to the feature distance to complete feature clustering; For Token feature merging, we first define an importance score P to clearly indicate the importance of each Token. P is calculated by the Token feature through the fully connected layer, expressed as: P = FC(X), where FC represents the linear mapping of the fully connected layer; the importance score P is weighted and averaged by the Softmax function: In the formula, C n is the set of n-th clusters, p m Is the Token feature x m The importance score of It is the new Token feature after Token clustering and merging. After feature clustering is completed, the features after clustering and merging are Input to the second stage encoding module, j represents the jth Token feature.

4. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 3, characterized in that: Construct the channel attention value calculated by the global feature statistics of the average, maximum and variance, and obtain the channel attention score through multi-layer perceptron mapping, so as to achieve the weighting of the interactive features and obtain the final output visible light feature F″ vi and thermal image feature F″ ir , the process is as follows: F″ vi =F′ vi ⊙MLP(Concat(mean(F′ vi ),max(F′ vi ),var(F′ vi ))) F″ ir =F′ ir ⊙MLP(Concat(mean(F′ ir ),max(F′ ir ),var(F′ tr ))) F fused =F″ vi +F″ ir In the formula, ⊙ represents the Hadamard product, MLP is a multi-layer perceptron, Concat is a concatenation operation, mean is the mean function, max is the maximum function, var is the variance function, and the final fusion output result is the interactive fusion feature Ffused.

5. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 4, characterized in that: In S33, the features with a scale of (H / 4)×(W / 4) obtained after interactive fusion of visible light and thermal images in the first stage of S32 and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) respectively obtained in the following three stages are transmitted to the multi-scale feature aggregation decoding module through skip connections to obtain decoding features with a pixel size of (H / 4)×(W / 4). The multi-scale feature aggregation decoding module is specifically as follows: The features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) are upsampled to obtain features with scales of (H / 4)×(W / 4) and then concatenated with the features with scales of (H / 4)×(W / 4) obtained by interactive fusion of visible light and thermal images obtained in the first stage: XC=Concat(H 1 ,H 2 ,H 3 ,H 4 ) In the formula, H 1 , H 2 , H 3 , H 4 They represent the output features of the first, second, third, and fourth stages of the encoder respectively; XC represents the aggregated features obtained by concatenating features of multiple scales; Then, the semantic distance is defined based on the Euclidean distance And use the Softmax function to predict the segmentation results, as follows: In the formula, xt represents the learnable semantic category Token of the tth category, xs is the sth Token feature, and Rf is the final segmentation result; since the Softmax function is an increasing function, The smaller the median value, the closer the distance, but the smaller the classification probability, which is the opposite of the goal, so before entering the Softmax function Take the negative.

Images