A Low-Light Scene Analysis Method Based on Multimodal Feature Fusion and Clustering

Abstract

This invention discloses a low-light scene analysis method based on multimodal feature fusion and clustering, belonging to artificial intelligence technology. It constructs a single-branch feature extraction network based on Transformer; a multimodal feature interaction and fusion module to achieve multimodal feature interaction and fusion; a multimodal fusion feature clustering module to cluster the features after multimodal interaction and fusion, utilizing the semantic and spatial distances between different features to achieve clustering while simultaneously downsampling the features, mitigating the problem of loss of detailed edge information in structured downsampling; and a multi-scale feature aggregation and decoding module to receive feature information from the encoding network and classify each feature pixel according to the semantic distance of the multi-scale features. This invention can fully utilize visible light and thermal image information and can be applied to scene analysis and navigation of unmanned systems in diverse and complex low-light scenes.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a low-light scene analysis method based on multimodal feature fusion and clustering. Background Technology

[0002] With the rapid development of artificial intelligence technology, the requirements for unmanned systems in perceiving and understanding complex scenes are becoming increasingly stringent, especially for autonomous vehicles, intelligent robots, and drones, which face challenges in complex low-light environments. In diverse and complex environments, single visible light image sensors are easily affected by varying low-light conditions, such as nighttime, rainy days, foggy days, and dusty scenes. Therefore, combining them with thermal image sensors can compensate for the shortcomings of visible light image sensors, ensuring that unmanned systems possess consistent perception capabilities in daylight, nighttime, and adverse weather conditions. In conclusion, researching efficient fusion methods for visible light and thermal images can effectively improve the stability of unmanned systems in scene understanding under low-light night vision conditions.

[0003] Wang et al., in their paper "SGFNet: Semantic-Guided Fusion Network for RGB-Thermal Semantic Segmentation," proposed a novel semantic-guided network for multimodal fusion. The authors first propose a semantic guide to extract semantic information in the thermal image branch. In the RGB branch, they propose a unit that coordinates and distills multimodal features to first fuse them. Then, they further propose a cross-level and semantically guided enhancement unit to enhance the fused cross-level and semantic features. For the decoder, to obtain a larger receptive field and edge details, they propose the edge-aware LauinASPP decoder. While this method achieves excellent semantic segmentation performance, the introduction of additional semantic information reduces its generalization across different datasets. Intermodal overload also exists, leading to parameter redundancy and increased computational cost with limited accuracy improvement, which is not conducive to deployment on unmanned systems. Summary of the Invention

[0004] The purpose of this invention is to provide, for the first time, a low-light scene analysis method based on multimodal feature fusion and clustering, which efficiently utilizes the complementary characteristics of multimodal feature information. A multimodal feature interaction fusion module is constructed to achieve early multimodal feature interaction and fusion; a multimodal fusion feature clustering module is constructed to cluster the features after multimodal interaction fusion, using semantic and spatial distances between different features to complete clustering while simultaneously downsampling the features, alleviating the problem of loss of detail edge information in structured downsampling. This invention provides a low-light night vision scene understanding method based on multimodal image fusion, which can be applied to autonomous driving on urban roads at night and unmanned navigation systems in dimly lit underground spaces.

[0005] The technical solution to achieve the purpose of this invention is: a low-light scene analysis method based on multimodal feature fusion and clustering, comprising the following steps:

[0006] Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; divide the uniformly sized images into training and testing datasets according to a ratio of 784 / 393, perform data augmentation on the training dataset to form the network training dataset; proceed to Step 2.

[0007] Step 2: Construct a multimodal feature fusion and clustering network:

[0008] The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multi-scale feature aggregation decoding module. The multimodal feature encoding network is constructed based on a Transformer network pre-trained on the ImageNet dataset and serves as the encoding network for feature extraction. The multimodal interaction fusion module is used for multimodal image feature interaction and fusion. The multimodal fusion feature clustering module is used to cluster the fused features obtained after multimodal interaction fusion. The multi-scale feature aggregation decoding module is used to receive multi-scale feature information obtained from the encoding network and classify the category of each feature pixel based on the semantic distance of the multi-scale features. Proceed to step 3.

[0009] Step 3: Train the multimodal feature fusion and clustering network using the network training dataset to obtain the trained multimodal feature fusion and clustering model.

[0010] S31. The feature extraction of the multimodal feature coding network is divided into four stages. The first stage is used to extract visible light and thermal image features, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features at the corresponding scales, which are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Proceed to S32.

[0011] S32. Input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interaction fusion module to obtain features of scale (H / 4)×(W / 4) after interaction fusion of visible light and thermal images. Then, transmit these features to the multimodal fusion feature clustering module to obtain clustering features, which are then input into the second feature extraction stage to obtain features of scale (H / 8)×(W / 8). Upsample the features of scale (H / 8)×(W / 8) obtained in the second stage and combine them with the features of scale (H / 4)×(W / 4) obtained in the first stage after interaction fusion of visible light and thermal images. The features are added together, and the result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the third feature extraction stage to obtain features with a scale of (H / 16)×(W / 16). The features with a scale of (H / 16)×(W / 16) obtained in the third stage are upsampled and added to the features with a scale of (H / 8)×(W / 8) obtained in the second stage. The result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the fourth feature extraction stage to obtain features with a scale of (H / 32)×(W / 32). Proceed to S33.

[0012] S33. The features with a scale of (H / 4)×(W / 4) obtained from the interaction and fusion of visible light and thermal images in the first stage of S32, and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are transmitted to the multi-scale feature aggregation and decoding module through a skip connection to obtain the decoded features with a pixel size of (H / 4)×(W / 4), and then proceed to S34.

[0013] S34. Calculate the Euclidean distance between the decoded features and category features output in S33. Take the negative of the calculated distance value to obtain the output features. Construct a cross-entropy loss between the output features and the labels. Use this loss to update the network parameters of the multimodal feature fusion and clustering network to obtain the trained multimodal feature fusion and clustering model. Proceed to step 4.

[0014] Step 4: Input the test dataset into the trained multimodal feature fusion and clustering model, output the prediction result for each sample in the test dataset, and test the accuracy of the trained multimodal feature fusion and clustering model.

[0015] Compared with the prior art, the significant advantages of this invention are:

[0016] (1) This invention proposes a low-light scene parsing method based on multimodal feature fusion and clustering. It is the first to propose a single-branch feature extraction network based on Transformer and unstructured clustering downsampling method, which significantly reduces the number of parameters and computation.

[0017] (2) This invention proposes a multimodal feature interaction fusion module, which enables the sharing of information between two modalities and establishes interactive attention from the spatial context, thereby realizing effective interaction and fusion between different modal feature information and reducing the differences between modal information.

[0018] (3) This invention proposes a feature clustering module based on spatial semantic distance, which effectively utilizes the semantic distance and positional distance of features to alleviate the problem of loss of feature information and edge detail information caused by structured downsampling.

[0019] (4) This invention proposes a lightweight and efficient multi-scale feature aggregation decoding module, which classifies the multi-scale encoded features extracted by the feature extraction network according to the semantic distance between the features and each category feature, making full use of the potential semantic distance information of the features to achieve more accurate category generation. Attached Figure Description

[0020] Figure 1 This is a model diagram of a low-light scene parsing method based on multimodal feature fusion and clustering.

[0021] Figure 2 The image shows the experimental results of an urban road scene on the MFNet dataset.

[0022] Figure 3 Visualization results of structured and unstructured downsampling of features at different stages of the network. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below:

[0024] Combination Figure 1 A low-light scene parsing method based on multimodal feature fusion and clustering includes the following steps:

[0025] Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; divide the uniformly sized images into training and testing datasets according to a ratio of 784 / 393, perform data augmentation on the training dataset to form the network training dataset; proceed to Step 2.

[0026] Step 2: Construct a multimodal feature fusion and clustering network:

[0027] The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multi-scale feature aggregation decoding module. The multimodal feature encoding network is constructed based on a Transformer network pre-trained on the ImageNet dataset and serves as the encoding network for feature extraction. The multimodal interaction fusion module is used for multimodal image feature interaction and fusion. The multimodal fusion feature clustering module is used to cluster the fused features obtained after multimodal interaction fusion. The multi-scale feature aggregation decoding module receives multi-scale feature information obtained from the encoding network and classifies each feature pixel according to the semantic distance of the multi-scale features, then proceeds to step 3.

[0028] Step 3: Train the multimodal feature fusion and clustering network using the network training dataset to obtain the trained multimodal feature fusion and clustering model.

[0029] S31. The feature extraction of the multimodal feature coding network is divided into four stages. The first stage is used to extract visible light and thermal image features, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features at the corresponding scales, which are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Proceed to S32.

[0030] S32. Input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interaction fusion module to obtain features of scale (H / 4)×(W / 4) after interaction fusion of visible light and thermal images. Then, transmit these features to the multimodal fusion feature clustering module to obtain clustering features, which are then input into the second feature extraction stage to obtain features of scale (H / 8)×(W / 8). Upsample the features of scale (H / 8)×(W / 8) obtained in the second stage and combine them with the features of scale (H / 4)×(W / 4) obtained in the first stage after interaction fusion of visible light and thermal images. The features are added together, and the result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the third feature extraction stage to obtain features with a scale of (H / 16)×(W / 16). The features with a scale of (H / 16)×(W / 16) obtained in the third stage are upsampled and added to the features with a scale of (H / 8)×(W / 8) obtained in the second stage. The result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the fourth feature extraction stage to obtain features with a scale of (H / 32)×(W / 32), as detailed below:

[0031] The visible light characteristic is defined as F. vi Thermal image features are defined as F ir Then, the visible light feature F vi Visible light feature interaction vectors are obtained through convolution. With visible light eigenvectors Similarly, the thermal image features F ir Mapping thermal image feature interaction vectors With thermal image feature vector Interactive attention score A I Then it is calculated from the interaction vectors of the two modes:

[0032]

[0033] In the formula, represents matrix multiplication, and T represents the transpose operation.

[0034] The weighted visible light feature X is obtained by weighting and summing the value vectors of the two modalities using the interactive attention scores. vi and thermal image features X ir The weighted features of the obtained visible light and thermal images are convolved and then combined with the input visible light features F. vi and thermal image features F ir Adding them together yields the visible light characteristic F'. vi and thermal image features F' ir The process is as follows:

[0035]

[0036] F′ ir =Conv(X ir )+F ir ,F′ vi =Conv(X vi )+F vi

[0037] In the formula, Softmax is the normalized exponential function, and Conv is the convolution operation.

[0038] Channel attention values ​​are constructed by statistically calculating global features such as mean, maximum, and variance, and then mapped through a multilayer perceptron to obtain channel attention scores, thereby achieving weighted summation of interaction features and obtaining the final output visible light features F″. vi and thermal image features F″ ir The process is as follows:

[0039] F″ vi =F′ vi ⊙MLP(Concat(mean(F′ vi ),max(F′ vi ),var(F′ vi )))

[0040] F″ ir =F′ ir ⊙MLP(Concat(mean(F′ ir ),max(F′ ir ),var(F′ ir )))

[0041] F fused =F vi +F″ ir

[0042] In the formula, ⊙ represents the Hadamard product, MLP is a multilayer perceptron, Concat is the concatenation operation, mean is the mean function, max is the maximum value function, var is the variance function, and the final fusion output is the interactive fusion feature F. fused .

[0043] The multimodal interaction fusion features obtained above are input into the multimodal fusion feature clustering module, as follows:

[0044] First, during the feature clustering process, the feature map is divided into N token features, and an improved k-nearest neighbor density peak clustering algorithm is used.

[0045] The token set required by the feature clustering module is X = {x1, x2, ..., x}.n ,…x m ,…,x N The corresponding set of token spatial locations is Y = {y1, y2, ..., y}. n ,…y m ,…y N}, where N is the total number of tokens in the set, and d is the distance between any two tokens. n,m for:

[0046] d n,m =τ||x n -x m ||2+(1-τ)||y n -y m ||2

[0047] In the formula, τ is the weighting factor for semantic distance and spatial distance, ||·||2 represents Euclidean distance, and n and m represent the nth and mth token features, respectively.

[0048] Calculate the local density of each token based on its k nearest neighbors:

[0049] Wherein, the local density ρ calculated by the k nearest neighbors of the nth token is... n as follows:

[0050]

[0051] In the formula KNN(x) n ) represents the k nearest neighbors of the nth token, x n With x m These respectively represent the features of the nth and mth tokens.

[0052] Then, the minimum distance between each token and other tokens with higher local density is calculated as a distance metric. For the token with the highest local density, the maximum distance between it and any other token is calculated as a distance metric, as shown in the following formula:

[0053]

[0054] In the formula δ n It is a distance metric; combining local density and distance metric yields a score ρ for each token. n ×δ n The higher the score, the higher the score of the token will be determined as the cluster center.

[0055] Finally, based on feature distance, other tokens are assigned to the nearest cluster center to complete feature clustering;

[0056] For token feature merging, an importance score P is first defined to explicitly represent the importance of each token. P is calculated from the token features through a fully connected layer, expressed as: P = FC(X), where FC represents the linear mapping of the fully connected layer. The importance score P is then weighted and averaged using the Softmax function.

[0057]

[0058] In the formula, C n p is the set of the nth cluster. m It is a token feature x m Importance score It is the new token feature after token clustering and merging, completing feature clustering, and merging the clustered features. The input is fed into the second-stage encoding module, where j represents the j-th token feature.

[0059] S33. The features with a scale of (H / 4)×(W / 4) obtained from the interaction and fusion of visible light and thermal images in the first stage of S32, and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are transmitted to the multi-scale feature aggregation and decoding module through a skip connection to obtain the decoded features with a pixel size of (H / 4)×(W / 4), and then proceed to S34.

[0060] S34. Calculate the Euclidean distance between the decoded features and category features output in S33. Take the negative of the calculated distance value to obtain the output features. Construct a cross-entropy loss between the output features and the labels. Use this loss to update the network parameters of the multimodal feature fusion and clustering network to obtain the trained multimodal feature fusion and clustering model. Proceed to step 4.

[0061] Step 4: Input the test dataset into the multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test dataset, and test the accuracy of the multimodal feature fusion and clustering model.

[0062] Example 1

[0063] The low-light scene analysis method based on multimodal feature fusion and clustering described in this invention comprises the following steps:

[0064] Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; divide the uniformly sized images into training and testing datasets according to a ratio of 784 / 393, perform data augmentation on the training dataset to form the network training dataset; proceed to Step 2.

[0065] Step 2: Construct a multimodal feature fusion and clustering network:

[0066] The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multi-scale feature aggregation decoding module. The multimodal feature encoding network is constructed based on a Transformer network pre-trained on the ImageNet dataset and serves as the encoding network for feature extraction. The multimodal interaction fusion module is used for multimodal image feature interaction and fusion. The multimodal fusion feature clustering module is used to cluster the fused features obtained after multimodal interaction fusion. The multi-scale feature aggregation decoding module is used to receive multi-scale feature information obtained from the encoding network and classify the category of each feature pixel based on the semantic distance of the multi-scale features. Proceed to step 3.

[0067] Step 3: Train the multimodal feature fusion and clustering network using the network training dataset to obtain the trained multimodal feature fusion and clustering model.

[0068] S31. The feature extraction of the multimodal feature coding network is divided into four stages. The first stage is used to extract visible light and thermal image features, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features at the corresponding scales, which are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Proceed to S32.

[0069] S32. Input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interaction fusion module to obtain features of scale (H / 4)×(W / 4) after interaction fusion of visible light and thermal images. Then, transmit these features to the multimodal fusion feature clustering module to obtain clustering features, which are then input into the second feature extraction stage to obtain features of scale (H / 8)×(W / 8). Upsample the features of scale (H / 8)×(W / 8) obtained in the second stage and combine them with the features of scale (H / 4)×(W / 4) obtained in the first stage after interaction fusion of visible light and thermal images. The features are added together, and the result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the third feature extraction stage to obtain features with a scale of (H / 16)×(W / 16). The features with a scale of (H / 16)×(W / 16) obtained in the third stage are upsampled and added to the features with a scale of (H / 8)×(W / 8) obtained in the second stage. The result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the fourth feature extraction stage to obtain features with a scale of (H / 32)×(W / 32). The details are as follows:

[0070] The visible light characteristic is defined as F. vi Thermal image features are defined as F ir Then, the visible light feature F vi Visible light feature interaction vectors are obtained through convolution. With visible light eigenvectors Similarly, the thermal image features F ir Mapping thermal image feature interaction vectors With thermal image feature vector Interactive attention score A I Then it is calculated from the interaction vectors of the two modes:

[0071]

[0072] In the formula, represents matrix multiplication, and T represents the transpose operation.

[0073] The weighted visible light feature X is obtained by weighting and summing the value vectors of the two modalities using the interactive attention scores. vi and thermal image features X ir The weighted features of the obtained visible light and thermal images are convolved and then combined with the input visible light features F. vi and thermal image features F ir Adding them together yields the visible light characteristic F'. vi and thermal image features F' ir The process is as follows:

[0074]

[0075] F′ ir =Conv(X ir )+F ir ,F′ vi =Conv(X vi )+F vi

[0076] In the formula, Softmax is the normalized exponential function, and Conv is the convolution operation.

[0077] Channel attention values ​​are constructed by statistically calculating global features such as mean, maximum, and variance, and then mapped through a multilayer perceptron to obtain channel attention scores, thereby achieving weighted summation of interaction features and obtaining the final output visible light features F'. v ' i and thermal image features F' i ' r The process is as follows:

[0078] F″vi =F′ vi ⊙MLP(Concat(mean(F′ vi ),max(F′ vi ),var(F′ vi )))

[0079] F″ ir =F′ ir ⊙MLP(Concat(mean(F′ ir ),max(F′ ir ),var(F′ ir )))

[0080] F fused =F″ vi +F″ ir

[0081] In the formula, ⊙ represents the Hadamard product, MLP is a multilayer perceptron, Concat is the concatenation operation, mean is the mean function, max is the maximum value function, var is the variance function, and the final fusion output is the interactive fusion feature F. fused .

[0082] The multimodal interaction fusion features obtained above are input into the multimodal fusion feature clustering module, as follows:

[0083] First, during the feature clustering process, the feature map is divided into N token features, and an improved k-nearest neighbor density peak clustering algorithm is used.

[0084] The token set required by the feature clustering module is X = {x1, x2, ..., x}. n ,…x m ,…,x N The corresponding set of token spatial locations is Y = {y1, y2, ..., y}. n ,…y m ,…y N}, where N is the total number of tokens in the set, and d is the distance between any two tokens. n,m for:

[0085] d n,m =τ||x n -x m ||2+(1-τ)||y n -y m ||2

[0086] In the formula, τ is the weighting factor for semantic distance and spatial distance, ||·||2 represents Euclidean distance, and n and m represent the nth and mth token features, respectively.

[0087] Calculate the local density of each token based on its k nearest neighbors.

[0088] Wherein, the local density ρ calculated by the k nearest neighbors of the nth token is... n as follows:

[0089]

[0090] In the formula KNN(x) n ) represents the k nearest neighbors of the nth token, x n With x m These respectively represent the features of the nth and mth tokens.

[0091] Then, the minimum distance between each token and other tokens with higher local density is calculated as a distance metric. For the token with the highest local density, the maximum distance between it and any other token is calculated as a distance metric, as shown in the following formula:

[0092]

[0093] In the formula δ n It is a distance metric; combining local density and distance metric yields a score ρ for each token. n ×δ n The higher the score, the higher the score of the token will be determined as the cluster center.

[0094] Finally, based on feature distance, other tokens are assigned to the nearest cluster center, completing feature clustering.

[0095] For token feature merging, an importance score P is first defined to explicitly represent the importance of each token. P is calculated from the token features through a fully connected layer, expressed as: P = FC(X), where FC represents the linear mapping of the fully connected layer. The importance score P is then weighted and averaged using the Softmax function.

[0096]

[0097] In the formula, C n p is the set of the nth cluster. m It is a token feature x m Importance score It is the new token feature after token clustering and merging, completing feature clustering, and merging the clustered features. The input is fed into the second-stage encoding module, where j represents the j-th token feature.

[0098] S33. The multimodal interaction fusion feature with a scale of (H / 4)×(W / 4) obtained in the first stage of S32 and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained in the following three stages are respectively transmitted to the multi-scale feature aggregation and decoding module through a skip connection to obtain the decoded feature with a pixel size of (H / 4)×(W / 4), and then proceed to S34.

[0099] S34. Calculate the Euclidean distance between the decoded features and the category features output in S33, take the negative of the distance value to obtain the output features, construct the cross-entropy loss between the output features and the labels, and update the network parameters of the multimodal feature fusion and clustering network accordingly to obtain the trained multimodal feature fusion and clustering model, and proceed to step 4.

[0100] Step 4: Input the test dataset into the multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test dataset, and test the accuracy of the multimodal feature fusion and clustering model.

[0101] The method of this invention was tested on a computer configured with an i7 CPU and an NVIDIA V100 GPU, using a network built with Python programming language and the PyTorch deep learning framework. During training, the batch size was set to 2, the optimizer used was the AdamW optimizer with weight decay of 0.01, and the learning rate adjustment method employed was the Warm-up Polymer learning rate adjustment method, with an initial learning rate of 0.00006. Multiple batches were trained on the training sample set to obtain a low-light scene parsing method based on multimodal feature fusion and clustering. Visualization of the experimental results is shown below. Figure 2 As shown.

[0102] To demonstrate the superior performance of this invention, a recently advanced multimodal semantic segmentation method was selected as a comparison model. The comparative experimental results are shown in Table 1:

[0103] Table 1 shows the comparative experimental results of different methods on the MFNet database.

[0104]

[0105]

[0106] The model's parameter count and segmentation accuracy were evaluated on the MFNet dataset, with inputs being a visible light image (480×640×3) and a thermal image (480×640×3). Experimental results show that the method of this invention has 29.54M parameters, 36.6G computation, and achieves a segmentation accuracy of 57.9%, outperforming other algorithms.

Claims

1. A low-light scene analysis method based on multimodal feature fusion and clustering, characterized in that, The steps are as follows: Step 1: Normalize the 1569 images in the MFNet dataset, unifying the pixel size to H×W, where H represents the length and W represents the width; The images, after being standardized in size, are divided into training and testing datasets in a ratio of 784 / 393. Data augmentation is performed on the training dataset to form the network training dataset; proceed to step 2. Step 2: Construct a multimodal feature fusion and clustering network: The multimodal feature fusion and clustering network includes a multimodal feature encoding network, a multimodal feature interaction fusion module, a multimodal fusion feature clustering module, and a multi-scale feature aggregation decoding module. The multimodal feature encoding network is constructed based on a Transformer network pre-trained on the ImageNet dataset and serves as the encoding network for feature extraction. The multimodal interaction fusion module is used for multimodal image feature interaction and fusion. The multimodal fusion feature clustering module is used to cluster the fused features obtained after multimodal interaction fusion. The multi-scale feature aggregation decoding module is used to receive multi-scale feature information obtained from the encoding network and classify the category of each feature pixel based on the semantic distance of the multi-scale features. Proceed to step 3. Step 3: Train the multimodal feature fusion and clustering network using the network training dataset to obtain the trained multimodal feature fusion and clustering model. S31. The feature extraction of the multimodal feature coding network is divided into four stages. The first stage is used to extract visible light and thermal image features, and the following three stages are used to extract the features after the fusion of visible light and thermal images. Each stage extracts image features at the corresponding scales, which are (H / 4)×(W / 4), (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32), respectively. Proceed to S32. S32. Input the visible light and thermal image features extracted in the first stage of S31 into the multimodal feature interaction fusion module to obtain features of scale (H / 4)×(W / 4) after interaction fusion of visible light and thermal images. Then, transmit these features to the multimodal fusion feature clustering module to obtain clustering features, which are then input into the second feature extraction stage to obtain features of scale (H / 8)×(W / 8). Upsample the features of scale (H / 8)×(W / 8) obtained in the second stage and combine them with the features of scale (H / 4)×(W / 4) obtained in the first stage after interaction fusion of visible light and thermal images. The features are added together, and the result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the third feature extraction stage to obtain features with a scale of (H / 16)×(W / 16). The features with a scale of (H / 16)×(W / 16) obtained in the third stage are upsampled and added to the features with a scale of (H / 8)×(W / 8) obtained in the second stage. The result is input into the multimodal fusion feature clustering module to obtain clustering features, which are then input into the fourth feature extraction stage to obtain features with a scale of (H / 32)×(W / 32). Proceed to S33. S33. The features with a scale of (H / 4)×(W / 4) obtained from the first stage of S32 after interactive fusion of visible light and thermal images, and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained from the following three stages are transmitted to the multi-scale feature aggregation and decoding module through a skip connection to obtain the decoded features with a pixel size of (H / 4)×(W / 4), and then proceed to S34. S34. Calculate the Euclidean distance between the decoded features and category features output in S33. Take the negative of the calculated distance value to obtain the output features. Construct a cross-entropy loss between the output features and the labels. Update the network parameters of the multimodal feature fusion and clustering network based on this loss to obtain the trained multimodal feature fusion and clustering model. Proceed to step 4. Step 4: Input the test dataset into the multimodal feature fusion and clustering model, output the prediction result corresponding to each sample in the test dataset, and test the accuracy of the multimodal feature fusion and clustering model.

2. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 1, characterized in that, In step S32, the visible light and thermal image features extracted in the first stage of S31 are input into the multimodal feature interaction fusion module to obtain features with a scale of (H / 4)×(W / 4) after interaction fusion of visible light and thermal images. These features are then transmitted to the multimodal fusion feature clustering module to obtain clustering features, which are then input into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8). The specific process of the multimodal feature interaction fusion module is as follows: The visible light characteristic is defined as F. vi Thermal image features are defined as F ir Then, the visible light feature F vi Visible light feature interaction vectors are obtained through convolution. With visible light eigenvectors Similarly, the thermal image features F ir Mapping thermal image feature interaction vectors With thermal image feature vector Interactive attention score A I Then it is calculated from the interaction vectors of the two modes: In the formula, This represents matrix multiplication, and T represents the transpose operation; The weighted visible light feature X is obtained by weighting and summing the value vectors of the two modalities using the interactive attention scores. vi and thermal image features X ir The weighted features of the obtained visible light and thermal images are convolved and then combined with the input visible light features F. vi and thermal image features F ir Adding them together, we obtain the visible light feature F′. vi and thermal image features F′ ir The process is as follows: F′ ir =Conv(X ir )+F ir ,F′ vi =Conv(X vi )+F vi In the formula, Softmax is the normalized exponential function, and Conv is the convolution operation; Channel attention values ​​are constructed by statistically calculating global features of mean, maximum, and variance, and then mapped through a multilayer perceptron to obtain channel attention scores, thereby achieving weighted summation of interaction features and obtaining the final output visible light features F″. vi and thermal image features F″ ir .

3. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 2, characterized in that, The visible light and thermal image features extracted in the first stage of S32 are input into the multimodal feature interaction fusion module to obtain features with a scale of (H / 4)×(W / 4) after interaction fusion of visible light and thermal images. These features are then transmitted to the multimodal fusion feature clustering module to obtain clustered features, which are then input into the second feature extraction stage to obtain features with a scale of (H / 8)×(W / 8). The specific feature clustering operation is as follows: First, during the feature clustering process, the feature map is divided into N token features, and an improved k-nearest neighbor density peak clustering algorithm is used. The token set required by the feature clustering module is X = {x1, x2, ..., x}. n ,…x m ,…,x N The corresponding set of token spatial locations is Y = {y1, y2, ..., y}. n ,…y m ,…y N }, where N is the total number of tokens in the set, and d is the distance between any two tokens. n,m for: d n,m =τ||x n -x m ||2+(1-τ)||y n -y m ||2 In the formula, τ is the weighting factor for semantic distance and spatial distance, ||·||2 represents Euclidean distance, and n and m represent the nth and mth token features, respectively; Calculate the local density of each token based on its k nearest neighbors; The local density ρn of the k nearest neighbors of the nth token is calculated as follows: In the formula, KNN(xn) represents the k nearest neighbors of the nth token, and xn and xm represent the features of the nth and mth tokens, respectively; Then, the minimum distance between each token and other tokens with higher local density is calculated as a distance metric. For the token with the highest local density, the maximum distance between it and any other token is calculated as a distance metric, as shown in the following formula: In the formula, δ n It is a distance metric; combining local density and distance metric yields a score ρ for each token. n ×δ n The higher the score, the higher the score of the token will be, and the higher the score of the token will be, which will be the cluster center; ρ m This represents the local density calculated from the k nearest neighbors of the m-th token; Finally, based on feature distance, other tokens are assigned to the nearest cluster center to complete feature clustering; For token feature merging, an importance score P is first defined to explicitly represent the importance of each token. P is calculated from the token features through a fully connected layer, expressed as: P = FC(X), where FC represents the linear mapping of the fully connected layer. The importance score P is then weighted and averaged using the Softmax function. In the formula, C n p is the set of the nth cluster. m It is a token feature x m Importance score It is the new token feature after token clustering and merging, completing feature clustering, and merging the clustered features. The input is fed into the second-stage encoding module, where j represents the j-th token feature.

4. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 3, characterized in that, Channel attention values ​​are constructed by statistically calculating global features of mean, maximum, and variance, and then mapped through a multilayer perceptron to obtain channel attention scores, thereby achieving weighted summation of interaction features and obtaining the final output visible light features F″. vi and thermal image features F″ ir The process is as follows: F″ vi =F′ vi ⊙MLP(Concat(mean(F′ vi ),max(F′ vi ),var(F′ vi ))) F″ ir =F′ ir ⊙MLP(Concat(mean(F′ ir ),max(F′ ir ),var(F′ tr ))) F fused =F″ vi +F″ ir In the formula, ⊙ represents the Hadamard product, MLP is a multilayer perceptron, Concat is the concatenation operation, mean is the mean function, max is the maximum value function, var is the variance function, and the final fusion output is the interactive fusion feature Ffused.

5. The low-light scene analysis method based on multimodal feature fusion and clustering according to claim 4, characterized in that, In S33, the features with a scale of (H / 4)×(W / 4) obtained from the first stage of S32 after interactive fusion of visible light and thermal images, and the features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) obtained from the following three stages are transmitted to the multi-scale feature aggregation and decoding module through a skip connection to obtain decoded features with a pixel size of (H / 4)×(W / 4). The multi-scale feature aggregation and decoding module is specifically as follows: The features with scales of (H / 8)×(W / 8), (H / 16)×(W / 16), and (H / 32)×(W / 32) are upsampled to obtain features with a scale of (H / 4)×(W / 4), and then stitched together with the features with a scale of (H / 4)×(W / 4) obtained from the first stage of visible light and thermal image interaction fusion. XC=Concat(H 1 ,H 2 ,H 3 ,H 4 ) In the formula, H 1 H 2 H 3 H 4 These represent the output features of the first, second, third, and fourth stages of the encoder, respectively; XC represents the aggregated feature obtained by concatenating features from multiple scales. Then, semantic distance is defined based on Euclidean distance. The Softmax function is used to predict the segmentation results, as follows: In the formula, xt represents the learnable semantic category token of the t-th category, xs is the feature of the s-th token, and Rf is the final segmentation result; since the Softmax function is an increasing function, A smaller median indicates a closer distance, but the resulting classification probability is actually lower, contrary to the target. Therefore, before inputting the data into the Softmax function, the median value should be... Take the negative.

Images