Multi-modal remote sensing image clustering method based on dual feature selection

Abstract

The invention discloses a multi-mode remote sensing image clustering method based on dual feature selection, and belongs to the field of multi-mode remote sensing image clustering in remote sensing image information processing, and the method specifically comprises the steps: carrying out the dynamic correlation convolution processing of an initial hyperspectrum and a laser radar feature map, and obtaining initial feature maps in different modes; performing multi-scale convolution processing according to the preliminary feature maps in different modalities to obtain cross-scale feature maps of all the modalities; correlation weights between the hyperspectral data features and the laser radar data features are calculated according to the modal cross-scale feature maps, and residual connection and fusion are carried out on the correlation weights and the modal cross-scale feature maps to obtain a multi-modal fusion feature map; feature extraction is carried out according to the multi-modal fusion feature map, and the features are flattened into feature vectors; based on an online prototype clustering mechanism, combining the comparison loss function and the clustering loss function to obtain a clustering result; the multi-modal remote sensing image clustering method improves the accuracy and timeliness of multi-modal remote sensing image clustering.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-modal remote sensing image clustering in remote sensing image information processing, and particularly relates to a multi-modal remote sensing image clustering method based on dual feature selection. BACKGROUND

[0002] In the field of remote sensing image information processing, hyperspectral data and lidar data are two important data sources. Hyperspectral data can provide rich spectral information for identifying subtle component differences of ground objects; lidar data can provide high-precision terrain and ground object three-dimensional structure information. Joint clustering analysis of these two kinds of data can more comprehensively mine the class information of ground objects, such as vegetation types, land cover types, etc., which is of great significance to the fields of environmental monitoring, urban planning and resource management. However, due to the influence of factors such as sensor performance, environmental conditions, etc. in the acquisition process of hyperspectral data and lidar data, there are problems such as information misalignment and noise interference between the data. Existing clustering methods are mostly optimized for single data sources, and it is difficult to effectively handle multi-modal remote sensing image clustering problems. Therefore, an effective novel clustering method is needed, which can fully utilize the spectral information of hyperspectral data and the spatial structure information of lidar data, while overcoming the complexity and inconsistency of the data, to achieve more accurate ground object class division. SUMMARY

[0003] In order to solve the above problems, the present application provides a multi-modal remote sensing image clustering method based on dual feature selection, which comprises the following steps:

[0004] Respectively performing dynamic correlation convolution processing on the initial hyperspectral and lidar feature maps to obtain preliminary hyperspectral and lidar feature maps.

[0005] Performing multi-scale convolution processing on the preliminary hyperspectral and lidar feature maps to obtain cross-scale hyperspectral and lidar feature maps.

[0006] According to the cross-scale hyperspectral and lidar feature maps, the correlation weight between the hyperspectral data features and the lidar data features is calculated, and then the obtained correlation weight is connected with the cross-scale hyperspectral and lidar feature maps; then, the hyperspectral and lidar feature maps after residual connection are fused to obtain a multi-modal fusion feature map.

[0007] According to the multi-modal fusion feature map, feature extraction is performed and flattened into a feature vector.

[0008] According to each modal data, two enhanced views are generated, i.e. sample representations obtained after different transformations of the initial data, and a self-supervised joint training is performed by combining a contrast loss function and a clustering loss function.

[0009] Based on the online prototype clustering mechanism, the prototype clustering head is used for online clustering to obtain a clustering result.

[0010] According to the obtained clustering result, a clustering accuracy of the hyperspectral and laser radar data set is calculated.

[0011] Further, the expression of the dynamic correlation convolution processing of the initial hyperspectral and laser radar feature maps respectively is as follows:

[0012]

[0013] Y D_HSI represents a hyperspectral preliminary feature map processed by dynamic correlation convolution, Y D_LiDAR represents a laser radar preliminary feature map processed by dynamic correlation convolution, represents a convolution operation, X HSI represents an initial hyperspectral feature map, X LiDAR represents an initial laser radar feature map, K HSI represents a dynamically generated convolution kernel when the initial hyperspectral feature map is input, K LiDAR represents a dynamically generated convolution kernel when the initial laser radar data feature map is input, K HSI and K LiDAR The expression is as follows:

[0014]

[0015] K HSI and K LiDAR represent a dynamically generated convolution kernel, and represent a learnable parameter, represents an element-wise multiplication, T HSI and T LiDAR represent dynamic weights generated under different modalities, and the expression is as follows:

[0016]

[0017] T HSI and T LiDAR represent dynamic weights generated under different modalities, Conv(·) represents a 1×1 convolution operation, Pool(·) represents an adaptive average pooling operation, Softmax(·) represents an activation function, C HSI represents a channel number of X HSI , C LiDAR represents a channel number of X LiDAR , and C represents a channel number of X HSI and X LiDARThe number of channels, H represents X HSI and X LiDAR The high, W represents X HSI and X LiDAR The width of the convolution kernel is k, where k represents the spatial size of the convolution kernel.

[0018] Furthermore, the expression for multi-scale convolution processing based on the preliminary feature maps of hyperspectral imaging and lidar is as follows:

[0019]

[0020] Among them, Y MC_HSI Y represents the multi-scale hyperspectral feature map after multi-scale convolution processing. MC_LiDAR D represents the cross-scale lidar feature map after multi-scale convolution processing. k This represents the k-th convolutional kernel. This indicates that the feature maps are divided into four sub-feature maps on average according to the channel dimension. U represents the convolution operation between the k-th sub-feature map and the k-th convolution kernel. HSI For Y D_HSI Hyperspectral feature map after channel attention processing, U LiDAR For Y D_LiDAR The LiDAR feature map after channel attention processing is expressed as follows:

[0021]

[0022] Where Sigmoid(·) represents the activation function, Conv(·) represents a 1×1 convolution operation, and C represents Y. D_HSI and Y D_LiDAR The number of channels, H represents Y D_HSI and Y D_LiDAR The higher the value of Y, the more W represents the higher value of Y. D_HSI and Y D_LiDAR The width.

[0023] Furthermore, the expression for the multimodal fusion feature map is as follows:

[0024] Y cross_model =Concat(An HSI→LiDAR +Y MC_HSI An LiDAR→HSI +Y MC_LiDAR )

[0025] Among them, Y cross_model This represents a multimodal fusion feature map, where Concat(·) represents a concatenation operation along the channel dimension. HSI→LiDAR With An LiDAR→HSIrespectively represent the association weight of hyperspectral data features to lidar data features and the association weight of lidar data features to hyperspectral data features, and the expression is as follows:

[0026]

[0027] wherein, represents the dot product operation, Softmax(·) represents the activation function, d K_HSI and d K_LiDAR respectively represent the dimensions of K HSI and K LiDAR , Q HSI and Q LiDAR respectively represent the query matrix obtained by Y MC_HSI and Y MC_LiDAR , K HSI and K LiDAR respectively represent the key matrix obtained by Y MC_HSI and Y MC_LiDAR , V HSI and V LiDAR respectively represent the value matrix obtained by Y MC_HSI and Y MC_LiDAR , and the expression is as follows:

[0028]

[0029] wherein, Linear(·) represents the linear transformation operation.

[0030] Further, the expression of the feature extraction and flattening into a feature vector according to the multi-modal fusion feature map is as follows:

[0031] Z = flatten(Pool 4C×H×W→4C×1×1 (Conv 2C×H×W→4C×H×W (Y cross_model )))

[0032] wherein, Z represents the feature vector obtained by the multi-modal fusion feature map, Conv(·) represents the 1x1 convolution operation, flatten(·) represents the flattening operation, H represents the height of Y cross_model , and W represents the width of Y cross_model .

[0033] Further, the expression of the self-supervised joint training of the joint contrast loss function and the clustering loss function is as follows:

[0034] L total_Loss = L ins + λL clu

[0035] wherein, L total_Loss represents the total loss function, L insdenotes the contrastive loss function, L clu denotes the clustering loss function, λ denotes the weight coefficient, L ins The expression of L is as follows:

[0036]

[0037] where B denotes the batch size, i.e., the number of initial samples sampled in a single training, denotes the contrastive loss of the augmented view a of the jth sample, denotes the contrastive loss of the augmented view b of the jth sample, and The expression of L is as follows:

[0038]

[0039] where, denotes the feature representation of the augmented view a of the jth sample, denotes the feature representation of the augmented view a of the jth sample, y k denotes the feature representation of the kth sample in the batch, sim(·,·) denotes the cosine similarity function, τ denotes the temperature coefficient, l k≠j denotes an indicator function, which is 1 when k≠j, otherwise 0, the clustering loss L clu The expression of L is as follows:

[0040]

[0041] where c denotes the hyperspectral and lidar dataset class index, C denotes the total number of classes, y jc denotes the initial probability that the model output sample j belongs to class c, p jc denotes the normalized probability that sample j belongs to class c, and the expression is as follows:

[0042]

[0043] where c' denotes the class index, ρ c denotes the density weight of class c, and the expression is as follows:

[0044]

[0045] where, denotes the summation operation over all samples and augmented views within the batch.

[0046] Further, the expression of the online clustering using the prototype clustering head is as follows:

[0047]

[0048] where zj is the feature vector representation of sample j, and c is the prototype cluster center of class c, and is the class c that maximizes the probability of sample z jc is the class c that maximizes the probability of sample z j The class with the highest probability of sample is selected as the final clustering result, and the clustering results of all samples are obtained.

[0049] Further, according to the clustering results, the clustering accuracy is calculated.

[0050] The application provides a multi-modal remote sensing image clustering method based on dual feature selection, which has the following advantages:

[0051] (1) The method uses dynamic correlation convolution technology, which can dynamically generate different convolution kernels for different modal data, better capture the features of input data, and extract rich intra-modal local features.

[0052] (2) The method uses multi-scale convolution technology, which can fully utilize the rich spatial context information within the modal, and enhance the overall expression ability of data features under different modal.

[0053] (3) The method uses dual feature selection technology, which fully utilizes the information complementarity of data under different modal through feature interaction and selection between modal, and displays the long-distance dependence of cross-modal, thereby improving the clustering accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0055] Figure 1 is the flow chart of the multi-modal remote sensing image clustering method based on dual feature selection provided by the application.

[0056] Figure 2 is a schematic diagram of a multi-modal remote sensing data set Trento. DETAILED DESCRIPTION

[0057] For the purposes of promoting an understanding of the principles of the application, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the application. It will, nevertheless, be understood that no limitation of the scope of the application is intended by this description. Further, in the following description, numerous specific details are discussed in order to provide a thorough understanding of the application. However, it will be apparent to one skilled in the art that the application can be practiced without these specific details. In other instances, well-known structures and techniques have not been described in order to avoid obscuring the understanding of the application.

[0058] Exemplary method

[0059] As Figure 1 , the application provides a multi-modal remote sensing image clustering method based on dual feature selection, and the method steps are as follows:

[0060] Step S110: input the initial feature maps X HSI and X LiDAR of hyperspectral and laser radar respectively, calculate the dynamic weights T HSI and T LiDAR under different modalities, and the expressions are as follows:

[0061]

[0062] wherein Conv(·) represents a 1x1 convolution operation, Pool(·) represents an adaptive average pooling operation, and Softmax(·) represents an activation function. Then, according to the dynamic weights under different modalities, the dynamically generated convolution kernels K HSI and K LiDAR under different modalities are obtained, and the expressions are as follows:

[0063]

[0064] wherein and represent learnable parameters, represents element-wise multiplication. Then, dynamic correlation convolution is performed on the initial feature maps under each modality, so as to obtain the preliminary feature maps under different modalities, and the expressions are as follows:

[0065]

[0066] wherein Y D_HSI represents a hyperspectral preliminary feature map, Y D_LiDAR represents a laser radar preliminary feature map, represents a convolution operation.

[0067] Step S120: perform multi-scale convolution processing on the obtained preliminary feature maps Y D_HSI and Y D_LiDAR under different modalities, first calculate the weighted feature maps U HSI and U LiDAR , and the expressions are as follows:

[0068]

[0069] Where Sigmoid(·) represents the activation function, and Conv(·) represents a 1×1 convolution operation. Multi-scale convolution operations are then performed to further extract cross-scale features, as shown in the following expression:

[0070]

[0071] Among them, Y MC_HSI Y represents the multi-scale hyperspectral feature map after multi-scale convolution processing. MC_LiDAR D represents the cross-scale lidar feature map after multi-scale convolution processing. k This represents the k-th convolutional kernel. This indicates that the feature maps are divided into four sub-feature maps on average according to the channel dimension. U represents the convolution operation between the k-th sub-feature map and the k-th convolution kernel. HSI For Y D_HSI Hyperspectral feature map after channel attention processing, U LiDAR For Y D_LiDAR The LiDAR feature map after channel attention processing.

[0072] Step S130: Y MC_HSI With Y MC_LiDAR To perform dual feature selection, first calculate the value of Y. MC_HSI and Y MC_LiDAR The obtained query matrix Q HSI With Q LiDAR By Y MC_HSI and Y MC_LiDAR The obtained bond matrix K HSI With K LiDAR And by Y MC_HSI and Y MC_LiDAR The obtained value matrix V HSI With V LiDAR The expression is as follows:

[0073]

[0074] Here, Linear(·) represents a linear transformation operation. Then, the association weight An from the hyperspectral data features to the lidar data features is calculated. HSI→LiDAR The association weight An between lidar data features and hyperspectral data features LiDAR→HSI The expression is as follows:

[0075]

[0076] Where Θ represents the dot product operation, Softmax(·) represents the activation function, and dK_HSI with d K_LiDAR represent K HSI with K LiDAR dimension. Then An HSI→LiDAR and Y MC_HSI are connected in residual connection, An LiDAR→HSI and Y MC_LiDAR are connected in residual connection, and finally the two are spliced in the channel dimension to obtain the multi-modal fusion feature map Y cross_model , the expression is as follows:

[0077] Y cross_model = Concat(An HSI→LiDAR +Y MC_HSI , An LiDAR→HSI +Y MC_LiDAR ) (8)

[0078] Where, Concat(·) represents the splicing operation in the channel dimension.

[0079] Step S140: The obtained multi-modal fusion feature map Y cross_model is extracted, and then it is flattened into a feature vector Z, the expression is as follows:

[0080] Z = flatten(Pool 4C×H×W→4C×1×1 (Conv 2C×H×W→4C×H×W (Y cross_model ))) (9)

[0081] Where, flatten(·) represents the flattening operation. First, the multi-modal fusion feature map is processed by convolution, then further adaptive average pooling operation is performed to obtain Z, and then the flattening operation is performed to change it into a feature vector.

[0082] Step S150: The method model is trained, and two enhanced views are generated for each modal data. During training, a self-supervised joint training method combining the contrast loss function and the clustering loss function is used, the expression is as follows:

[0083] L total_Loss = L ins + λL clu (10)

[0084] Where, λ represents the weight coefficient, and the optimal clustering result is obtained by minimizing the total loss. Where, L total_Loss represents the total loss function, L ins represents the contrast loss function, L clu represents the clustering loss function, λ represents the weight coefficient, and the expression of L ins is as follows:

[0085]

[0086] where B denotes the batch size, i.e., the number of initial samples sampled in a single training, denotes the contrastive loss of the augmented view a of the jth sample, denotes the contrastive loss of the augmented view b of the jth sample, and The expression of L

[0087]

[0088] where denotes the feature representation of the augmented view a of the jth sample, denotes the feature representation of the augmented view a of the jth sample, y k denotes the feature representation of the kth sample in the batch, sim(·,·) denotes the cosine similarity function, τ denotes the temperature coefficient, l k≠j denotes an indicator function that is 1 when k≠j and 0 otherwise, the clustering loss L clu The expression of L

[0089]

[0090] where c denotes the hyperspectral and lidar dataset class index, C denotes the total number of classes, y jc denotes the initial probability that the model output sample j belongs to class c, p jc denotes the normalized probability that sample j belongs to class c, the expression is as follows:

[0091]

[0092] where c' denotes the class index, p c denotes the density weight of class c, the expression is as follows:

[0093]

[0094] where denotes the summation operation over all samples and augmented views within the batch.

[0095] Step S160: The feature vector Z obtained by the method model is sent to the online prototype clustering module, and first, the similarity between the feature representation z j of each sample and each prototype cluster center μ c is measured, that is, the probability that the sample belongs to the cth class is calculated, and the expression is as follows:

[0096]

[0097] where, a represents a degree of freedom parameter, for each sample, the class with the maximum probability is selected as its cluster label, and the expression is as follows:

[0098]

[0099] where, c is a class index, C is the total number of classes, finds the value of c that makes y jc The value of c that makes y

[0100] Step S170: According to the obtained clustering result, the clustering accuracy on the hyperspectral and lidar dataset is calculated.

[0101] Through the embodiment, first, the initial feature maps of the hyperspectral and lidar are respectively subjected to dynamic correlation convolution processing to obtain preliminary hyperspectral and lidar feature maps, then the preliminary hyperspectral and lidar feature maps are subjected to multi-scale convolution processing to obtain cross-scale hyperspectral and lidar feature maps, then the cross-scale hyperspectral and lidar feature maps are subjected to dual feature selection processing, the correlation weight between the hyperspectral data features and the lidar data features is calculated and connected with the cross-scale hyperspectral and lidar feature maps to obtain a multi-modal fusion feature map, then the multi-modal fusion feature map is subjected to feature extraction and flattened into a feature vector, a self-supervised joint training is performed on the contrast loss function and the clustering loss function, a clustering result is obtained based on an online prototype clustering mechanism, and finally the clustering accuracy on the dataset is obtained.

[0102] Further explanation, assuming that a multi-modal remote sensing dataset is clustered according to the embodiment, a clustering result with an accuracy higher than most methods will be obtained.

[0103] DETAILED DESCRIPTION

[0104] The embodiment adopts a published multi-modal remote sensing dataset. The details of the dataset are described as follows:

[0105] The Trento dataset is captured in the rural area of the city of Trento in Italy, composed of hyperspectral and lidar data, the hyperspectral data is collected by AISA Eagle sensor, the lidar data is captured by ALTM 3100EA sensor, containing six different ground objects, namely apple trees, buildings, ground, wood, vineyards, and roads, and the total number of samples of the dataset is 30214.

[0106] To verify the superiority of the present embodiment (Ours), the present embodiment is compared with several existing multi-modal remote sensing image clustering methods, including SMVSC, DCCA, BMVC and MDC method, and the clustering accuracy of six ground objects on the above public data set and the overall clustering accuracy (OA) on the above public data set will be compared. The specific data comparison is shown in Table 1.

[0107] Table 1 Clustering accuracy (%) of six ground objects on the Trento data set and overall clustering accuracy (%) on the Trento data set

[0108] Class SMVSC DCCA BMVC MDC Ours Apple trees 40.78 61.35 74.12 80.34 99.03 Buildings 52.05 60.87 0.23 99.93 100.00 Ground 15.63 0.12 5.87 5.01 28.60 Wood 95.42 99.10 97.89 98.99 100.00 Vineyard 70.89 71.52 45.12 60.73 99.12 Roads 90.60 89.63 88.80 79.14 91.71 OA 67.65 73.52 64.23 80.31 95.81

[0109] From the above table data comparison, it can be clearly seen that Ours achieves the best performance and significantly improves the clustering accuracy of multi-modal remote sensing images. The quantitative results fully illustrate the superiority of Ours. A large number of experiments show that the method is superior to existing methods. Regarding the parameter setting of the present embodiment, in all experiments, the weight coefficient λ is set to 1, the temperature coefficient τ is set to 0.5, and the degree of freedom parameter α is set to 1.

[0110] The present embodiment proposes a multi-modal remote sensing image clustering method based on dual feature selection. Through dynamic correlation convolution processing, preliminary hyperspectral and laser radar feature maps are obtained. Through multi-scale convolution, cross-scale hyperspectral and laser radar feature maps are obtained by further processing the preliminary hyperspectral and laser radar feature maps. The correlation weight between the hyperspectral data features and the laser radar data features is calculated by dual feature selection, and the cross-scale hyperspectral and laser radar feature maps are connected in residual to obtain multi-modal fusion feature maps. The multi-modal fusion feature maps are extracted and flattened into feature vectors. Finally, the self-supervised joint training is performed by combining the contrast loss function and the clustering loss function, and the final clustering result is obtained based on the online prototype clustering mechanism. The experimental results on the public data set Trento show that the present embodiment has better performance than other methods and has significant advantages.

[0111] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes of the principles of the present application, and do not constitute a limitation on the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. A multimodal remote sensing image clustering method based on dual feature selection, characterized in that, The method includes the following steps: Dynamic correlation convolution processing is performed on the initial hyperspectral and lidar feature maps respectively to obtain preliminary hyperspectral and lidar feature maps; Multi-scale convolution processing is performed on the preliminary feature maps of hyperspectral and lidar to obtain cross-scale hyperspectral and lidar feature maps. The correlation weights between hyperspectral data features and lidar data features are calculated based on the cross-scale hyperspectral and lidar feature maps. Then, the obtained correlation weights are residually connected with the cross-scale hyperspectral and lidar feature maps. Subsequently, the hyperspectral and lidar feature maps after residual connection are fused to obtain a multimodal fused feature map. Feature extraction is performed based on the multimodal fusion feature map, and then flattened into feature vectors; Two enhanced views are generated based on each modality of data, and the contrastive loss function and the clustering loss function are jointly trained under self-supervised supervision. Based on the online prototype clustering mechanism, online clustering is performed using the prototype clustering head to obtain the clustering results; Based on the obtained clustering results, the clustering accuracy of the hyperspectral and lidar datasets is calculated.

2. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, Dynamic correlation convolution is performed on the initial hyperspectral and lidar feature maps respectively, as shown below: Among them, Y D_HSI This represents the preliminary hyperspectral feature map after dynamic correlation convolution processing, Y D_LiDAR This represents the initial feature map of the lidar after dynamic correlation convolution processing. X represents the convolution operation. HSI Represents the initial hyperspectral feature map, X LiDAR K represents the initial lidar feature map. HSI With K LiDAR The following figures represent the dynamically generated convolutional kernels when the initial hyperspectral feature map and lidar feature map are input, respectively: Among them, K HSI With K LiDAR This represents a dynamically generated convolutional kernel, where k represents the spatial size of the convolutional kernel. and Indicates learnable parameters, T represents element-wise multiplication. HSI With T LiDAR The dynamic weights generated under different modalities are shown below: Among them, T HSI With T LiDAR Represents the dynamic weights generated under different modalities, Conv(·) represents a 1×1 convolution operation, Pool(·) represents an adaptive average pooling operation, Softmax(·) represents the activation function, and C HSI X represents HSI The number of channels, C LiDAR X represents LiDAR The number of channels, C represents X after adaptive average pooling and convolution processing. HSI and X LiDAR The number of channels, H represents X HSI and X LiDAR The high, W represents X HSI and X LiDAR The width of the convolution kernel is k, where k represents the spatial size of the convolution kernel.

3. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, Multi-scale convolution processing is performed based on the preliminary feature maps from hyperspectral and lidar data, as shown below: Among them, Y MC_HSI Represents the feature map of multi-scale hyperspectral data, Y MC_LiDAR D represents the feature map of cross-scale lidar data. k This represents the k-th convolutional kernel. This indicates that the feature maps are divided into four sub-feature maps on average according to the channel dimension. U represents the convolution operation between the k-th sub-feature map and the k-th convolution kernel. HSI For Y D_HSI Hyperspectral feature map after channel attention processing, U LiDAR For Y D_LiDAR The LiDAR feature map after channel attention processing is shown below: Where Sigmoid(·) represents the activation function, Conv(·) represents a 1×1 convolution operation, and C represents Y. D_HSI and Y D_LiDAR The number of channels, H represents Y D_HSI and Y D_LiDAR The higher the value of Y, the more W represents the higher value of Y. D_HSI and Y D_LiDAR The width.

4. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, The resulting multimodal fusion feature map is shown below: Y cross_model =Concat(An HSI→LiDAR +Y MC_HSI ,An LiDAR→HSI +Y MC_LiDAR ) Among them, Y cross_model This represents a multimodal fusion feature map, where Concat(·) represents a concatenation operation along the channel dimension. HSI→LiDAR With An LiDAR→HSI The association weights from hyperspectral data features to lidar data features and from lidar data features to hyperspectral data features are shown below, respectively: Where Θ represents the dot product operation, Softmax(·) represents the activation function, and d K_HSI With d K_LiDAR K HSI With K LiDAR The dimension of Q HSI With Q LiDAR They represent Y respectively MC_HSI and Y MC_LiDAR The obtained query matrix, K HSI With K LiDAR They represent Y respectively MC_HSI and Y MC_LiDAR The obtained bond matrix, V HSI With V LiDAR They represent Y respectively MC_HSI and Y MC_LiDAR The resulting value matrix is ​​shown below: Here, Linear(·) represents a linear transformation operation.

5. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, Feature extraction is performed based on the multimodal fusion features, and the features are flattened into feature vectors, as shown below: Z=flatten(Pool 4C×H×W→4C×1×1 (Conv 2C×H×W→4C×H×W (Y cross_model ))) Where Z represents the feature vector obtained by multimodal fusion, Conv(·) represents a 1×1 convolution operation, flatten(·) represents a flattening operation, and H represents Y. cross_model The higher the value of Y, the more W represents the higher value of Y. cross_model The width.

6. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, The joint contrastive loss function and the clustering loss function are subjected to self-supervised joint training, as shown below: THE total_Loss =L ins +λL clu Among them, L total_Loss L represents the total loss function. clu Let L represent the clustering loss function, λ represent the weight coefficients, and L represent the clustering loss function. ins The contrastive loss function is shown below: Where B represents the batch size, which is the initial number of samples collected in a single training iteration. This represents the contrast loss of the augmented view a for the j-th sample. The contrast loss of the augmented view b for the j-th sample is shown below: in, This represents the feature representation of the augmented view a of the j-th sample. The feature representation of the augmented view a of the j-th sample, y k Let sim(·,·) represent the feature representation of the k-th sample in the batch, τ represent the temperature coefficient, and l represent the feature representation of the k-th sample in the batch. k≠j Let L be an indicator function, which is 1 when k ≠ j and 0 otherwise. clu As shown below: Where c represents the category index of the hyperspectral and lidar dataset, C represents the total number of categories, and y jc p represents the initial probability that sample j in the model output belongs to class c. jc The normalized probability that sample j belongs to class c is shown below: Where c′ represents the category index, ρ c The density weights for category c are shown below: in, This indicates a summation operation on all samples and the augmented view within the batch.

7. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, Online clustering is performed using the prototype clustering head, as shown below: Among them, z j μ represents the feature vector representation of sample j. c The cluster center represents the prototype cluster center of category c, and α represents the degree of freedom parameter. This means finding, among all possible values ​​of category c, the one that makes y... jc To obtain the maximum value of c; calculate the sample z. j For each category c, the category with the highest probability is selected as the final clustering result, thus obtaining the clustering results for all samples.

8. The multimodal remote sensing image clustering method based on dual feature selection according to claim 1, characterized in that, Based on the clustering results, the clustering accuracy is calculated.

Images