Multi-modal remote sensing image classification method based on double hybrid experts

Abstract

The invention discloses a multi-modal remote sensing image classification method based on double hybrid experts, and the method comprises the steps: firstly carrying out the diversified dense mixing of spectral bands through a multi-head hybrid band spectrum hybrid expert; then, designing a multi-modal fusion hybrid expert to realize diversified feature fusion of the hyperspectral image and the laser radar data; in addition, selection of experts is balanced and fused by introducing information entropy; and finally, the candidate pseudo labels are trimmed through a proposed semi-supervised learning algorithm based on candidate pseudo label trimming, so that the unlabeled samples can obtain more reliable pseudo labels.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal remote sensing image processing, and in particular to a multimodal remote sensing image classification method based on dual hybrid experts. Background Art

[0002] Multimodal remote sensing images are obtained by photographing the ground with different sensors. They can reflect the characteristics of different aspects of ground objects. Therefore, they are widely used in many fields such as mineral exploration, environmental monitoring, forest management, precision agriculture, etc. Among them, hyperspectral image (HSI) is a high-dimensional image with a large number of spectral bands. It can simultaneously reflect the spatial distribution information and spectral reflectance information of ground objects and is often used as the main data for land cover classification tasks. LiDAR (Light detection and ranging) data can not only record the elevation information of the observed object but also has the characteristics of being less affected by environmental factors, such as weather. Due to the different imaging mechanisms, HSI and LiDAR data can be used to obtain different aspects of the characteristics of ground objects. Effective use of their complementary information can improve the performance of land cover classification tasks. Therefore, in remote sensing image classification, research on joint classification methods based on HSI and LiDAR data is a hot topic.

[0003] In the early stage of remote sensing image classification, the main method relied on data of one modality. With the development of deep learning technology in recent years, it has been widely used in classification tasks due to its powerful feature learning and pattern recognition capabilities. The Convolutional Neural Network (CNN) in deep learning uses weight sharing and local connection mechanisms, which can extract deep nonlinear abstract features with fewer parameters and has strong feature expression capabilities. Therefore, in the field of remote sensing image classification, many classification works are designed based on CNN. For example, a ripple CNN is designed to extract spectral features using the ripple transformation layer. Another example is to use the fast patchless global learning framework of the full convolutional network to fuse the global spatial information and semantic information of HSI. Usually, CNN pays more attention to local extraction and often does not work well in processing global spectral information, which may further make it difficult for the model to capture significant changes in the spectrum. For this reason, transformers have begun to attract people's attention. In particular, with the introduction of the vision transformer (ViT), people have found that applying transformers in the field of vision is a worthy exploration, and usually when applying transformers, the characteristics of CNN local feature extraction are also used. In order to simultaneously utilize ViT's self-attention mechanism to capture global spectral information and CNN's processing of spatial position information or associations between pixels, thereby achieving effective transmission and interaction of spatial information, many methods combining CNN and ViT have been proposed. For example, the spectral spatial feature tokenization transformer is used to capture spectral spatial features and high-level semantic features. Another example is combining convolutional telemetry with transformers to achieve the annotation and utilization of spectral differential information and local spatial context information.

[0004] Because different types of ground objects may have similar spectral curves, the method that only uses HSI as a modal data will have more misclassification phenomena. Fortunately, LiDAR data can distinguish them by providing elevation information of the objects to be classified, thus effectively solving the above problem. Therefore, in the process of land cover classification, the complementary information of HSI and LiDAR modal data is effectively used to extract more discriminative fusion features, which helps to improve the classification performance. In the field of remote sensing image classification, some joint classification methods of HSI and LiDAR modal data based on CNN and transformer have also been proposed one after another. For example, two independent CNN branches are used to extract the deep features of HSI and LiDAR respectively, and a decision-level fusion method is designed to realize the fusion of multi-modal deep features. For example, CNN is used to extract multi-scale local features of HSI and LiDAR data respectively, and then the fusion of HSI and LiDAR data features is realized based on the designed feature-level fusion method, and finally the transformer branch is used to extract the fused features globally.

[0005] The above-mentioned multimodal joint classification methods based on CNN and transformer all adopt a single fusion strategy, which will limit the model to a one-sided or even suboptimal fusion strategy, such as over-reliance on the characteristics of a certain modality data. By fusing multiple expert networks through the Mixtures of Experts (MoE), diversified and dynamic multimodal fusion can be achieved. In addition, while HSI provides rich spectral features, there is also a phenomenon that neighboring bands have strong similarities. Using the selection and mixing strategy of MoE, a subset is selected from the redundant bands and weighted aggregation is performed. However, this sparse selection strategy will inevitably lead to the loss of spectral information.

[0006] Almost all supervised deep learning methods rely on a large number of labeled samples, otherwise serious overfitting will occur. However, the cost of pixel-by-pixel labeling in remote sensing images is high. For this reason, many semi-supervised learning strategies have been proposed one after another. For example, a semi-supervised image registration framework based on multimodal cross-attention is proposed, which maps cross-modal features to the decoder for key point detection, and expands the key point set of unlabeled samples by continuously adding new reliable key points extracted. Another example is clustering the labels of unlabeled samples into pseudo-labels and applying them to the pre-training of deep CNNs, thereby improving the initialization of the classification model. The performance of the aforementioned semi-supervised learning methods based on pseudo-labels depends to a large extent on the accuracy of pseudo-labels. Summary of the invention

[0007] The present application provides a multimodal remote sensing image classification method based on dual hybrid experts to solve the problems faced in the joint classification process of HSI and LiDAR data, such as redundant bands of HSI, single fusion strategy of multimodal data, and high cost of pixel-by-pixel labeling of remote sensing images.

[0008] The present application embodiment provides a multimodal remote sensing image classification method based on dual hybrid experts, comprising the following steps:

[0009] Step 1: By selecting several pixels around a specific pixel, a neighbor representation of the hyperspectral image is constructed, the spectral features of the hyperspectral image are diversified and mixed using multi-head mixed bands, and a two-dimensional convolutional neural network is used to extract spatial features;

[0010] Step 2: By selecting several pixels around a specific pixel, a neighbor representation of the lidar data is constructed, and a two-dimensional convolutional neural network is used to extract spatial features of the lidar data to obtain the same number of channels as the hyperspectral image. The hyperspectral image and lidar data features are fused using a multimodal fusion hybrid expert to obtain fused features;

[0011] Step 3: Use the visual transformer to perform global feature extraction on the fused features, and use the candidate pseudo-label pruning method to screen the candidate pseudo-labels of unlabeled samples twice, both within and between samples, from the perspective of model prediction confidence and feature similarity.

[0012] Optionally, in one embodiment of the present application, in step 1, given a hyperspectral image Where m and n are the width and height of the region, respectively, and d (H) is the number of channels of the hyperspectral image; the neighbor representation of the hyperspectral image is Where s is the spatial size of the nearest neighbor representation, and the vectorized hyperspectral image representation definition is the i-th spectral mixing expert, then the band mixing feature is expressed as:

[0013]

[0014] In the formula, is the band mixing feature; definition Spectral mixing matrix Then the multi-spectral mixing feature of the i-th spectral mixing expert is Define the contribution scores of different hybrid experts obtained by the gating network It is expressed as:

[0015]

[0016] In the formula, is the weight of the gating network, n (B) is the number of spectral mixing experts; according to the contribution score g, from n (B) Select K from among the spectral mixture experts B activation, the process is expressed as:

[0017] G B =σ(TopK B [g])

[0018] In the formula, σ(·) is the softmax function, TopK B [·] is to select K B maximum value and set the rest of the values ​​to -∞. n (B) The activation value of the spectral mixture expert; using the activation value G B The weighted summation of the selected spectral mixing experts is realized, and the process is expressed as:

[0019]

[0020] In the formula, is the output feature of the spectral mixing band; then a two-dimensional convolutional neural network is used to (H) Perform spatial feature extraction. is the output feature after convolution, where d M is the feature dimension of the output hyperspectral image.

[0021] Optionally, in one embodiment of the present application, in step 2, the laser radar data of the same area corresponding to the original hyperspectral image is In this paper, we construct a neighbor representation of the lidar data. Among them, d (L) is the number of channels of the LiDAR data; the output features of the LiDAR data are obtained using a two-dimensional convolutional neural network d M As the output feature dimension, multimodal fusion hybrid expert pair and Perform feature fusion, the process is expressed as:

[0022]

[0023] In the formula, α i is the i-th fusion expert, ⊙ is the dot multiplication operation, F i is the fusion result of the i-th fusion hybrid expert; for each neighbor representation, its corresponding fusion strategy is from n M The fusion hybrid experts are selected based on the contribution scores provided by the gating network. The contribution score h corresponding to each fusion hybrid expert is calculated as follows:

[0024]

[0025] In the formula, is the weight of the gating network in the multimodal fusion hybrid expert; the selection strategy of the fusion hybrid expert is expressed as:

[0026] G M =σ(TopK F [h])

[0027] In the formula, through TopK F [·] Select the top K F The rest of the values ​​are set to -∞. is the activation value corresponding to the fusion hybrid expert; the fusion result F of multiple fusion hybrid experts (M) It is obtained by weighted summation of the selected fusion hybrid experts. The process is expressed as:

[0028]

[0029] In the formula, It is the fusion feature output by the multimodal fusion hybrid expert. The information entropy is used to measure the selection balance of the fusion hybrid expert. The process is expressed as:

[0030]

[0031] Where P = σ(h) is the probability distribution of the hybrid expert being selected; when H (M) When the value is small, it indicates that only a few fusion experts correspond to higher probabilities. (M) When the value is large, it indicates that the probability of the fusion expert being selected is relatively average, and the minimum H (M) To encourage the gating network to make more balanced choices, the process is expressed as:

[0032]

[0033] Where n (S) is the number of labeled samples, n (U) is the number of unlabeled samples, This is the load balancing loss.

[0034] Optionally, in one embodiment of the present application, in step 3, a fusion feature is obtained using a multimodal fusion hybrid network, and information entropy is used to calculate the balanced loss; the fusion feature F is fused using a visual transformer. (M) To extract at a deeper level, the fusion results of multiple fusion experts are combined into (M) With a classification Splice, get Mark the position of h0 The computation process of the visual transformer is expressed as:

[0035]

[0036] Where LN(·) and MHA(·) represent layer normalization and multi-head attention mechanisms respectively, and L represents the number of self-attention modules in the visual transformer. Multi-head attention is obtained by splicing multiple self-attention heads, and its process is expressed as:

[0037] MHA(Q,K,V)=Concat(SA1,SA2,…,SA h )W M

[0038] In the formula, Concat(·) represents the feature concatenation operation, represents the output of the i-th self-attention head, Q i , K i and V i The query, key, and value of the i-th self-attention head are respectively, through three mapping matrices and get, is the scaling factor, h is the number of attention heads, is the linear mapping matrix; the output feature F of the visual transformer (T) It is expressed as:

[0039]

[0040] In the formula, FFN(·) is a feed-forward layer. The pseudo labels of unlabeled samples are obtained by using the above process, and the process is expressed as:

[0041]

[0042] In the formula, f θ (·) is the feature transformation achieved by the above process, p i is the class probability vector predicted for the i-th unlabeled sample, and C is the total number of categories. The label set is screened from the perspective of the sample, and the process is expressed as:

[0043] p′ i =TopK L (p i )

[0044] In the formula, TopK L (·) is to retain the largest K L values, and set the rest of the values ​​to 0. The candidate label set for the i-th unlabeled sample is Use the K samples with the highest similarity to the i-th sample feature N The auxiliary set of nearest neighbors The number of times the candidate tags appear is counted. The process is expressed as:

[0045]

[0046] In the formula, A larger count(c) indicates that the number of times label c appears in the auxiliary set is large, and the candidate label has a higher class consistency in the local neighborhood of sample i. The final pseudo label is obtained by using the threshold clipping strategy. The process is expressed as:

[0047] γ i =τ(|p i ′|-1)

[0048] Where τ is a hyperparameter, |·| is the number of elements to be removed; when count(c)<γ i , the category c will be changed from Prune it in the middle to get the final pseudo label and the corresponding one-hot vector Loss function of multimodal remote sensing image classification method based on dual mixed experts The classification loss on the labeled sample set Classification loss on the unlabeled sample set and load balancing loss The composition process can be expressed as:

[0049]

[0050] In the formula, λ is the balance coefficient.

[0051] The multimodal remote sensing image classification method based on dual hybrid experts in the embodiment of the present application has the following beneficial effects:

[0052] (1) Design one or more mixed bands. On the one hand, the gating mechanism is used to give important bands higher weights, thereby avoiding the problem of band information loss caused by sparse MoE; on the other hand, in order to avoid over-reliance on a single aggregation result and obtain more diverse aggregation features, the above method is expanded to multiple heads;

[0053] (2) In view of the fact that a single multimodal fusion strategy leads to a one-sided and suboptimal fusion model, a multimodal fusion hybrid expert strategy is proposed. Different experts correspond to different multimodal fusion strategies, and the various fusion strategies are weighted and mixed through the importance score of the gating network. In addition, the load imbalance problem is alleviated and a more balanced expert selection is achieved by minimizing the information entropy;

[0054] (3) In order to alleviate the overfitting problem caused by insufficient labeled samples, a candidate pseudo-label pruning method is proposed. The candidate label set is screened from two perspectives, namely, within-sample and between-sample, to obtain more reliable pseudo-labels for unlabeled samples, thereby reducing the negative impact of incorrect pseudo-labels on pseudo-label-dependent semi-supervised learning algorithms.

[0055] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0057] Figure 1 The present invention provides a flowchart of a multimodal remote sensing image classification method based on dual hybrid experts according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0059] Multimodal joint classification methods based on convolutional neural networks (CNNs) and transformers usually adopt a single fusion strategy, which limits the model to one-sided or even suboptimal fusion strategies, such as over-reliance on the features of a certain modality data. In addition, while hyperspectral images provide rich spectral features, there is also a phenomenon that neighboring bands have strong similarities. The use of hybrid experts can achieve diversified and dynamic multimodal fusion, but the sparse selection strategy of hybrid experts will inevitably lead to the loss of spectral information. In addition, the performance of conventional pseudo-label-based semi-supervised learning methods depends to a large extent on the accuracy of pseudo-labels. To this end, the present invention proposes a method based on candidate pseudo-label pruning and dual hybrid experts, and uses it for the joint classification task of hyperspectral images and lidar data. Firstly, a neighbor representation of the hyperspectral image is constructed by selecting several pixels around a specific pixel, and then the spectral features of the hyperspectral image are diversified by using multi-head mixed bands, and the spatial feature extraction is further realized with the help of a two-dimensional convolutional neural network; then, a neighbor representation of the lidar data is constructed by selecting several pixels around a specific pixel, and then the lidar data is spatially extracted by using two-dimensional convolution to obtain the same number of channels as the hyperspectral image, and then the multimodal fusion hybrid expert is used to dynamically and diversifiedly fuse the features of the hyperspectral image and the lidar data, and the corresponding fusion features are obtained; finally, the visual transformer is used to realize global feature extraction of the fused features. In addition, in order to obtain more reliable pseudo-labels and thus help realize effective semi-supervised learning, the candidate pseudo-label pruning method is used to screen the candidate pseudo-labels of unlabeled samples twice, both within the sample and between the samples, from the perspectives of model prediction confidence and feature similarity.

[0060] Specifically, Figure 1 The present invention provides a flowchart of a multimodal remote sensing image classification method based on dual hybrid experts according to an embodiment of the present application.

[0061] like Figure 1 As shown, the multimodal remote sensing image classification method based on dual mixed experts includes the following steps:

[0062] Step 1: By selecting several pixels around a specific pixel, a neighbor representation of the hyperspectral image is constructed, and then the spectral features of the hyperspectral image are diversified by using multi-head mixed bands, and the spatial feature extraction is further realized with the help of a two-dimensional convolutional neural network;

[0063] Step 2: By selecting several pixels around a specific pixel, a neighbor representation of the lidar data is constructed. Then, in order to obtain the same number of channels as the hyperspectral image, two-dimensional convolution is used to extract spatial features of the lidar data. Then, multimodal fusion hybrid experts are used to dynamically and diversely fuse the features of the hyperspectral image and lidar data, and the corresponding fusion features are obtained.

[0064] Step 3: Use visual transformer to extract global features from fused features. In addition, in order to obtain more reliable pseudo-labels and help achieve effective semi-supervised learning, the candidate pseudo-label pruning method is used to screen the candidate pseudo-labels of unlabeled samples twice, both within the sample and between samples, from the perspective of model prediction confidence and feature similarity.

[0065] In step 1, given a hyperspectral image Where m and n represent the width and height of this area respectively, d (H) Represents the number of channels of the hyperspectral image; the nearest neighbor representation of the hyperspectral image Where s is the spatial size of the nearest neighbor representation, and the vectorized hyperspectral image representation While HSI provides rich spectral features, there is also a phenomenon that neighboring bands have strong similarities. Usually, MoE is based on a sparse strategy, that is, the gating network only activates some experts for forward calculations each time. Therefore, MoE can improve the capabilities of the model without increasing the forward calculation cost. However, when MoE is directly applied to the band mixing of HSI, the sparse strategy will cause only some bands to be activated, resulting in the loss of spectral information. To this end, a method based on dense multi-head mixing bands is proposed to solve the above problem. Definition is the i-th spectral mixing expert, then the band mixing feature is expressed as:

[0066]

[0067] In the formula, is the band mixing feature; definition Spectral mixing matrix Then the multi-spectral mixing feature of the i-th spectral mixing expert is Define the contribution scores of different hybrid experts obtained by the gating network It is expressed as:

[0068]

[0069] In the formula, is the weight of the gating network, n (B) is the number of spectral mixing experts; according to the contribution score g, from n (B)The process of selecting KB activations from spectral mixture experts is expressed as:

[0070] G B =σ(TopK B [g])

[0071] In the formula, σ(·) is the softmax function, TopK B [·] selects the maximum value of KB and sets the rest of the values ​​to -∞. n (B) The activation value of the spectral mixture expert; using the activation value G B The weighted summation of the selected spectral mixing experts is realized, and the process is expressed as:

[0072]

[0073] In the formula, is the output feature of the spectral mixing band; then a two-dimensional convolutional neural network is used to (H) Perform spatial feature extraction. is the output feature after convolution, where d M is the feature dimension of the output hyperspectral image.

[0074] In step 2, HSI and LiDAR data can reflect different aspects of ground objects. By effectively fusing the images of the two modalities, more discriminative features can be obtained, which helps improve classification accuracy. In order to overcome the one-sidedness of the single HSI and LiDAR data fusion strategy, a multimodal fusion hybrid expert method is proposed to achieve diversified fusion of HSI and LiDAR data. First, the lidar data of the same area corresponding to the original hyperspectral image is obtained. In this paper, we construct a neighbor representation of the lidar data. where d (L) Represents the number of channels of LiDAR data; Use a two-dimensional convolutional neural network to obtain the output features of LiDAR data dM is the feature dimension of its output. and The process of feature fusion is expressed as follows:

[0075]

[0076] In the formula, α i is the i-th fusion expert, ⊙ is the dot multiplication operation, F i is the fusion result of the i-th fusion expert; for each neighbor representation, its corresponding fusion strategy can be selected from n MThe fusion experts are selected based on the contribution scores provided by the gating network. The contribution score h corresponding to each fusion expert is calculated as follows:

[0077]

[0078] In the formula, is the weight of the gating network in the multimodal fusion hybrid expert; the selection strategy of the fusion expert is expressed as:

[0079] G M =σ(TopK F [h])

[0080] In the formula, through TopK F [·] Select the top K F The rest of the values ​​are set to -∞. is the activation value corresponding to the fusion expert; the fusion result F of multiple fusion experts (M) It can be obtained by weighted summation of the selected fusion experts, and the process is expressed as:

[0081]

[0082] In the formula, It is the fusion feature output by the multimodal fusion hybrid expert. The information entropy is used to measure the selection balance of the fusion expert. The process is expressed as:

[0083]

[0084] Where P = σ(h) is the probability distribution of the hybrid expert being selected; when H (M) When the value is small, it indicates that only a few fusion experts have a higher probability. (M) When the value is large, it indicates that the probability of the fusion expert being selected is relatively average, and the minimum H (M) To encourage the gating network to make more balanced choices, the process is expressed as:

[0085]

[0086] Where n (S) is the number of labeled samples, n (U) is the number of unlabeled samples, This is the load balancing loss.

[0087] In step 3, a multimodal fusion hybrid network is used to obtain fusion features, and information entropy is used to calculate the balanced loss; the fusion feature F is transformed using a visual transformer. (M) To perform a deeper extraction, F (M) With a classification token Splice, get Mark the position of h0 The computation process of the visual transformer is expressed as:

[0088]

[0089] Where LN(·) and MHA(·) represent layer normalization and multi-head attention mechanisms respectively, and L represents the number of self-attention modules in the visual transformer. Multi-head attention is obtained by splicing multiple self-attention heads, and its process is expressed as:

[0090] MHA(Q,K,V)=Concat(SA1,SA2,…,SA h )W M

[0091] In the formula, Concat(·) represents the feature concatenation operation, represents the output of the i-th self-attention head, Q i , K i and V i The query, key, and value of the i-th self-attention head are respectively, through three mapping matrices and get, is the scaling factor, h is the number of attention heads, is the linear mapping matrix; the output feature F of the visual transformer (T) It is expressed as:

[0092]

[0093] In the formula, FFN(·) is a feed-forward layer. The pseudo labels of unlabeled samples can be obtained by using the above process, and the process is expressed as:

[0094]

[0095] In the formula, f θ (·) is the feature transformation achieved by the above process, p i is the class probability vector predicted for the i-th unlabeled sample, and C is the total number of categories. The label set is screened from the perspective of the sample, and the process is expressed as:

[0096] p′ i =TopK L (p i )

[0097] In the formula, TopK L (·) is to retain the largest K L values, and set the rest of the values ​​to 0. The candidate label set for the i-th unlabeled sample is Use the K samples with the highest similarity to the i-th sample feature N The auxiliary set of nearest neighbors The process of counting the number of times the candidate tags appear is expressed as follows:

[0098]

[0099] In the formula, A larger count(c) indicates that the number of times label c appears in the auxiliary set is large, and the candidate label has a higher class consistency in the local neighborhood of sample i. The final pseudo label is obtained by using the threshold clipping strategy, and the process is expressed as:

[0100] γ i =τ(|p i ′|-1)

[0101] Where τ is a hyperparameter, |·| is the number of elements to be removed; when count(c)<γ i , the category c will be changed from Prune it in the middle to get the final pseudo label and the corresponding one-hot vector Loss function of multimodal remote sensing image classification method based on dual mixed experts Mainly composed of the classification loss on the labeled sample set Classification loss on the unlabeled sample set and load balancing loss The process can be expressed as:

[0102]

[0103] In the formula, λ is the balance coefficient.

[0104] According to the multimodal remote sensing image classification method based on dual hybrid experts proposed in an embodiment of the present invention, a spectral mixing expert with multiple mixed bands is used to perform diverse dense mixing of spectral bands; a multimodal fusion hybrid expert is designed to realize diversified feature fusion of hyperspectral images and lidar data; the selection of fusion experts is balanced by introducing information entropy; and candidate pseudo-labels are pruned by the proposed semi-supervised learning algorithm based on candidate pseudo-label pruning, so that unlabeled samples obtain more reliable pseudo-labels.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0107] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

Claims

1. A multimodal remote sensing image classification method based on dual mixed experts, characterized in that: The following steps are involved: Step 1: By selecting several pixels around a specific pixel, a neighbor representation of the hyperspectral image is constructed, the spectral features of the hyperspectral image are diversified and mixed using multi-head mixed bands, and a two-dimensional convolutional neural network is used to extract spatial features; Step 2: By selecting several pixels around a specific pixel, a neighbor representation of the lidar data is constructed, and a two-dimensional convolutional neural network is used to extract spatial features of the lidar data to obtain the same number of channels as the hyperspectral image. The hyperspectral image and lidar data features are fused using a multimodal fusion hybrid expert to obtain fused features; Step 3: Use the visual transformer to perform global feature extraction on the fused features, and use the candidate pseudo-label pruning method to screen the candidate pseudo-labels of unlabeled samples twice, both within and between samples, from the perspective of model prediction confidence and feature similarity.

2. The method according to claim 1, characterized in that In step 1, given a hyperspectral image Where m and n are the width and height of the region, respectively, and d (H) is the number of channels of the hyperspectral image; the nearest neighbor representation of the hyperspectral image Where s is the spatial size of the nearest neighbor representation, and the vectorized hyperspectral image representation definition is the i-th spectral mixing expert, then the band mixing feature is expressed as: In the formula, is the band mixing feature; definition Spectral mixing matrix Then the multi-spectral mixing feature of the i-th spectral mixing expert is Define the contribution scores of different hybrid experts obtained by the gating network It is expressed as: In the formula, is the weight of the gating network, n (B) is the number of spectral mixing experts; according to the contribution score g, from n (B) Select K from among the spectral mixture experts B activation, the process is expressed as: G B σ(TopK B [g]) In the formula, σ(·) is the softmax function, TopK B [·] is to select K B maximum value and set the rest of the values ​​to -∞. n (B) The activation value of the spectral mixture expert; using the activation value G B The weighted summation of the selected spectral mixing experts is realized, and the process is expressed as: In the formula, is the output feature of the spectral mixed band; then a two-dimensional convolutional neural network is used to (H) Perform spatial feature extraction. is the output feature after convolution, where d M is the feature dimension of the output hyperspectral image.

3. The method according to claim 2, characterized in that In step 2, the lidar data of the same area corresponding to the original hyperspectral image is obtained. In this paper, we construct a neighbor representation of the lidar data. Among them, d (L) is the number of channels of the LiDAR data; the output features of the LiDAR data are obtained using a two-dimensional convolutional neural network d M As the output feature dimension, multimodal fusion hybrid expert pair and Perform feature fusion, the process is expressed as: In the formula, α i is the i-th fusion expert, ⊙ is the dot multiplication operation, F i is the fusion result of the I-th fusion hybrid expert; for each neighbor representation, its corresponding fusion strategy is from n M The fusion hybrid experts are selected based on the contribution scores provided by the gating network. The contribution score h corresponding to each fusion hybrid expert is calculated as follows: In the formula, is the weight of the gating network in the multimodal fusion hybrid expert; the selection strategy of the fusion hybrid expert is expressed as: G M σ(TopK F [h]) In the formula, through Top K F [·] Select the top K F The rest of the values ​​are set to -∞. is the activation value corresponding to the fusion hybrid expert; the fusion result F of multiple fusion hybrid experts (M) It is obtained by weighted summation of the selected fusion hybrid experts. The process is expressed as: In the formula, It is the fusion feature output by the multimodal fusion hybrid expert. The information entropy is used to measure the selection balance of the fusion hybrid expert. The process is expressed as: Where P = σ(h) is the probability distribution of the hybrid expert being selected; when H (M) When the value is small, it indicates that only a few fusion experts correspond to higher probabilities. (M) When the value is large, it indicates that the probability of the fusion expert being selected is relatively average, and the minimum H (M) To encourage the gating network to make more balanced choices, the process is expressed as: Where n (S) is the number of labeled samples, n (U) is the number of unlabeled samples, This is the load balancing loss.

4. The method according to claim 3, characterized in that In step 3, a multimodal fusion hybrid network is used to obtain fusion features, and information entropy is used to calculate the balanced loss; the fusion feature F is transformed using a visual transformer. (M) To extract at a deeper level, the fusion results of multiple fusion experts are combined into (M) With a classification token Splice, get Mark the position of h0 The computation process of the visual transformer is expressed as: Where LN(·) and MHA(·) represent layer normalization and multi-head attention mechanisms respectively, and L represents the number of self-attention modules in the visual transformer. Multi-head attention is obtained by splicing multiple self-attention heads, and its process is expressed as: MHA(Q,K,V)=Concat(SA1,SA2,…,SA h )W M In the formula, Concat(·) represents the feature concatenation operation, represents the output of the i-th self-attention head, Q i , K i and V i The query, key, and value of the i-th self-attention head are respectively, through three mapping matrices and get, is the scaling factor, h is the number of attention heads, is the linear mapping matrix; the output feature F of the visual transformer (T) It is expressed as: In the formula, FFN(·) is a feed-forward layer. The pseudo labels of unlabeled samples are obtained by using the above process, and the process is expressed as: In the formula, f θ (·) is the feature transformation achieved by the above process, p i is the class probability vector predicted for the i-th unlabeled sample, and C is the total number of categories. The label set is screened from the perspective of the sample, and the process is expressed as: p′ i =TopK L (p i ) In the formula, TopK L (·) is to retain the largest K L values, and set the rest of the values ​​to 0. The candidate label set for the i-th unlabeled sample is c i ={c|p′ i，c ≠0}; use the K samples with the highest similarity to the i-th sample feature N The auxiliary set of nearest neighbors The number of times the candidate tags appear is counted. The process is expressed as: Where c∈C i ; A larger count(c) indicates that the number of times label c appears in the auxiliary set is large, and the candidate label has a higher class consistency in the local neighborhood of sample i; the final pseudo label is obtained by using the threshold clipping strategy, and the process is expressed as: c i =τ(|p i ′|-1) Where τ is a hyperparameter, |·| is the number of elements to be removed; when count(c)<γ i , the category c will be changed from C i Prune it in the middle to get the final pseudo label and the corresponding one-hot vector Loss function of multimodal remote sensing image classification method based on dual mixed experts The classification loss on the labeled sample set Classification loss on the unlabeled sample set and load balancing loss The composition process can be expressed as: In the formula, λ is the balance coefficient.

Images