Multimodal land cover classification method, device, computer equipment and medium

Abstract

Embodiments of the present invention provide a multimodal land cover classification method, apparatus, computer device, and medium. The method includes the following steps: constructing a first backbone network and a second backbone network, building a multimodal cascade expert forest, and constructing a multimodal feature fusion model using the first backbone network, the second backbone network, and the multimodal cascade expert forest; inputting SAR image data into the first backbone network to output a first feature s; inputting optical image data into the second backbone network to output a second feature o; and combining the first feature s and the second feature o to output a third feature z; setting an optimization target for the multimodal feature fusion model, generating fused features, determining whether the fused features meet the optimization target, and if so, classifying land cover based on the fused features. Because this solution classifies land cover using the multimodal feature fusion model, it improves land cover classification performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a multimodal land cover classification method, device, computer equipment and medium. Background Art

[0002] Land cover classification based on remote sensing imagery involves pixel-level classification according to predefined semantic sets. Therefore, it is also known as remote sensing image semantic segmentation. In recent years, multimodal land cover classification has garnered increasing attention due to the relevant and complementary information contained in heterogeneous remote sensing imagery. Optical imagery can provide detailed descriptions of the spectrum, color, and texture, but is susceptible to complex environmental conditions such as cloud and fog. In contrast, SAR imagery enables active observation throughout the day and in all weather conditions, capturing richer geometric information. Land cover classification using optical and SAR imagery has become a research hotspot in recent years.

[0003] The effectiveness of multimodal fusion has a significant impact on land cover classification results. Existing fusion strategies can be roughly divided into three categories: data fusion, decision fusion, and feature fusion. Data fusion combines input images to produce higher-quality images, relying on handcrafted features or generative models. Decision fusion utilizes certain rules to combine the learning results of different modalities, such as probabilistic ensembles of voting strategies. Feature fusion first extracts features from each modality separately and then performs more flexible processing. With the development of deep learning, feature fusion strategies in deep space have been widely studied, significantly improving the performance of land cover classification.

[0004] In multimodal classification methods based on deep learning, channel concatenation and element summation are relatively simple fusion strategies. However, this cannot fully and effectively integrate multimodal knowledge and inevitably introduces redundant information. Although various fusion strategies have emerged, there are still two problems to be solved. For example, Figure 10 As shown in the above, (1) there is a lack of deep fusion mechanism. Most methods usually design a one- or two-layer fusion module, which is considered as shallow fusion and limited to the domain level. (2) They emphasize discriminability but ignore diversity. Existing methods aim to improve the class separability of multimodal features. However, diversity is equally important because achieving excellent land cover classification requires multi-view feature representation rather than a single compact feature. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a multimodal land cover classification method to solve the technical problems of lack of deep fusion and lack of diversity in the existing technology. The method includes:

[0006] Constructing a first backbone network and a second backbone network, constructing a multimodal cascade expert forest, and constructing a multimodal feature fusion model through the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests;

[0007] Input the SAR image data into the first backbone network, output the first feature s, input the optical image data into the second backbone network, output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and output the third feature z;

[0008] An optimization target is set for the multimodal feature fusion model. After resizing, channel connection, and convolution processing of the third feature z, a fused feature is generated. It is determined whether the fused feature meets the optimization target. If so, the land cover is classified according to the fused feature.

[0009] The present invention also provides a multimodal land cover classification device to address the technical problems of lack of deep fusion and lack of diversity in the existing technology. The device includes:

[0010] A model construction module is used to construct a first backbone network and a second backbone network, and to construct a multimodal cascade expert forest. A multimodal feature fusion model is constructed by using the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests.

[0011] A feature fusion module is configured to input the SAR image data into the first backbone network, output the first feature s, input the optical image data into the second backbone network, output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and output the third feature z;

[0012] The land cover classification module is used to set the optimization target for the multimodal feature fusion model. After resizing, channel connection and convolution processing of the third feature z, the fused feature is generated. It is judged whether the fused feature meets the optimization target. If so, the land cover is classified according to the fused feature.

[0013] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the above-mentioned multimodal land cover classification methods to solve the technical problems of lack of deep fusion and lack of diversity in the existing technology.

[0014] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned multimodal land cover classification methods to solve the technical problems of lack of deep fusion and lack of diversity in the existing technology.

[0015] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0016] The feature fusion method of cascaded multimodal expert forest is used to make the fusion method have deeper and broader fusion capabilities, thereby improving the performance of land cover classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 is a flow chart of a multimodal land cover classification method provided by an embodiment of the present invention;

[0019] Figure 2 is a structural diagram of a multimodal feature fusion model provided by an embodiment of the present invention;

[0020] Figure 3 is a structural diagram of a multimodal expert tree in a multimodal feature fusion model provided by an embodiment of the present invention;

[0021] Figure 4 4 is a structural diagram of a multimodal expert cascade in a multimodal feature fusion model provided by an embodiment of the present invention;

[0022] Figure 5 Schematic diagram of the exchange strategy in the multimodal feature fusion model provided by an embodiment of the present invention;

[0023] Figure 6 This is the classification result diagram on the WHU-OPT-SAR dataset using different classification methods;

[0024] Figure 7 This is the classification result diagram on the Pohang dataset using different classification methods;

[0025] Figure 8 This is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0026] Figure 9This is a structural block diagram of a multimodal land cover classification device provided by an embodiment of the present invention;

[0027] Figure 10 3 is a schematic diagram comparing the structures of the existing feature fusion strategy and the multimodal expert forest of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0030] In an embodiment of the present invention, a multimodal land cover classification method is provided, such as Figure 1 As shown, the method includes:

[0031] Step S101: constructing a first backbone network and a second backbone network, and constructing a multimodal cascade expert forest. A multimodal feature fusion model is constructed by the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests.

[0032] Step S102: Input the SAR image data into the first backbone network to output the first feature s, input the optical image data into the second backbone network to output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest to output the third feature z;

[0033] Step S103: Set an optimization target for the multimodal feature fusion model, resize, connect channels, and perform convolution processing on the third feature z to generate a fused feature, and determine whether the fused feature meets the optimization target. If so, classify the land cover according to the fused feature.

[0034] Specifically, the overall structure of the multimodal feature fusion model is as follows Figure 2As shown in the figure, in the multimodal feature fusion model, each multimodal expert tree is a multimodal feature learner. Expert trees at the same level are combined into an expert forest, and forests at different levels are connected to form a cascade forest. Therefore, the present invention provides an ensemble learning strategy for multimodal feature fusion. Figure 2 The workflow of the multimodal feature fusion model is shown. SAR images and optical images are input into different backbone networks, where the outputs of the four blocks are represented as s i and o i , where i = 1; 2; 3; 4 and indexes different network blocks. Multimodal expert forests are connected in parallel between different branches. Each multimodal expert forest receives the output of two unimodal blocks and generates a multimodal feature z i The segmentation head uses features at different levels z i For predictive reasoning, the Segformer-B2 model is used as the backbone network, where each network block contains multiple transformer layers.

[0035] Specifically, SAR imaging, or Synthetic Aperture Radar (SAR), is a technology that uses the principle of synthetic aperture to achieve high-resolution microwave imaging. Its all-day, all-weather capabilities, high resolution, and wide bandwidth make it widely used in Earth observation, environmental monitoring, military reconnaissance, and other fields.

[0036] In specific implementation, the following steps are used to build a multimodal cascade expert forest:

[0037] Construct the first linear projection matrix and the second linear projection matrix experts, among whom, d r is the rank, d is the output dimension, rank d r is smaller than the input dimension d, and R is the feature dimension space; constructing a multimodal expert tree comprising multiple experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer; connecting multiple multimodal expert trees to form the multimodal expert forest, and cascading multiple multimodal expert forests to form the multimodal cascade expert forest.

[0038] In specific implementation, in order to achieve a wider range of fusion, the expert trees at the same layer and different layers are combined into a cascade forest. The following steps are implemented: SAR image data is input into the first backbone network to output the first feature s; optical image data is input into the second backbone network to output the second feature o; the first feature s and the second feature o are input into each multimodal expert forest of the multimodal cascade expert forest to output the third feature z:

[0039] Input the first feature s and the second feature o into the multimodal expert tree, and output the learned multimodal features corresponding to the multimodal expert tree;

[0040] Get the jth multimodal expert tree T in the i-th multimodal expert forest i,jj The learned multimodal features z i,j , by learning the multimodal features z i,j The channel connection of the multimodal expert trees at the same level is combined to generate a multimodal expert forest. The feature z output by the multimodal expert forest is i =concate(z i,j ), where j = 1, 2, 3, z i is the feature of the i-th multimodal expert forest; construct a cross-level cascade of multimodal expert forests to generate a multimodal cascade expert forest, then the feature z output by the multimodal cascade expert forest is i,j =T i,j (cov(z i-1,j )+concate(s i ,o i )),i=2,3,4, where s i is the first feature of the ith network block of the first backbone network, o i is the second feature of the i-th network block of the second backbone network; a multimodal expert tree T in the multimodal expert forest is converted into i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest.

[0041] In specific implementation, the following steps are performed to input the first feature s and the second feature o into the multimodal expert tree and output the learned multimodal features corresponding to the multimodal expert tree:

[0042] Concatenate the first feature s and the second feature o along the channel dimension to generate the input token sequence Transform the input token sequence t into a super token sequence Among them, N t is the number of tokens, d is the feature dimension; the super token sequence P is input into the multimodal expert tree to generate the feature output P of each layer of the multimodal expert tree l+1,i ,in, i is the index of the super token, is the number of super tokens that each expert needs to process at level l, is the index of the expert network; obtain the feature output of the expert in the last layer Feature output for the last layer of experts Perform token recovery to generate learned multimodal features.

[0043] Specifically, as the basic unit of feature fusion, the multimodal expert tree is as follows Figure 3 As shown. It contains multiple expert networks stacked at different layers, and the number of experts (denoted as N h ) is doubled layer by layer to form a tree structure. Therefore, it can perform deeper fusion to produce more discriminative multimodal features. l,h represents the hth expert in the lth layer, where l∈[1,...,N l ], Each expert contains two and The linear projection matrix of rank d r is smaller than the input dimension d, and R is the feature dimension space. Different experts learn different features, and then feature integration is achieved through dynamic routing.

[0044] Taking the expert tree corresponding to the first network as an example, concatenate s1 and o1 along the channel dimension to obtain the input token sequence Symbol N t is the number of tokens, and d is the feature dimension. Then, it is transformed into a super token sequence

[0045]

[0046] in, is a learnable parameter. The matrix w g It is actually the result of applying the softmax function to the columns of tφ, which takes the length N t The t is fused to a length of N p p. N p The value is greater than N t This not only adaptively combines the features of different tokens and enriches the representation capability, but also shortens the length of the sequence that needs to be processed and optimizes the computational efficiency.

[0047] Next, different expert networks perform feature learning layer by layer on the corresponding parts of the super token sequence:

[0048]

[0049] where i indexes the super token, represents the number of super tokens that each expert needs to process at level l, Index Expert Network. The length of the super token sequence remains unchanged. Therefore, at any layer l, there is an identity relation The feature transfer between experts at different levels produces a deeper fusion capability.

[0050] Finally, the feature output of the last layer of experts (denoted as ) to recover the token:

[0051]

[0052] in, is the result of applying the softmax function to the rows of tφ, is the multimodal feature learned by the expert tree. In formulas (2) and (5), the transformation matrix w g and w r are all generated from the input features t. This means that the learning and routing process of multimodal features is unique for each input image, that is, at the instance level.

[0053] Specifically, in order to achieve a wider range of fusion, the expert trees at the same layer and different layers are combined into a cascade forest. Figure 4 In, T i,j represents the jth tree in the i-th forest, z i,j Represents the multimodal features of its output. For the four network blocks, i ranges from [1, 2, 3, 4]. Based on hyperparameter experiments, j ranges from [1, 2, 3, 4]. Each tree is assigned a different rank, and each tree is at a different level and position, receiving different inputs. These factors together lead to the diversity of multimodal features.

[0054] First, by i,j The expert trees at the same level are combined into an expert forest:

[0055] z i =concate(z i,j ), j=1,2,3, (6),

[0056] Among them, zi represents the output of the i-th expert forest. Then, a cross-level cascade between forests is established:

[0057] z i,j =T i,j (conv(z i-1,j )+concate(s i ,o i )),i=2,3,4, (7),

[0058] Among them, the dilated convolution is responsible for reducing the z i-1,j The dimension is used for subsequent addition operations. For expert trees in layers 2 to 4, their input consists of two aspects: the output of the previous layer and the unimodal features of that layer. Formulas 6 and 7 combine the expert trees into a cascade forest, enabling ensemble learning of diverse multimodal features.

[0059] In the specific implementation, in order to realize the knowledge transfer between different expert trees and further optimize the effect of feature integration, the following steps are implemented to realize the expert exchange strategy to transfer a multimodal expert tree T in the multimodal expert forest. i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest:

[0060] Multimodal expert tree T i,j The features of the l+1 layer output Among them, T i,j′ and T i,j″ are other multimodal expert trees in the same multimodal expert forest, q is a random number generated with an average distribution between 0 and 1, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″.

[0061] In the specific implementation, in order to realize the knowledge transfer between different expert trees and further optimize the effect of feature integration, the following steps are implemented to realize the expert exchange strategy to transfer a multimodal expert tree T in the multimodal expert forest. i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest:

[0062] Multimodal expert tree T i,j The features of the l+1 layer output Among them, T i,j′ and T i,j″ are other multimodal expert trees in the same multimodal expert forest, θ is a learnable weight matrix, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″, and E is the expert network in the multimodal expert tree.

[0063] Specifically, Formula 6 constructs the expert forest only by connecting features, and the communication between different expert trees remains closed. To this end, Figure 5 As shown in the figure, two expert exchange strategies are proposed to achieve knowledge transfer between different expert trees and further optimize the effect of feature integration.

[0064] The completely random exchange strategy is to change the tree T i,j The expert at the lth level is replaced with another tree in the same forest with a certain probability (T i，j′ or T i，j″ )Tier 1 Experts:

[0065]

[0066] Where q is a random number generated from an average distribution between 0 and 1, and l can be set to any level in the tree. The value of j ranges from [1, 2, 3] and j ≠ j′ ≠ j", ensuring that expert exchange can occur in every tree.

[0067] The learnable soft switching strategy first transforms the tree T i,j The super tokens at layer l are assigned to the corresponding experts in other trees, and then the outputs of experts in different trees are weighted combined:

[0068]

[0069] Among them, θ is a learnable weight matrix. Compared with the first strategy, it can realize the dynamic integration of knowledge learned from different trees.

[0070] In specific implementation, the following steps are performed to set the optimization target for the multimodal feature fusion model:

[0071] Optimization goal Among them, y is the true label, y c is the probability distribution of the true label, To predict the label, is the probability distribution of the predicted label, N c is the number of land cover types, and c is the index of the land cover type.

[0072] Specifically, the classification method (CM 2 FE) follows the general supervised multimodal learning paradigm and is optimized using cross-entropy loss:

[0073]

[0074] Among them, y and represent the true label and the predicted label respectively, N c is the number of land cover types.

[0075] The effect of land cover classification using the embodiment of the present invention is verified below.

[0076] The experiment uses two multimodal remote sensing datasets, WHU-OPT-SAR and Pohang. WHU-OPT-SAR includes four-band multispectral images (RGB and near-infrared) and one-band SAR images with a spatial resolution of 5 meters and 7 land cover types. It was cropped into 7,000 images with a pixel size of 512×512, of which 5,600 were used for training and 1,400 were used for testing. Pohang includes three-band optical images and one-band SAR images with a spatial resolution of 1 meter and 6 types of surface cover. It was cropped into 6,145 images with a size of 256×256, of which 4,916 were used for training and 1,229 were used for testing. Among them, the sixth land cover type was discarded in the analysis because its percentage was too small.

[0077] The experiment uses the AdamW optimizer, where β1 and β2 are set to 0..9 and 0.999 respectively, and the decay coefficient is 0.01. The learning rate is 6e-5, the number of iterations is 80000, and the batch size is 8. Each expert tree contains four layers, N h Set to (4, 8, 16, 32) in sequence, N s Set to (32,,16,8,4) in sequence. N p The value of is always 128. For WHU-OPT-SAR, the initial ranks of the four forests are set to (16, 32, 80, 128) from shallow to deep. For Pohang, this setting is changed to (8, 16, 40, 64). Each forest contains three trees, with the rank of each tree being doubled. The learnable parameters φ and θ are implemented using two fully connected layers. The second layer of each tree is used for expert exchange. When an expert in one tree is randomly exchanged, the other two trees each have a 15% probability of learning features.

[0078] As shown in Table 1, the classification results of different methods on WHU-OPT-SAR.

[0079]

[0080] Table 1

[0081] As shown in Table 2, the classification results of different methods on Pohang.

[0082]

[0083] Table 2

[0084] Tables 1 and 2 compare the classification results of different methods, where CM 2 FE-R and CM 2FE-L represents the classification method of the embodiment of the present invention using a completely random exchange and a learnable soft exchange strategy. It can be seen that the method proposed in the embodiment of the present invention can achieve the best performance. On WHU-OPTSAR, compared with the second place, CM 2 The mIoU of FE-L increased by 3.25% and mF1 increased by 2.89%. On Pohang, the improvements were 1.60% and 1.00%, respectively. For each category, the performance of the proposed method is still better than other methods. This shows that CM 2 FE can learn more robust and discriminative multimodal features through deeper and wider fusion. 2 The results of FE-L are slightly better than those of CM 2 FE-R, indicating that learnable soft swapping has a better promoting effect on feature fusion than random swapping. Figure 6 and Figure 7 The classification results of different land cover classification methods are visualized. 2 The classification graph of FE-L is more accurate and corrects the noise and misclassification phenomena. The effectiveness of the proposed method is verified again from the perspective of visualization.

[0085] In this embodiment, a computer device is provided, such as Figure 8 As shown, it includes a memory 801, a processor 802 and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned multimodal land cover classification methods is implemented.

[0086] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0087] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned multimodal land cover classification methods.

[0088] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0089] Based on the same inventive concept, an embodiment of the present invention further provides a multimodal land cover classification device, as described in the following embodiments. Since the principle of solving the problem by the multimodal land cover classification device is similar to that of the multimodal land cover classification method, the implementation of the multimodal land cover classification device can refer to the implementation of the multimodal land cover classification method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceived.

[0090] Figure 9 This is a structural block diagram of a multimodal land cover classification device according to an embodiment of the present invention. Figure 9 As shown, it includes: a model building module 901, a feature fusion module 902 and a land cover classification module 903. The structure is described below.

[0091] A model construction module 901 is configured to construct a first backbone network and a second backbone network, and to construct a multimodal cascade expert forest. A multimodal feature fusion model is constructed using the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests.

[0092] Feature fusion module 902, configured to input SAR image data into a first backbone network, output a first feature s, input optical image data into a second backbone network, output a second feature o, input the first feature s and the second feature o into each multimodal expert forest of a multimodal cascade expert forest, and output a third feature z;

[0093] The land cover classification module 903 is used to set the optimization target for the multimodal feature fusion model, resize the third feature z, connect the channels and perform convolution processing to generate a fused feature, and determine whether the fused feature meets the optimization target. If so, the land cover is classified according to the fused feature.

[0094] In one embodiment, the model building module includes:

[0095] Expert construction unit for constructing a matrix containing the first linear projection and the second linear projection matrix experts, among whom, d r is the rank, d is the output dimension, rank d r Less than the input dimension d, R is the feature dimension space;

[0096] An expert tree construction unit, configured to construct a multimodal expert tree comprising a plurality of experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer;

[0097] The expert forest construction unit is used to connect multiple multimodal expert trees to form a multimodal expert forest, and to cascade multiple multimodal expert forests to form a multimodal cascade expert forest.

[0098] In one embodiment, the feature fusion module includes:

[0099] An expert tree feature acquisition unit, configured to input the first feature s and the second feature o into the multimodal expert tree and output the learned multimodal features corresponding to the multimodal expert tree;

[0100] Expert forest feature acquisition unit, used to obtain the j-th multimodal expert tree T in the i-th multimodal expert forest i,j The learned multimodal features z i,j , by learning the multimodal features z i,j The channel connection of the multimodal expert trees at the same level is combined to generate a multimodal expert forest. The feature z output by the multimodal expert forest is i =concate(z i,j ), where j = 1, 2, 3, z i is the feature of the i-th multimodal expert forest;

[0101] The feature cascade unit is used to construct a cross-level cascade of the multimodal expert forest to generate a multimodal cascade expert forest. The feature z output by the multimodal cascade expert forest is i,j =T i,j (cov(z i-1,j )+concate(s i ,o i )),i=2,3,4, where s i is the first feature of the ith network block of the first backbone network, o i is the second characteristic of the i-th network block of the second backbone network;

[0102] The feature exchange unit is used to exchange a multimodal expert tree T in the multimodal expert forest through the expert exchange strategy. i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest.

[0103] In one embodiment, the expert tree feature acquisition unit is used to concatenate the first feature s and the second feature o along the channel dimension to generate an input token sequence Transform the input token sequence t into a super token sequence Among them, N t is the number of tokens, d is the feature dimension; the super token sequence P is input into the multimodal expert tree to generate the feature output P of each layer of the multimodal expert tree l+1,i ,in, i is the index of the super token, is the number of super tokens that each expert needs to process at level l, is the index of the expert network; obtain the feature output of the expert in the last layer Feature output for the last layer of experts Perform token recovery to generate learned multimodal features.

[0104] In one embodiment, the feature exchange unit is used for the multimodal expert tree T i,,j The features of the l+1 layer output Among them, T i,j′ and T ii,j″ are other multimodal expert trees in the same multimodal expert forest, q is a random number generated with an average distribution between 0 and 1, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″.

[0105] In one embodiment, the feature exchange unit is also used for the multimodal expert tree T i,j The features of the l+1 layer output Among them, T i,j′ and Ti,j′′ are other multimodal expert trees in the same multimodal expert forest, θ is a learnable weight matrix, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j ≠ j′ ≠ j", and E is the expert network in the multimodal expert tree.

[0106] In one embodiment, the land cover classification module includes:

[0107] Optimization target setting unit, used to optimize the target Among them, y is the true label, y c is the probability distribution of the true label, To predict the label, is the probability distribution of the predicted label, N c is the number of land cover types, and c is the index of the land cover type.

[0108] The embodiments of the present invention achieve the following technical effects:

[0109] In order to further improve the performance of land cover classification, a new feature fusion method using cascaded multimodal expert forests is proposed. The cascaded multimodal expert forest contains three key components with a progressive relationship (multimodal expert tree, multimodal expert forest and multimodal cascade expert forest), which gives it deeper and broader fusion capabilities; each modal expert tree is constructed in the form of multi-layer experts to extract more discriminative multimodal features, and different expert trees are combined into a cascade forest to fully integrate the different knowledge learned; the land cover classification method uses two expert exchange strategies to further optimize the effect of feature fusion and integration, and improve the generalization performance of multimodal land cover classification; by comparing the classification maps of different classification methods, the classification map generated by the land cover classification method of the embodiment of the present invention is more accurate, and the noise and misclassification phenomena are corrected, which once again verifies the effectiveness of the land cover classification method of this embodiment from a visualization perspective.

[0110] Obviously, those skilled in the art should understand that the various modules or steps of the above-mentioned embodiments of the present invention can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices. Alternatively, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into separate integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0111] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multimodal land cover classification method, characterized in that: include: Constructing a first backbone network and a second backbone network, constructing a multimodal cascade expert forest, and constructing a multimodal feature fusion model through the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests; Construct a multimodal cascade expert forest, including: Construct the first linear projection matrix and the second linear projection matrix experts, among whom, d r For order, d is the input dimension, rank d r Smaller than the input dimension d , R is a feature dimensional space; constructing a multimodal expert tree comprising a plurality of experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer; connecting a plurality of the multimodal expert trees to form the multimodal expert forest, and cascading a plurality of the multimodal expert forests to form the multimodal cascade expert forest; In the multimodal feature fusion model, each of the multimodal expert trees is a learner of multimodal features, the multimodal expert trees at the same level are combined into a multimodal expert forest, and the multimodal expert forests at different levels are connected to obtain a multimodal cascade expert forest; Input the SAR image data into the first backbone network and output the first feature s , input the optical image data into the second backbone network and output the second feature o , the first feature s And the second feature o Input to each of the multimodal expert forests of the multimodal cascade expert forest, and output the third feature z ,include: The first feature s And the second feature o Input to the multimodal expert tree, and output the learned multimodal features corresponding to the multimodal expert tree; obtain the first i The first j Multimodal expert tree The learned multimodal features , by learning the multimodal features The multimodal expert trees at the same level are combined to generate the multimodal expert forest. The features output by the multimodal expert forest are ,in, , For the i The features of the multimodal expert forest are constructed; a cross-level cascade of the multimodal expert forest is constructed to generate the multimodal cascade expert forest, and the features output by the multimodal cascade expert forest are ,in, is the first feature of the i-th network block of the first backbone network, is the second characteristic of the i-th network block of the second backbone network; One of the multimodal expert trees in the multimodal expert forest is replaced by an expert exchange strategy. Any layer l The output feature is replaced by any layer of another multimodal expert tree in the multimodal expert forest l Output characteristics; The SAR image data and the optical image data are input into different backbone networks, where the outputs of the four blocks are represented as s i and o i , where i = 1; 2; 3; 4 and indexes different network blocks, the multimodal expert forests are connected in parallel between different branches, each of the multimodal expert forests receives the output of two single-modal blocks and generates multimodal features z i , the segmentation head uses features at different levels z i Perform predictive reasoning; Set an optimization target for the multimodal feature fusion model, and z After resizing, channel connection, and convolution processing, fused features are generated, and it is determined whether the fused features meet the optimization goal. If the optimization goal is met, the land cover is classified according to the fused features.

2. The multimodal land cover classification method according to claim 1, wherein: The first feature s And the second feature o Inputting the multimodal expert tree and outputting the learned multimodal features corresponding to the multimodal expert tree includes: The first feature is transformed along the channel dimension s And the second feature o Concatenate to generate a sequence of input tokens , the input token sequence t Transformed into a super token sequence ,in, N t is the number of tokens, d is the characteristic dimension; Input the super token sequence P into the multimodal expert tree to generate the feature output of each layer of the multimodal expert tree ,in, , is the index of the super token, For each expert The number of super tokens that the layer needs to process, is the index of the expert network, E is the expert network in the multimodal expert tree; Get the feature output of the expert in the last layer , the feature output of the expert in the last layer Perform token recovery to generate learned multimodal features.

3. The multimodal land cover classification method according to claim 1, wherein: One of the multimodal expert trees in the multimodal expert forest is replaced by an expert exchange strategy. Any layer l The output feature is replaced by any layer of another multimodal expert tree in the multimodal expert forest l Output features include: The multimodal expert tree in l +1 layer output features in, and are other multimodal expert trees in the same multimodal expert forest, q is a random number generated between 0 and 1 with an average distribution. l is any layer in the multimodal expert tree, j The value range of is [1,2,3] and , E is the expert network in the multimodal expert tree.

4. The multimodal land cover classification method according to claim 1, wherein: One of the multimodal expert trees in the multimodal expert forest is replaced by an expert exchange strategy. Any layer l The output feature is replaced by any layer of another multimodal expert tree in the multimodal expert forest l Output features include: The multimodal expert tree T i,j in l +1 layer output features ,in, and are other multimodal expert trees in the same multimodal expert forest, is the learnable weight matrix, l is any layer in the multimodal expert tree, j The value range of is [1,2,3] and , E is the expert network in the multimodal expert tree.

5. The multimodal land cover classification method according to any one of claims 1 to 4, characterized in that: Setting an optimization objective for the multimodal feature fusion model includes: Optimization goal ,in, is the true label, is the probability distribution of the true label, To predict the label, is the probability distribution of the predicted label, is the number of land cover types, c The index of land cover type.

6. A multimodal land cover classification device, characterized in that: include: A model construction module is used to construct a first backbone network and a second backbone network, and to construct a multimodal cascade expert forest, and to construct a multimodal feature fusion model through the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests; Model building modules, including: Expert construction unit for constructing a matrix containing the first linear projection and the second linear projection matrix experts, among whom, d r For order, d is the input dimension, rank d r Smaller than the input dimension d , R is the feature dimension space; An expert tree construction unit, configured to construct a multimodal expert tree comprising a plurality of experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer; An expert forest construction unit, configured to connect multiple multimodal expert trees to form a multimodal expert forest, and to cascade multiple multimodal expert forests to form a multimodal cascade expert forest; A multimodal cascade expert forest construction unit is used to, in the multimodal feature fusion model, combine the multimodal expert trees at the same level into a multimodal expert forest, and connect the multimodal expert forests at different levels to obtain a multimodal cascade expert forest; Feature fusion module, used to input SAR image data into the first backbone network and output the first feature s , input the optical image data into the second backbone network and output the second feature o , the first feature s And the second feature o Input to each of the multimodal expert forests of the multimodal cascade expert forest, and output the third feature z ; Feature fusion module, including: Expert tree feature acquisition unit, used to convert the first feature s and the second feature o Input into the multimodal expert tree and output the learned multimodal features corresponding to the multimodal expert tree; Expert forest feature acquisition unit, used to obtain the i The first j Multimodal expert tree The learned multimodal features , by learning multimodal features The channel connection of the multimodal expert trees at the same level is combined to generate a multimodal expert forest. The features output by the multimodal expert forest are ,in, , For the i Characteristics of a multimodal expert forest; Feature cascade unit is used to construct a cross-level cascade of multimodal expert forests to generate a multimodal cascade expert forest. The feature output of the multimodal cascade expert forest is ,in, is the first feature of the i-th network block of the first backbone network, is the second characteristic of the i-th network block of the second backbone network; Feature exchange unit, used to exchange a multimodal expert tree in the multimodal expert forest through the expert exchange strategy Any layer l The output feature is replaced by any layer of another multimodal expert tree in the multimodal expert forest l Output characteristics; The prediction and inference unit is used to input the SAR image data and the optical image data into different backbone networks, wherein the outputs of the four blocks are respectively represented as s i and o i , where i = 1; 2; 3; 4 and indexes different network blocks, the multimodal expert forests are connected in parallel between different branches, each of the multimodal expert forests receives the output of two single-modal blocks and generates multimodal features z i , the segmentation head uses features at different levels z i Perform predictive reasoning; The land cover classification module is used to set the optimization target for the multimodal feature fusion model and to z After resizing, channel connection, and convolution processing, fused features are generated, and it is determined whether the fused features meet the optimization goal. If the optimization goal is met, the land cover is classified according to the fused features.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal land cover classification method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program for executing the multimodal land cover classification method according to any one of claims 1 to 5.

Images