Multi-modal land coverage classification method and device, computer equipment and medium

Abstract

The embodiment of the invention provides a multi-modal land coverage classification method and device, computer equipment and a medium, and the method comprises the following steps: constructing a first backbone network and a second backbone network, constructing a multi-modal cascade expert forest, and constructing a multi-modal cascade expert forest; constructing a multi-modal feature fusion model through the first backbone network, the second backbone network and the multi-modal cascade expert forest; inputting the SAR image data into the first backbone network, outputting a first feature s, inputting the optical image data into the second backbone network, outputting a second feature o, and outputting the first feature s and the second feature o into a third feature z; and setting an optimization target for the multi-modal feature fusion model, generating fused features, judging whether the fused features meet the optimization target or not, and if the fused features meet the optimization target, classifying land coverage according to the fused features. According to the scheme, land coverage is classified through the multi-modal feature fusion model, so that the performance of land coverage classification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a multimodal land cover classification method, device, computer equipment and medium. Background Art

[0002] Land cover classification based on remote sensing images refers to pixel-level classification according to a predefined semantic set. Therefore, it is also called remote sensing image semantic segmentation. In recent years, multimodal land cover classification has received increasing attention due to the relevant and complementary information of heterogeneous remote sensing images. Optical images can provide detailed descriptions of spectrum, color, and texture, but are easily affected by complex environments such as clouds and fog. In contrast, SAR images can achieve active observations throughout the day and all weather, and capture richer geometric information. In recent years, land cover classification using optical and SAR images has become a research hotspot.

[0003] The effect of multimodal fusion has an important impact on the results of land cover classification. Existing fusion strategies can be roughly divided into three categories: data fusion, decision fusion and feature fusion. Data fusion combines input images to produce higher quality images, which relies on manual features or generative models. Decision fusion combines the learning results of different modalities using certain rules, such as probabilistic integration of voting strategies. Feature fusion first extracts the features of each modality separately, and then performs more flexible processing. With the development of deep learning, feature fusion strategies in deep space have been widely studied, which has significantly improved the performance of land cover classification.

[0004] In multimodal classification methods based on deep learning, channel concatenation and element summation are relatively simple fusion strategies. However, this cannot fully and effectively integrate multimodal knowledge and inevitably introduces redundant information. Although various fusion strategies have emerged, there are still two problems to be solved. For example, Fig.10 As shown in the figure, (1) there is a lack of deep fusion mechanism. Most methods usually design one or two layers of fusion modules, which is considered as shallow fusion and limited to the domain level. (2) They emphasize discriminability but ignore diversity. Existing methods aim to improve the class separability of multimodal features. However, diversity is equally important because achieving excellent land cover classification requires multi-view feature representation rather than a single compact feature. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a multimodal land cover classification method to solve the technical problems of lack of deep fusion and lack of diversity in the prior art. The method includes:

[0006] Constructing a first backbone network and a second backbone network, constructing a multimodal cascade expert forest, and constructing a multimodal feature fusion model through the first backbone network, the second backbone network and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests;

[0007] Input the SAR image data into the first backbone network, output the first feature s, input the optical image data into the second backbone network, output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and output the third feature z;

[0008] An optimization target is set for the multimodal feature fusion model. After resizing, channel connection and convolution processing of the third feature z, a fused feature is generated. It is determined whether the fused feature meets the optimization target. If so, the land cover is classified according to the fused feature.

[0009] The embodiment of the present invention also provides a multi-modal land cover classification device to solve the technical problems of lack of deep fusion and lack of diversity in the prior art. The device includes:

[0010] A model construction module is used to construct a first backbone network and a second backbone network, and to construct a multimodal cascade expert forest. A multimodal feature fusion model is constructed through the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests;

[0011] A feature fusion module is used to input the SAR image data into the first backbone network, output the first feature s, input the optical image data into the second backbone network, output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and output the third feature z;

[0012] The land cover classification module is used to set the optimization target for the multimodal feature fusion model, generate the fused features after resizing, channel connection and convolution processing of the third feature z, and judge whether the fused features meet the optimization target. If so, the land cover is classified according to the fused features.

[0013] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned multimodal land cover classification methods when executing the computer program, so as to solve the technical problems of lack of deep fusion and lack of diversity in the prior art.

[0014] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned multimodal land cover classification methods to solve the technical problems of lack of deep fusion and lack of diversity in the prior art.

[0015] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0016] The feature fusion method of cascaded multimodal expert forests was used to make the fusion method have deeper and broader fusion capabilities, thereby improving the performance of land cover classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a flow chart of a multimodal land cover classification method provided by an embodiment of the present invention;

[0019] Figure 2 is a structural diagram of a multimodal feature fusion model provided by an embodiment of the present invention;

[0020] Figure 3 is a structural diagram of a multimodal expert tree in a multimodal feature fusion model provided by an embodiment of the present invention;

[0021] Figure 4 is a structural diagram of a multimodal expert cascade in a multimodal feature fusion model provided by an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of an exchange strategy in a multimodal feature fusion model provided by an embodiment of the present invention;

[0023] Figure 6 This is the classification result diagram on the WHU-OPT-SAR dataset using different classification methods;

[0024] Figure 7 This is the classification result diagram on the Pohang dataset using different classification methods;

[0025] Figure 8 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0026] Fig. 9is a structural block diagram of a multi-modal land cover classification device provided by an embodiment of the present invention;

[0027] Fig.10 It is a schematic diagram of the structural comparison between the existing feature fusion strategy and the multimodal expert forest of the embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0030] In an embodiment of the present invention, a multimodal land cover classification method is provided, such as Figure 1 As shown, the method includes:

[0031] Step S101: constructing a first backbone network and a second backbone network, constructing a multimodal cascade expert forest, and constructing a multimodal feature fusion model through the first backbone network, the second backbone network and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests;

[0032] Step S102: inputting the SAR image data into the first backbone network, outputting the first feature s, inputting the optical image data into the second backbone network, outputting the second feature o, inputting the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and outputting the third feature z;

[0033] Step S103: Set an optimization target for the multimodal feature fusion model, resize the third feature z, connect the channels and perform convolution processing to generate a fused feature, and determine whether the fused feature meets the optimization target. If so, classify the land cover according to the fused feature.

[0034] Specifically, the overall structure of the multimodal feature fusion model is as follows Figure 2As shown. In the multimodal feature fusion model, each multimodal expert tree is a learner of multimodal features, expert trees at the same level are combined into an expert forest, and forests at different levels are connected to obtain a cascade forest. Therefore, the present invention is an integrated learning strategy for multimodal feature fusion. Figure 2 The workflow of the multimodal feature fusion model is shown. The SAR image and the optical image are input into different backbone networks, where the outputs of the four blocks are represented as s i and i , where i = 1; 2; 3; 4 and indexes different network blocks. Multimodal expert forests are connected in parallel between different branches. Each multimodal expert forest receives the output of two unimodal blocks and generates a multimodal feature z i The segmentation head uses features at different levels z i For predictive reasoning, the Segformer-B2 model is used as the backbone network, where each network block contains multiple transformer layers.

[0035] Specifically, SAR imaging, or Synthetic Aperture Radar imaging, is a technology that uses the principle of synthetic aperture to achieve high-resolution microwave imaging. It has the characteristics of all-day, all-weather, high resolution, and wide bandwidth, and is widely used in earth observation, environmental monitoring, military reconnaissance and other fields.

[0036] In specific implementation, the following steps are used to build a multimodal cascade expert forest:

[0037] Construct the first linear projection matrix and the second linear projection matrix experts, among whom, r is the rank, d is the output dimension, rank d r is smaller than the input dimension d, where R is the feature dimension space; constructing a multimodal expert tree comprising multiple experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer; connecting multiple multimodal expert trees to form the multimodal expert forest, and cascading multiple multimodal expert forests to form the multimodal cascade expert forest.

[0038] In the specific implementation, in order to achieve a wider range of fusion, the expert trees at the same layer and different layers are combined into a cascade forest. The following steps are implemented to input the SAR image data into the first backbone network, output the first feature s, input the optical image data into the second backbone network, output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and output the third feature z:

[0039] Input the first feature s and the second feature o into the multimodal expert tree, and output the learned multimodal features corresponding to the multimodal expert tree;

[0040] Get the jth multimodal expert tree T in the i-th multimodal expert forest i,jj The learned multimodal features z i,j , by learning the multimodal features z i,j The multimodal expert trees at the same level are combined to generate a multimodal expert forest. The feature z output by the multimodal expert forest is i =concate(z i,j ), where j = 1, 2, 3, z i is the feature of the i-th multimodal expert forest; construct a cross-level cascade of multimodal expert forests to generate a multimodal cascade expert forest, then the feature z output by the multimodal cascade expert forest is i,j =T i,j (cov(z i-1,j )+concate(s i ,o i )),i=2,3,4, where s i is the first feature of the ith network block of the first backbone network, o i is the second feature of the i-th network block of the second backbone network; a multimodal expert tree T in the multimodal expert forest is replaced by an expert exchange strategy i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest.

[0041] In the specific implementation, the first feature s and the second feature o are input into the multimodal expert tree through the following steps, and the learned multimodal features corresponding to the multimodal expert tree are output:

[0042] Concatenate the first feature s and the second feature o along the channel dimension to generate the input token sequence Transform the input token sequence t into a super token sequence Among them, N t is the number of tokens, d is the feature dimension; the super token sequence P is input into the multimodal expert tree to generate the feature output P of each layer of the multimodal expert tree l+1,i ,in, i is the index of the super token, is the number of super tokens that each expert needs to process at level l, is the index of the expert network; obtain the feature output of the last layer of experts Feature output for the last layer of experts Perform token recovery to generate learned multimodal features.

[0043] Specifically, as the basic unit of feature fusion, the multimodal expert tree is Figure 3 As shown. It contains multiple expert networks stacked at different layers, and the number of experts (denoted as N h ) is doubled layer by layer to form a tree structure. Therefore, it is able to perform deeper fusion to produce more discriminative multimodal features. l,h represents the hth expert in the lth layer, where l∈[1,...,N l ], Each expert contains two and The linear projection matrix of rank d r is smaller than the input dimension d, and R is the feature dimension space. Different experts learn different features, and then feature integration is achieved through dynamic routing.

[0044] Taking the expert tree corresponding to the first network as an example, s1 and o1 are concatenated along the channel dimension to obtain the input token sequence Symbol N t is the number of tokens, and d is the feature dimension. Then, it is transformed into a super token sequence

[0045]

[0046] in, is a learnable parameter. The matrix w g It is actually the result of applying the softmax function to the columns of tφ, which takes a length of N t The length of t is N. p p. N p The value is greater than N t This not only adaptively combines the features of different tokens and enriches the representation capability, but also shortens the length of the sequence that needs to be processed and optimizes the computational efficiency.

[0047] Next, different expert networks perform feature learning layer by layer on the corresponding parts of the super token sequence:

[0048]

[0049] where i indexes the super token, represents the number of super tokens that each expert needs to process at level l, Index Expert Network. The length of the super token sequence remains constant. Therefore, at any layer l, there exists an identity relation The feature transfer between experts at different levels produces a deeper fusion capability.

[0050] Finally, using the feature output of the last layer of experts (denoted as ) to restore the token:

[0051]

[0052] in, is the result of applying the softmax function to the rows of tφ, is the multimodal feature learned by the expert tree. In formulas (2) and (5), the transformation matrix w g and w r are all generated from the input features t. This means that the learning and routing process of multimodal features is unique for each input image, that is, at the instance level.

[0053] Specifically, in order to achieve more extensive fusion, the expert trees at the same layer and different layers are combined into a cascade forest. Figure 4 In, T i,j represents the jth tree in the i-th forest, z i,j Represents the multimodal features of its output. Corresponding to the four network blocks, the value range of i is [1,2,3,4]. Based on the hyperparameter experiment, the range of j is set to [1,2,3,4]. Different ranks are set for each tree, and each tree is at a different level and position, receiving different inputs. These factors together lead to the diversity of multimodal features.

[0054] First, by i,j The expert trees at the same level are combined into an expert forest by connecting the channels:

[0055] z i =concate(z i,j ), j = 1, 2, 3, (6),

[0056] Among them, zi represents the output of the i-th expert forest. Then, a cross-level cascade between forests is established:

[0057] z i,j =T i,j (conv(z i-1,j )+concate(s i ,o i )),i=2,3,4, (7),

[0058] Among them, the dilated convolution is responsible for reducing z i-1,j The dimension is used for subsequent addition operations. For the expert trees of the 2nd to 4th layers, their input includes two aspects: the output of the previous layer and the unimodal features of this layer. Formulas 6 and 7 combine the expert trees into a cascade forest, realizing the integrated learning of diverse multimodal features.

[0059] In the specific implementation, in order to realize the knowledge transfer between different expert trees and further optimize the effect of feature integration, the following steps are implemented to transfer a multimodal expert tree T in the multimodal expert forest through the expert exchange strategy: i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest:

[0060] Multimodal Expert Tree T i,j The features of the l+1 layer output Among them, T i,j′ and T i,j″ are other multimodal expert trees in the same multimodal expert forest, q is a random number generated from an average distribution between 0 and 1, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″.

[0061] In the specific implementation, in order to realize the knowledge transfer between different expert trees and further optimize the effect of feature integration, the following steps are implemented to transfer a multimodal expert tree T in the multimodal expert forest through the expert exchange strategy: i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest:

[0062] Multimodal Expert Tree T i,j The features of the l+1 layer output Among them, T i,j′ and T i,j″ are other multimodal expert trees in the same multimodal expert forest, θ is a learnable weight matrix, l is any layer in the multimodal expert tree, the value range of j is [1,2,3] and j≠j′≠j″, and E is the expert network in the multimodal expert tree.

[0063] Specifically, Formula 6 constructs the expert forest only by connecting features, and the communication between different expert trees remains closed. Figure 5 As shown, two expert exchange strategies are proposed to achieve knowledge transfer between different expert trees and further optimize the effect of feature integration.

[0064] The completely random exchange strategy is to change the tree T i,j The expert at the lth level is replaced with other trees in the same forest with a certain probability (T i，j′ or T i，j″ )Tier 1 Experts:

[0065]

[0066] Where q is a random number generated from an average distribution between 0 and 1, and l can be set to any layer in the tree. The value range of j is [1, 2, 3] and j ≠ j′ ≠ j", ensuring that expert exchange can occur in each tree.

[0067] The learnable soft switching strategy first transforms the tree T i,j The super tokens at the lth layer are assigned to the corresponding experts in other trees, and then the outputs of experts in different trees are weighted combined:

[0068]

[0069] Among them, θ is a learnable weight matrix. Compared with the first strategy, it can realize the dynamic integration of knowledge learned from different trees.

[0070] In specific implementation, the optimization target is set for the multimodal feature fusion model through the following steps:

[0071] Optimization goals Among them, y is the true label, y c is the probability distribution of the true label, To predict the label, is the probability distribution of the predicted label, N c is the number of land cover types, and c is the index of the land cover type.

[0072] Specifically, the classification method (CM) of the embodiment of the present invention 2 FE) follows the general supervised multimodal learning paradigm and is optimized using cross-entropy loss:

[0073]

[0074] Among them, y and Represent the true label and predicted label respectively, N c is the number of land cover types.

[0075] The effect of using the embodiment of the present invention to perform land cover classification is verified below.

[0076] The experiment uses two multimodal remote sensing datasets, WHU-OPT-SAR and Pohang. WHU-OPT-SAR includes four-band multispectral images (RGB and near-infrared) and one-band SAR images with a spatial resolution of 5 meters and 7 land cover types. It is cropped into 7,000 images with a pixel size of 512×512, of which 5,600 are used for training and 1,400 are used for testing. Pohang includes three-band optical images and one-band SAR images with a spatial resolution of 1 meter and 6 types of surface cover. It is cropped into 6,145 images with a size of 256×256, of which 4,916 are used for training and 1,229 are used for testing. Among them, the sixth land cover type was discarded during analysis because the percentage was too small.

[0077] The experiment uses the AdamW optimizer, where β1 and β2 are set to 0..9 and 0.999 respectively, and the decay coefficient is 0.01. The learning rate is 6e-5, the number of iterations is 80000 times, and the batch size is 8. Each expert tree contains four layers, N h Set to (4, 8, 16, 32) in sequence, N s Set to (32,,16,8,4) in sequence. N p The value of is always 128. For WHU-OPT-SAR, the initial rank values ​​of the four forests are set to (16, 32, 80, 128) from shallow to deep. For Pohang, the setting is changed to (8, 16, 40, 64). Each forest contains three trees, where the rank value of is doubled one by one. The learnable parameters φ and θ are implemented by two fully connected layers. The second layer of each tree is used for expert exchange. When an expert in one tree is randomly exchanged, the other two trees each have a 15% probability of feature learning.

[0078] As shown in Table 1, the classification results of different methods on WHU-OPT-SAR.

[0079]

[0080] Table 1

[0081] As shown in Table 2, the classification results of different methods on Pohang.

[0082]

[0083] Table 2

[0084] Tables 1 and 2 compare the classification results of different methods, among which CM 2 FE-R and CM 2FE-L represents the classification method of the embodiment of the present invention using a completely random exchange and a learnable soft exchange strategy. It can be seen that the method proposed in the embodiment of the present invention can achieve the best performance. On WHU-OPTSAR, compared with the second place, CM 2 The mIoU of FE-L increases by 3.25% and mF1 increases by 2.89%. On Pohang, the improvements are 1.60% and 1.00%, respectively. For each category, the performance of the proposed method is still better than that of other methods. This shows that CM 2 FE can learn more robust and discriminative multimodal features through deeper and wider fusion. 2 The results of FE-L are slightly better than those of CM 2 FE-R, indicating that learnable soft swapping has a better promoting effect on feature fusion than random swapping. Figure 6 and Figure 7 The classification results using different land cover classification methods are visualized. 2 The classification graph of FE-L is more accurate and corrects the noise and misclassification phenomena. The effectiveness of the proposed method is verified again from the perspective of visualization.

[0085] In this embodiment, a computer device is provided, such as Figure 8 As shown, it includes a memory 801, a processor 802, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any of the above-mentioned multimodal land cover classification methods is implemented.

[0086] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0087] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned multimodal land cover classification methods.

[0088] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0089] Based on the same inventive concept, a multimodal land cover classification device is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the multimodal land cover classification device is similar to that of the multimodal land cover classification method, the implementation of the multimodal land cover classification device can refer to the implementation of the multimodal land cover classification method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0090] Fig. 9 is a structural block diagram of a multi-modal land cover classification device according to an embodiment of the present invention, such as Fig. 9 As shown, it includes: a model building module 901, a feature fusion module 902 and a land cover classification module 903. The structure is described below.

[0091] A model construction module 901 is used to construct a first backbone network and a second backbone network, and to construct a multimodal cascade expert forest. A multimodal feature fusion model is constructed by using the first backbone network, the second backbone network, and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests.

[0092] The feature fusion module 902 is used to input the SAR image data into the first backbone network, output the first feature s, input the optical image data into the second backbone network, output the second feature o, input the first feature s and the second feature o into each multimodal expert forest of the multimodal cascade expert forest, and output the third feature z;

[0093] The land cover classification module 903 is used to set the optimization target for the multimodal feature fusion model, generate the fused feature after resizing, channel connection and convolution processing of the third feature z, and determine whether the fused feature meets the optimization target. If the optimization target is met, the land cover is classified according to the fused feature.

[0094] In one embodiment, the model building module includes:

[0095] Expert construction unit for constructing a linear projection matrix containing the first and the second linear projection matrix experts, among whom, r is the rank, d is the output dimension, rank d r Less than the input dimension d, R is the feature dimension space;

[0096] An expert tree construction unit, used for constructing a multimodal expert tree including a plurality of experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer;

[0097] The expert forest construction unit is used to connect multiple multimodal expert trees to form a multimodal expert forest, and to cascade multiple multimodal expert forests to form a multimodal cascade expert forest.

[0098] In one embodiment, the feature fusion module includes:

[0099] An expert tree feature acquisition unit, used for inputting the first feature s and the second feature o into the multimodal expert tree, and outputting the learned multimodal features corresponding to the multimodal expert tree;

[0100] Expert forest feature acquisition unit, used to obtain the jth multimodal expert tree T in the i-th multimodal expert forest i,j The learned multimodal features z i,j , by learning the multimodal features z i,j The multimodal expert trees at the same level are combined to generate a multimodal expert forest. The feature z output by the multimodal expert forest is i =concate(z i,j ), where j = 1, 2, 3, z i is the feature of the i-th multimodal expert forest;

[0101] The feature cascade unit is used to construct a cross-level cascade of a multimodal expert forest to generate a multimodal cascade expert forest. The feature z output by the multimodal cascade expert forest is i,j =T i,j (cov(z i-1,j )+concate(s i ,o i )),i=2,3,4, where s i is the first feature of the ith network block of the first backbone network, o i is the second characteristic of the i-th network block of the second backbone network;

[0102] The feature exchange unit is used to exchange a multimodal expert tree T in the multimodal expert forest through the expert exchange strategy. i,j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer l in another multimodal expert tree in the multimodal expert forest.

[0103] In one embodiment, the expert tree feature acquisition unit is used to concatenate the first feature s and the second feature o along the channel dimension to generate an input token sequence Transform the input token sequence t into a super token sequence Among them, N t is the number of tokens, d is the feature dimension; the super token sequence P is input into the multimodal expert tree to generate the feature output P of each layer of the multimodal expert tree l+1,i ,in, i is the index of the super token, is the number of super tokens that each expert needs to process at level l, is the index of the expert network; obtain the feature output of the last layer of experts Feature output for the last layer of experts Perform token recovery to generate learned multimodal features.

[0104] In one embodiment, the feature exchange unit is used for the multimodal expert tree T i,,j The features of the l+1 layer output Among them, T i,j′ and T ii,j″ are other multimodal expert trees in the same multimodal expert forest, q is a random number generated from an average distribution between 0 and 1, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″.

[0105] In one embodiment, the feature exchange unit is also used for the multimodal expert tree T i,j The features of the l+1 layer output Among them, T i,j′ and Ti,j′′ are other multimodal expert trees in the same multimodal expert forest, θ is a learnable weight matrix, l is any layer in the multimodal expert tree, the value range of j is [1,2,3] and j≠j′≠j", and E is the expert network in the multimodal expert tree.

[0106] In one embodiment, the land cover classification module comprises:

[0107] Optimization target setting unit, used to optimize the target Among them, y is the true label, y c is the probability distribution of the true label, To predict the label, is the probability distribution of the predicted label, N c is the number of land cover types, and c is the index of the land cover type.

[0108] The embodiments of the present invention achieve the following technical effects:

[0109] In order to further improve the performance of land cover classification, a new feature fusion method using cascaded multimodal expert forests is proposed. The cascaded multimodal expert forest contains three key components with a progressive relationship (multimodal expert tree, multimodal expert forest and multimodal cascade expert forest), which enables it to have deeper and broader fusion capabilities; each modal expert tree is constructed in the form of multi-layer experts to extract more discriminative multimodal features, and different expert trees are combined into a cascade forest to fully integrate the different knowledge learned; the land cover classification method uses two expert exchange strategies to further optimize the effects of feature fusion and integration, and improve the generalization performance of multimodal land cover classification; by comparing the classification maps of different classification methods, the classification map generated by the land cover classification method of the embodiment of the present invention is more accurate, and the noise and misclassification phenomena are corrected, which once again verifies the effectiveness of the land cover classification method of this embodiment from the perspective of visualization.

[0110] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal land cover classification method, characterized in that: include: Constructing a first backbone network and a second backbone network, constructing a multimodal cascade expert forest, and constructing a multimodal feature fusion model through the first backbone network, the second backbone network and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests; Inputting SAR image data into the first backbone network to output a first feature s, inputting optical image data into the second backbone network to output a second feature o, inputting the first feature s and the second feature o into each of the multimodal expert forests of the multimodal cascade expert forest to output a third feature z; An optimization target is set for the multimodal feature fusion model, and after resizing, channel connection and convolution processing of the third feature z, a fused feature is generated, and it is determined whether the fused feature meets the optimization target. If so, the land cover is classified according to the fused feature.

2. The multimodal land cover classification method according to claim 1, characterized in that: Construct a multimodal cascade expert forest, including: Construct the first linear projection matrix and the second linear projection matrix experts, among whom, r is the rank, d is the output dimension, rank d r Less than the input dimension d, R is the feature dimension space; Constructing a multimodal expert tree comprising a plurality of experts stacked in different layers, wherein the number of experts in the multimodal expert tree doubles layer by layer; A plurality of the multimodal expert trees are connected to form the multimodal expert forest, and a plurality of the multimodal expert forests are cascaded to form the multimodal cascade expert forest.

3. The multimodal land cover classification method according to claim 1, characterized in that: Inputting SAR image data into the first backbone network to output a first feature s, inputting optical image data into the second backbone network to output a second feature o, inputting the first feature s and the second feature o into each of the multimodal expert forests of the multimodal cascade expert forest to output a third feature z, including: Inputting the first feature s and the second feature o into a multimodal expert tree, and outputting a learned multimodal feature corresponding to the multimodal expert tree; Get the jth multimodal expert tree T in the i-th multimodal expert forest i，j The learned multimodal features z i，j , by the learned multimodal feature z i，j The multimodal expert trees at the same level are combined to generate the multimodal expert forest. The feature z output by the multimodal expert forest is i =concate(z i,j ), where j = 1, 2, 3, z i is the feature of the i-th multimodal expert forest; Construct a cross-level cascade of the multimodal expert forest to generate the multimodal cascade expert forest. The feature z output by the multimodal cascade expert forest is i，j =T i，j (cov(z i-1，j )+concate(s i , o i )), i = 2, 3, 4, where s i is the first feature of the ith network block of the first backbone network, o i is a second characteristic of the i-th network block of the second backbone network; One of the multimodal expert trees T in the multimodal expert forest is replaced by an expert exchange strategy. i，j The output feature of any layer in the multimodal expert tree is replaced by the output feature of any layer in another multimodal expert tree in the multimodal expert forest.

4. The multimodal land cover classification method according to claim 3, characterized in that: Inputting the first feature s and the second feature o into the multimodal expert tree, and outputting the learned multimodal features corresponding to the multimodal expert tree, including: Concatenate the first feature s and the second feature o along the channel dimension to generate an input token sequence Transform the input token sequence t into a super token sequence Where N is the number of tokens and d is the feature dimension; The super token sequence P is input into the multimodal expert tree to generate feature outputs Pl+1i of each layer of the multimodal expert tree, where: i is the index of the super token, is the number of super tokens that each expert needs to process at level l, An index of the expert network; Get the feature output of the expert in the last layer The feature output of the expert in the last layer Perform token recovery to generate learned multimodal features.

5. The multimodal land cover classification method according to claim 3, characterized in that: One of the multimodal expert trees T in the multimodal expert forest is replaced by an expert exchange strategy. i，j The feature outputted by any layer l in the multimodal expert tree is replaced by the feature outputted by any layer l in another multimodal expert tree in the multimodal expert forest, including: The multimodal expert tree T i，j The features of the l+1 layer output Among them, T i，j ′ and T i，j ″ is another multimodal expert tree in the same multimodal expert forest, q is a random number generated with an average distribution between 0 and 1, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″.

6. The multimodal land cover classification method according to claim 3, characterized in that: One of the multimodal expert trees T in the multimodal expert forest is replaced by an expert exchange strategy. i，j The features output by any layer l in the multimodal expert tree are replaced by the features output by any layer in another multimodal expert tree in the multimodal expert forest, including: The multimodal expert tree T i，j The features of the l+1 layer output Among them, T i，j′ and T i，j″ are other multimodal expert trees in the same multimodal expert forest, θ is a learnable weight matrix, l is any layer in the multimodal expert tree, the value range of j is [1, 2, 3] and j≠j′≠j″, and E is the expert network in the multimodal expert tree.

7. The multimodal land cover classification method according to any one of claims 1 to 6, characterized in that: Setting an optimization target for the multimodal feature fusion model includes: Optimization goals Among them, y is the true label, y c is the probability distribution of the true label, To predict the label, is the probability distribution of the predicted label, N c is the number of land cover types, and c is the index of the land cover type.

8. A multimodal land cover classification device, characterized in that: include: A model construction module, used to construct a first backbone network and a second backbone network, construct a multimodal cascade expert forest, and construct a multimodal feature fusion model through the first backbone network, the second backbone network and the multimodal cascade expert forest, wherein the first backbone network and the second backbone network each include a plurality of interconnected network blocks, and the multimodal cascade expert forest includes a plurality of interconnected multimodal expert forests; A feature fusion module, used for inputting SAR image data into the first backbone network, outputting a first feature s, inputting optical image data into the second backbone network, outputting a second feature o, inputting the first feature s and the second feature o into each of the multimodal expert forests of the multimodal cascade expert forest, and outputting a third feature z; The land cover classification module is used to set an optimization target for the multimodal feature fusion model, generate a fused feature after resizing, channel connection and convolution processing of the third feature z, and determine whether the fused feature meets the optimization target. If so, the land cover is classified according to the fused feature.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal land cover classification method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the multimodal land cover classification method according to any one of claims 1 to 7.

Images