Remote sensing image classification method and device based on balanced mixed aggregation Mama

Abstract

The invention discloses a remote sensing image classification method and device based on Mama, and the method comprises the steps: obtaining a remote sensing image sequence, and inputting the remote sensing image sequence into a Mama-based balanced mixed classification model to obtain a remote sensing image classification result; the balanced hybrid classification model comprises a gating network, a feature extraction network, a fusion module and a classification module. The gating network extracts remote sensing image features through hierarchical attention aggregation Mamba and outputs routing decision information; the feature extraction network activates expert branches according to the routing decision information to extract remote sensing image features for remote sensing image classification; the training mode of the balanced hybrid classification model is as follows: in the first stage, expert labels are generated for all samples, and the samples with the expert labels are utilized to train a gating network; in the second stage, gating network parameters are frozen, and a sample with an initial category is utilized to train a balanced mixing classification model. The method has good training and reasoning efficiency, and has good generalization ability and classification performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of remote sensing recognition, and particularly relates to a remote sensing image classification method and device based on an aggregated Mamba balanced mixed model. BACKGROUND

[0002] In remote sensing applications, identifying various ground objects through remote sensing image analysis and interpretation is the ultimate goal of remote sensing image processing. Among them, using a classification model to classify remote sensing images is one of the most commonly used means in remote sensing image information extraction. In practice, many applications will use classification models, for example: ground object extraction, land dynamic change monitoring, thematic map production and remote sensing image library establishment all cannot do without remote sensing image classification. Through the use of a classification model to classify and identify remote sensing images, the properties and characteristics of ground objects can be more accurately described, and positive effects will be produced in crop yield estimation, land use, environmental monitoring, engineering survey, natural disaster monitoring, infrastructure layout, transportation planning and management, tourism development and investigation of various resources, thereby better serving these fields.

[0003] In the prior art, Google proposed a high-efficiency large-scale sparse neural network architecture, namely Switch Transformer, in 2021. The core idea of the model is to introduce a mixed expert model to automatically route input data to different sub-networks for processing, thereby realizing image classification. This architecture reduces the computational complexity by simplifying the routing strategy while maintaining the model's expressive power, allowing the model parameter quantity to be significantly increased, and the actual computational cost is only related to the number of activated experts, thereby achieving higher training and inference efficiency. However, in existing mixed expert models, the gating and expert sub-networks are trained simultaneously, which makes the expert load heavily dependent on the gating initialization, ultimately causing the problem of severe imbalance in expert load. Therefore, if the existing mixed expert model is applied to remote sensing image classification, although good training and inference efficiency can be achieved, the problem of severe imbalance in expert load will lead to insufficient generalization ability and significant decline in classification performance of the remote sensing image classification model.

[0004] Therefore, there is an urgent need for a remote sensing image classification method that has good training and inference efficiency, as well as good generalization ability and classification performance. SUMMARY

[0005] In order to solve the above problems existing in the prior art, the application provides a remote sensing image classification method and device based on balanced mixed aggregated Mamba.

[0006] The technical problem to be solved by the application is solved by the following technical scheme: In a first aspect, the present invention provides a remote sensing image classification method based on balanced hybrid aggregation Mamba, comprising: Acquire a remote sensing image sequence; the remote sensing image sequence includes a multispectral image sequence and a panchromatic image sequence; Inputting the remote sensing image sequence into a pre-trained balanced hybrid classification model of aggregated Mamba to obtain a remote sensing image classification result; the balanced hybrid classification model of aggregated Mamba includes a gating network, a feature extraction network, a fusion module and a classification module; The gating network is used to extract features of the remote sensing image sequence through a hierarchical attention aggregation Mamba and output routing decision information based on the extracted features; the hierarchical attention aggregation Mamba is a feature extraction network formed by introducing a hierarchical attention aggregation mechanism into the visual Mamba; The feature extraction network includes an expert network, which is used to activate expert branches in the expert network according to the routing decision information to extract multispectral image features and panchromatic image features of the remote sensing image sequence; The fusion module is used to fuse the multispectral image features and panchromatic image features extracted by the feature extraction network to obtain fused features; The classification module is used to output the remote sensing image classification result according to the fusion feature; The balanced hybrid classification model of the aggregated Mamba is trained in the following way: In the first stage, all samples are divided into N abstract categories with balanced sample sizes according to the initial categories, and expert labels are generated for the samples. The gating network is trained using the samples with expert labels; the expert labels are used to indicate the expert branches in the expert network that need to be activated for the samples. In the second stage, the parameters of the gating network trained in the first stage are frozen, and the balanced hybrid classification model of the aggregated Mamba is trained using samples with initial categories.

[0007] Optionally, the gating network includes a first feature extraction module, a second feature extraction module, a first feature fusion module and a first multi-layer perceptron; The input of the first feature extraction module is the multispectral image sequence, the input of the second feature extraction module is the panchromatic image sequence, the output of the first feature extraction module and the output of the second feature extraction module are both connected to the input of the first feature fusion module, the output of the first feature fusion module is connected to the input of the first multilayer perceptron, and the first multilayer perceptron outputs the routing decision information; the first feature extraction module and the second feature extraction module both include at least one hierarchical attention aggregation Mamba.

[0008] Optionally, the hierarchical attention aggregation Mamba comprises a remodeling module and an improved visual Mamba; the remodeling module is connected to the improved visual Mamba. The remodeling module is configured to split the input sequence of the hierarchical attention aggregation Mamba along the channel dimension, and input the split input sequence to the improved visual Mamba after remodeling the split input sequence into a three-dimensional form according to spatial positions.

[0009] Optionally, the routing decision information comprises an expert branch number and a weight corresponding to the expert branch number. The feature extraction network comprises a third feature extraction module and a fourth feature extraction module. The input of the third feature extraction module is the multispectral image sequence and the routing decision information, and the input of the fourth feature extraction module is the panchromatic image sequence and the routing decision information; the output of the third feature extraction module and the output of the fourth feature extraction module are both connected to the input of the fusion module; the third feature extraction module and the fourth feature extraction module each comprise at least one expert Mamba equalization mixing module; when more than one expert Mamba equalization mixing module is included, the expert Mamba equalization mixing modules are connected in sequence. The expert Mamba equalization mixing module comprises N hierarchical attention aggregation Mambras and a feedforward neural network; the expert Mamba equalization mixing module activates a hierarchical attention aggregation Mamba with a corresponding number according to the expert branch number; the activated hierarchical attention aggregation Mamba extracts features from a sequence input to the expert Mamba equalization mixing module, weights and sums the extracted features according to a weight corresponding to the expert branch number to obtain a weighted feature sequence, element-wise adds the weighted feature sequence and the sequence input to the expert Mamba equalization mixing module to obtain a new feature sequence, and element-wise adds the new feature sequence after processing by the feedforward neural network to the new feature sequence to obtain a feature sequence output by the expert Mamba equalization mixing module; the expert network is the N hierarchical attention aggregation Mambras in the expert Mamba equalization mixing module; and the expert branch is the hierarchical attention aggregation Mamba in the expert Mamba equalization mixing module.

[0010] Optionally, the dividing all samples into N abstract categories with balanced sample sizes according to the initial categories and generating expert labels of the samples comprises: dividing the total number of samples by the number of expert branches to obtain an ideal load of each expert branch; mapping all samples into N abstract categories with balanced sample sizes according to the initial categories according to the ideal load, and taking the abstract category to which the sample is mapped as an expert label of the sample.

[0011] In a second aspect, the present application provides a remote sensing image classification device based on balanced mixed aggregation Mamba, comprising: an acquisition module configured to acquire a remote sensing image sequence; the remote sensing image sequence comprises a multispectral image sequence and a panchromatic image sequence; a classification module configured to input the remote sensing image sequence into a pre-trained balanced mixed classification model of aggregation Mamba to obtain a remote sensing image classification result; the balanced mixed classification model of aggregation Mamba comprises a gating network, a feature extraction network, a fusion module and a classification module; the gating network is configured to extract features of the remote sensing image sequence by a hierarchical attention aggregation Mamba, and output routing decision information according to the extracted features; the hierarchical attention aggregation Mamba is a feature extraction network formed by introducing a hierarchical attention aggregation mechanism into a visual Mamba; the feature extraction network comprises an expert network configured to activate an expert branch in the expert network according to the routing decision information to extract multispectral image features and panchromatic image features of the remote sensing image sequence; the fusion module is configured to fuse the multispectral image features and the panchromatic image features extracted by the feature extraction network to obtain fused features; the classification module is configured to output a remote sensing image classification result according to the fused features; the balanced mixed classification model of aggregation Mamba is trained in the following manner: in a first stage, all samples are divided into N abstract categories with balanced sample sizes according to initial categories, expert labels of the samples are generated, and the gating network is trained using samples with expert labels; the expert label is used to indicate the expert branch in the expert network that needs to be activated for the sample; in a second stage, the parameters of the gating network trained in the first stage are frozen, and the balanced mixed classification model of aggregation Mamba is trained using samples with initial categories.

[0012] Optionally, the dividing all samples into N abstract categories with balanced sample sizes according to the initial categories and generating expert labels of the samples comprises: Divide the total number of samples by the number of expert branches to obtain an ideal load of each expert branch; According to the ideal load, map all samples to N abstract categories with balanced sample sizes according to initial categories, and use the abstract category to which a sample is mapped as the expert label of the sample.

[0013] The remote sensing image classification method based on balanced mixed aggregation Mamba provided by the present application uses a pre-trained balanced mixed classification model of aggregation Mamba to classify remote sensing images; the balanced mixed classification model of aggregation Mamba is obtained through a two-stage training method: in the first stage, first divide all samples into N abstract categories with balanced sample sizes to generate sample expert labels, and then use samples with expert labels to train the gating network; in the second stage, the parameters of the gating network are frozen, and the balanced mixed classification model of aggregation Mamba is trained; by generating expert labels for samples and using the expert labels to guide the training of the gating network, the gating network can be forced to achieve balanced expert allocation, overcoming the problem of severe imbalance in expert load, thereby improving the generalization ability and classification performance of the balanced mixed classification model of aggregation Mamba; and the hierarchical attention aggregation Mamba used by the present application significantly improves the local feature extraction ability of the balanced mixed classification model of aggregation Mamba by introducing a hierarchical attention aggregation mechanism in visual Mamba, thereby improving the remote sensing image classification accuracy.

[0014] The present application will be further described in detail below with reference to the accompanying drawings and the present application. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a structural schematic diagram of the balanced mixed classification model of aggregation Mamba in the remote sensing image classification method based on balanced mixed aggregation Mamba provided by the present application; Figure 2 is a structural schematic diagram of the hierarchical attention aggregation Mamba in the balanced mixed classification model of aggregation Mamba provided by the present application; Figure 3 is a structural schematic diagram of the hierarchical enhancement state space model in the balanced mixed classification model of aggregation Mamba provided by the present application; Figure 4 is a structural schematic diagram of the balanced mixed module of expert Mamba in the balanced mixed classification model of aggregation Mamba provided by the present application; Figure 5 is an effect schematic diagram of the remodeling module in the hierarchical attention aggregation Mamba of the balanced mixed classification model of aggregation Mamba provided by the present application. DETAILED DESCRIPTION

[0016] The application will be described in further detail below with reference to specific embodiments, but the embodiments of the application are not limited thereto.

[0017] In order to overcome the problem that the generalization ability of a remote sensing image classification model is insufficient and the classification performance is significantly reduced due to serious unbalanced load of experts when the existing hybrid expert model is applied to remote sensing image classification, so as to realize a remote sensing image classification scheme with good training and reasoning efficiency, good generalization ability and classification performance, an embodiment of the application provides a remote sensing image classification method and device based on balanced hybrid aggregation Mamba.

[0018] Firstly, the remote sensing image classification method based on balanced hybrid aggregation Mamba provided by the embodiment of the application is described in detail, which includes the following steps: Step one, acquiring a remote sensing image sequence; the remote sensing image sequence includes a multi-spectral image sequence and a panchromatic image sequence.

[0019] In practice, after acquiring the remote sensing image including the multi-spectral (Multi-Spectral, MS) image and the panchromatic (PAN) image, the remote sensing image can also be preprocessed (i.e. Figure 1 PreMS and PrePAN in the above formula), including the following steps: (1) performing data normalization and block processing on the remote sensing image.

[0020] Specifically, the data normalization is to map the pixel value of the remote sensing image to a specified interval through linear scaling, so as to eliminate the influence of the dimension (unit) of the data feature itself. Then, the normalized multi-spectral image and panchromatic image are divided into image blocks of different sizes.

[0021] In this embodiment, before data normalization, first, the multi-spectral image and the panchromatic image of each band are subjected to outlier processing, that is, the outliers (abnormally large values or abnormally small values) existing in the image are assigned a value of 0 or 1, and then the maximum value and the minimum value are determined according to the pixel values in the normal range. ; wherein, represents the pixel value of the image at the i-th position, represents the pixel value of the linearly scaled image at the i-th position, ​​a minimum value among all pixel values of the image, a maximum value among all pixel values of the image.

[0022] After that, the normalized multi-spectral image and the panchromatic image are blocked to obtain multi-spectral image blocks and panchromatic image blocks, for example, the multi-spectral image is cut into 16*16 multi-spectral image blocks, and the panchromatic image is cut into 64*64 panchromatic image blocks.

[0023] (2) The multi-spectral image blocks and the panchromatic image blocks are respectively up-sampled and convolved to realize spatial resolution alignment between multi-source images.

[0024] Specifically, since the multi-spectral image blocks and the panchromatic image blocks have different block sizes, it causes trouble for subsequent feature extraction and feature fusion, so the multi-spectral image blocks are up-sampled before the remote sensing image classification to make the size of the multi-spectral image blocks consistent with that of the panchromatic image blocks. And the panchromatic image blocks are convolved to expand the channel number of the panchromatic image blocks to 4, so as to make it consistent with the channel number of the multi-spectral image. Then, the multi-spectral image blocks and the panchromatic image blocks are converted into a serialized form through patch embedding to obtain multi-spectral image sequences and panchromatic image sequences for subsequent processing.

[0025] Step two, input the remote sensing image sequence into the pre-trained balanced hybrid classification model of the aggregated Mamba to obtain the remote sensing image classification result.

[0026] Specifically, the balanced hybrid classification model of the aggregated Mamba includes a gating network, a feature extraction network, a fusion module and a classification module. The gating network is used to extract the features of the remote sensing image sequence through hierarchical attention aggregated Mamba (HAAM) and output routing decision information according to the extracted features. Here, the hierarchical attention aggregated Mamba is a feature extraction network formed by introducing a hierarchical attention aggregation mechanism into a visual Mamba. The feature extraction network includes an expert network, which is used to activate the expert branches in the expert network according to the routing decision information to extract the multi-spectral image features and the panchromatic image features of the remote sensing image sequence. The fusion module is used to fuse the multi-spectral image features and the panchromatic image features extracted by the feature extraction network to obtain fusion features. The classification module is used to output the remote sensing image classification result according to the fusion features.

[0027] In the present application, the balanced hybrid classification model of the aggregated Mamba is obtained by training in the following way: In the first stage, all samples are divided into N abstract classes with balanced sample size according to the initial classes, and expert labels of the samples are generated, and the gating network is trained using the samples with expert labels. The expert label is used to indicate the expert branch in the activated expert network that needs to be selected for the sample.

[0028] In the second stage, the parameters of the gating network trained in the first stage are frozen, and the balanced mixed classification model of the aggregated Mamba is trained using the samples with initial classes.

[0029] In the first stage, all samples are divided into N abstract classes with balanced sample size according to the initial classes, and expert labels of the samples are generated, and the gating network is trained using the samples with expert labels. The expert label is used to indicate the expert branch in the activated expert network that needs to be selected for the sample.

[0030] Specifically, the conventional mixed expert model usually adopts a random initialization method to simultaneously start training the gating network and the expert network, so that in the initial training stage, the gating network randomly selects any expert branch, and as the training progresses, the expert branch that is frequently selected is continuously strengthened due to continuous use, thereby attracting more selection, while other expert branches fall behind due to insufficient training, and finally lead to serious expert load imbalance. As can be seen from the above, the key to solving the problem of expert load imbalance of the mixed expert model lies in: realizing balanced routing decision. Therefore, the present application adopts a two-stage expert specialization training mechanism, and divides the training process of the balanced mixed classification model of the aggregated Mamba into two stages: In the first stage, the expert label is introduced to guide the training of the gating network. Specifically, first, the total number of samples is divided by the number of expert branches to obtain the ideal load of each expert branch; then, the initial classes are combined into N abstract classes (N is the number of expert branches) according to the ideal load, and the sample size of each abstract class is as close to the ideal load as possible. This design is derived from the fact that the routing decision information output by the gating network directly determines the selection of the expert branch in the second stage, and the balanced sample class output by the gating network (i.e., the balanced number of expert branch numbers output) will ensure that all expert branches in the second stage bear an even workload.

[0031] In the second stage, the parameters of the gating network trained in the first stage are frozen, and the balanced hybrid classification model of the aggregated Mamba is trained using samples with initial categories. Here, the sample block with the initial category is input into the balanced hybrid classification model of the aggregated Mamba, and each pixel point of the sample is classified using the balanced hybrid classification model of the aggregated Mamba to obtain a classification result (i.e., the category of the pixel point). The network parameters of the balanced hybrid classification model of the aggregated Mamba are optimized by solving the error between the classification result and the initial category and performing back propagation on the error, and a trained balanced hybrid classification model of the aggregated Mamba is obtained. By adopting the two-stage expert specialization training mechanism, the balanced hybrid classification model of the aggregated Mamba can achieve almost absolute expert balanced load, so that each expert branch can focus on a specific sample subset, thereby greatly improving the classification performance and generalization ability of the model.

[0032] In the present application, referring to Figure 1 , the gating network comprises a first feature extraction module, a second feature extraction module, a first feature fusion module and a first multilayer perceptron; wherein the input of the first feature extraction module is a multispectral image sequence, the input of the second feature extraction module is a panchromatic image sequence, the output of the first feature extraction module and the output of the second feature extraction module are both connected to the input of the first feature fusion module, the output of the first feature fusion module is connected to the input of the first multilayer perceptron, and the first multilayer perceptron outputs routing decision information. The first feature extraction module and the second feature extraction module each contain at least one hierarchical attention aggregated Mamba. And when the first feature extraction module and the second feature extraction module each contain a plurality of hierarchical attention aggregated Mambas, the plurality of hierarchical attention aggregated Mambas are connected in sequence.

[0033] Referring to Figure 2 and Figure 3 , the hierarchical attention aggregated Mamba comprises a reshaping module and an improved visual Mamba. The reshaping module is connected to the improved visual Mamba. Here, the reshaping module is used to split the input sequence of the hierarchical attention aggregated Mamba along the channel dimension, and the split input sequence is reshaped into a three-dimensional form according to the spatial position and then input into the improved visual Mamba. The improved visual Mamba is an improved module formed by introducing a multi-head mechanism into a 2D selective scanning block of the visual Mamba, wherein the multi-head mechanism is used to decompose the sequence input into the 2D selective scanning block along the feature dimension, so that the 2D selective scanning block independently processes the sub-sequences obtained by decomposition, and the results of independently processing each sub-sequence are spliced along the feature dimension as the output of the 2D selective scanning block.

[0034] Specifically, Mamba is a revolutionary state-space model (SSM) architecture proposed by a team from Carnegie Mellon University and Princeton University in 2023, and Visual Mamba introduces Mamba into the field of vision. Specifically, Visual Mamba follows the block design of the Mamba architecture, but deletes the gated z branch, and adds a feedforward neural network structure to the end of the network with reference to the Visual Transformer. This structure of Visual Mamba achieves global modeling of sequences under linear time complexity, and the Visual Mamba model has demonstrated good performance on multiple visual tasks. However, as Figure 5 As shown, when introducing the Mamba model into the visual field, Visual Mamba only replaces the one-dimensional convolution with the depth convolution. However, the convolution object at this time is the sequence data after the original image is patched, which are not adjacent in physical space. Therefore, the Visual Mamba model is not only difficult to supplement local features, but also introduces useless spatial noise. Therefore, the present invention introduces a reshaping module into the hierarchical attention aggregation Mamba to reshape the sequence of the input hierarchical attention aggregation Mamba to restore the proximity of the input sequence in physical space, so that the improved Visual Mamba can successfully extract the local features of the input sequence, thereby improving the sensitivity of the balanced hybrid classification model of the aggregation Mamba to local features and improving the classification accuracy of remote sensing images. For details, see Figure 2 In the improved visual Mamba, the first convolutional layer (Conv) extracts features from the reshaped 3D image, and the second convolutional layer (Conv) converts the extracted features back into sequence data as the input of the Hierarchical Enhanced SSM (HESSM).

[0035] Here, the Hierarchical Enhanced SSM (HESSM) is an improved module that introduces a multi-head mechanism into the 2D selective scanning block of the visual Mamba. Figure 3 In HESSM, the sequence input to the HESSM is split into multiple subsequences along the feature dimension by the Chunk module, allowing the 2D Selective Scan block to independently process each subsequence. The Concat module then concatenates the results of the independent processing of each subsequence along the feature dimension as the output of the 2D Selective Scan block. The introduction of the multi-head mechanism not only enhances the representational capabilities of the Visual Mamba but also shortens its sequence length, achieving attention aggregation.

[0036] In the present invention, the routing decision information output by the gating network includes: the expert branch number and the weight corresponding to the expert branch number.Figure 1 The feature extraction network comprises a third feature extraction module and a fourth feature extraction module. The input of the third feature extraction module is the multispectral image sequence and the routing decision information, and the input of the fourth feature extraction module is the panchromatic image sequence and the routing decision information. The output of the third feature extraction module and the output of the fourth feature extraction module are both connected to the input of the fusion module. Here, the third feature extraction module and the fourth feature extraction module each contain at least one Balanced Mixture of Experts Mamba (BMoEM). When there are more than one Balanced Mixture of Experts Mamba, the Balanced Mixture of Experts Mambas are connected in sequence.

[0037] The routing decision information is obtained by the routing decision network. Figure 4 The Balanced Mixture of Experts Mamba comprises N hierarchical attention aggregation Mambas (HAAM) and a feedforward neural network. According to the expert branch number, the corresponding number of hierarchical attention aggregation Mambas is activated. The activated hierarchical attention aggregation Mambas perform feature extraction on the sequence input to the Balanced Mixture of Experts Mamba. The extracted features are weighted and summed according to the weight corresponding to the expert branch number to obtain a weighted feature sequence. The weighted feature sequence and the sequence input to the Balanced Mixture of Experts Mamba are added element by element to obtain a new feature sequence. The new feature sequence is processed by the feedforward neural network and then added element by element with the new feature sequence to obtain the feature sequence output by the Balanced Mixture of Experts Mamba. The expert network is the N hierarchical attention aggregation Mambas in the Balanced Mixture of Experts Mamba. The expert branch is the hierarchical attention aggregation Mamba in the Balanced Mixture of Experts Mamba.

[0038] Specifically, the feature extraction network extracts multispectral image features through the third feature extraction module and extracts panchromatic image features through the fourth feature extraction module. Then, the fusion module fuses the multispectral image features and the panchromatic image features to obtain fusion features. The classification module outputs a remote sensing image classification result according to the fusion features. Here, the Balanced Mixture of Experts Mamba combines the mixed expert model with the Mamba backbone network (i.e., the expert network is composed of N hierarchical attention aggregation Mambas), so that the parameter amount of the Balanced Mixture of Experts Mamba classification model can be greatly increased, and the actual calculation cost is only related to the number of activated expert branches, thereby realizing higher training and inference efficiency.

[0039] The remote sensing image classification method based on balanced mixed aggregation Mamba provided by the present application uses a pre-trained balanced mixed classification model of aggregation Mamba to classify remote sensing images; the balanced mixed classification model of aggregation Mamba is obtained through a two-stage training method: in the first stage, first, all samples are divided into N abstract classes with balanced sample sizes to generate sample expert labels, and then the gating network is trained using the samples with expert labels; in the second stage, the parameters of the gating network are frozen, and the balanced mixed classification model of aggregation Mamba is trained; by generating expert labels for samples and using the expert labels to guide the training of the gating network, the gating network can be forced to achieve balanced expert allocation, overcoming the problem of serious imbalance in expert load, thereby improving the generalization ability and classification performance of the balanced mixed classification model of aggregation Mamba; and the hierarchical attention aggregation Mamba used in the present application restores the physical spatial adjacency of the input sequence before feature extraction by introducing a reshaping module, thereby significantly improving the local feature extraction capability of the balanced mixed classification model of aggregation Mamba, thereby improving the classification accuracy of remote sensing images.

[0040] The remote sensing image classification method based on balanced mixed aggregation Mamba provided by the present application is further described below through experiments.

[0041] The simulation conditions and experimental parameter settings are as follows: The hardware platform is GPU RTX2080Ti with 12G memory, and the software platform is PyTorch. The remote sensing images for simulation experiments are provided by the Ministry of Education Key Laboratory of Intelligent Perception and Image Understanding, and specifically include data from three city sites of Xi'an, Xi'an City and Nanjing. Each group of data includes multispectral images, panchromatic images and corresponding ground truth maps. The spatial resolution of the multispectral images in the Xi'an data set is 2m, the spatial resolution of the panchromatic images is 8m, and the ground scene label includes 12 different categories. The spatial resolution of the multispectral images in the Xi'an City data set is 2m, the spatial resolution of the panchromatic images is 8m, and the ground scene label includes 7 different categories. The spatial resolution of the multispectral images in the Nanjing data set is 2m, the spatial resolution of the panchromatic images is 8m, and the ground scene label includes 11 different categories.

[0042] In the method of the present application, the parameter settings of the hierarchical attention aggregation Mamba in the gating network are as follows: The number of input neurons of the input linear mapping layer (i.e. MLP1 in Figure 2 ) is 256, and the expansion coefficient is 1; the number of feature mapping maps of the two convolutional layers is 4, and the filter size is 3; the number of states of HESSM is 24, and the sequence length is 64; the number of input neurons of the output linear mapping layer (i.e. MLP2 in Figure 2The input neuron number of the MLP1 in the input linear mapping layer is 64, and the expansion coefficient is 1; the feature mapping number of the two convolution layers is 4, and the filter size is set to 3; the state number of the HESSM is 24, and the sequence length is set to 256; the output neuron number of the MLP2 in the output linear mapping layer is 64, and the expansion coefficient is 1; the input neuron number of the feedforward neural network layer is 64, the hidden layer neuron number is 256, and the output neuron number is 64.

[0043] The parameter settings of the expert Mamba in the equalization mixing module of the method are as follows: The number of expert branches in the equalization mixing module of the expert Mamba is set to 4, and the number of activated expert branches is set to 2; each expert branch is a hierarchical attention aggregation Mamba, and the parameter settings of the hierarchical attention aggregation Mamba are as follows: the input linear mapping layer (i.e. Figure 2 The input neuron number of the MLP1 in the input linear mapping layer is 64, and the expansion coefficient is 1; the feature mapping number of the two convolution layers is 4, and the filter size is set to 3; the state number of the HESSM is 24, and the sequence length is set to 256; the output neuron number of the MLP2 in the output linear mapping layer is 64, and the expansion coefficient is 1; the input neuron number of the feedforward neural network layer is 64, the hidden layer neuron number is 256, and the output neuron number is 64. Figure 2 The input neuron number of the MLP1 in the input linear mapping layer is 64, and the expansion coefficient is 1; the feature mapping number of the two convolution layers is 4, and the filter size is set to 3; the state number of the HESSM is 24, and the sequence length is set to 256; the output neuron number of the MLP2 in the output linear mapping layer is 64, and the expansion coefficient is 1; the input neuron number of the feedforward neural network layer is 64, the hidden layer neuron number is 256, and the output neuron number is 64.

[0044] The classification correct rates of the method and other four methods are compared in the experiment, and the results are shown in the following table, wherein F1 is a visual Mamba without improvement for remote sensing image classification; F2 is a hierarchical attention aggregation Mamba of the application for remote sensing image classification; F3 is a gating network of the application, but in the equalization mixing module of the expert Mamba, a mixed expert model is combined with a feedforward neural network (i.e.

[0045] Table 1 Classification correct rates of five methods in simulation

[0046] As can be seen from Table 1, compared with the other four methods, the classification accuracy is greatly improved by using the method of the present application, which is mainly because the present application realizes the balanced expert load of the mixed expert model through the two-stage expert specialization training mechanism, so that the balanced mixed classification model of the aggregated Mamba can improve the performance while reducing the computational overhead. In addition, the hierarchical attention aggregated Mamba introduces the reshaping module, so that the improved visual Mamba can successfully extract the local features of the input sequence. At the same time, by introducing the multi-head mechanism, the representation ability of the improved visual Mamba is enhanced, the pressure of global modeling is relieved, and finally the classification performance of the balanced mixed classification model of the aggregated Mamba is effectively improved.

[0047] Corresponding to the above-mentioned remote sensing image classification method based on balanced mixed aggregated Mamba, the embodiment of the present application also provides a remote sensing image classification device based on balanced mixed aggregated Mamba, which comprises: The acquisition module is configured to acquire a remote sensing image sequence. The remote sensing image sequence comprises a multispectral image sequence and a panchromatic image sequence.

[0048] The classification module is configured to input the remote sensing image sequence into the balanced mixed classification model of the aggregated Mamba which has been pre-trained, to obtain a remote sensing image classification result. The balanced mixed classification model of the aggregated Mamba comprises a gating network, a feature extraction network, a fusion module and a classification module.

[0049] The gating network is configured to extract features of the remote sensing image sequence by the hierarchical attention aggregated Mamba, and output routing decision information according to the extracted features. The hierarchical attention aggregated Mamba is a feature extraction network formed by introducing a hierarchical attention aggregation mechanism in the visual Mamba.

[0050] The feature extraction network comprises an expert network, which is configured to activate an expert branch in the expert network according to the routing decision information, to extract multispectral image features and panchromatic image features of the remote sensing image sequence.

[0051] The fusion module is configured to fuse the multispectral image features and the panchromatic image features extracted by the feature extraction network, to obtain fused features.

[0052] The classification module is configured to output a remote sensing image classification result according to the fused features.

[0053] In the present application, the balanced mixed classification model of the aggregated Mamba is trained in the following way: In the first stage, all samples are divided into N abstract categories with balanced sample size according to the initial categories, to generate expert labels of the samples, and the gating network is trained using the samples with expert labels. The expert label is used to indicate the expert branch in the expert network that needs to be activated for the sample.

[0054] In the second stage, the parameters of the trained gating network in the first stage are frozen, and the balanced mixed classification model of the aggregated Mamba is trained using samples with initial categories.

[0055] Optionally, all samples are divided into N abstract categories with balanced sample sizes according to initial categories, and expert labels of the samples are generated, including: Divide the total number of samples by the number of expert branches to obtain the ideal load of each expert branch; According to the ideal load, all samples are mapped to N abstract categories with balanced sample sizes according to initial categories, and the abstract category to which the sample is mapped is used as the expert label of the sample.

[0056] Optionally, the gating network includes a first feature extraction module, a second feature extraction module, a first feature fusion module, and a first multilayer perceptron. The input of the first feature extraction module is a multispectral image sequence, and the input of the second feature extraction module is a panchromatic image sequence. The output of the first feature extraction module and the output of the second feature extraction module are both connected to the input of the first feature fusion module. The output of the first feature fusion module is connected to the input of the first multilayer perceptron, and the first multilayer perceptron outputs routing decision information. The first feature extraction module and the second feature extraction module each include at least one hierarchical attention aggregated Mamba.

[0057] Optionally, the hierarchical attention aggregated Mamba includes a reshaping module and an improved visual Mamba. The reshaping module is connected to the improved visual Mamba. The reshaping module is configured to split an input sequence of the hierarchical attention aggregated Mamba along a channel dimension, and input the split input sequence to the improved visual Mamba in a three-dimensional form according to spatial positions. The improved visual Mamba is an improved module formed by introducing a multi-head mechanism into a 2D selective scanning block of a visual Mamba. The multi-head mechanism is configured to split an input sequence to the 2D selective scanning block along a feature dimension, so that the 2D selective scanning block independently processes sub-sequences obtained by the splitting, and splices results of the independent processing of the sub-sequences along the feature dimension as an output of the 2D selective scanning block.

[0058] Optionally, the routing decision information includes an expert branch number and a weight corresponding to the expert branch number. The feature extraction network includes a third feature extraction module and a fourth feature extraction module. The input of the third feature extraction module is a multispectral image sequence and routing decision information, and the input of the fourth feature extraction module is a panchromatic image sequence and routing decision information, and the output of the third feature extraction module and the output of the fourth feature extraction module are both connected to the input of the fusion module; the third feature extraction module and the fourth feature extraction module both include at least one expert Mamba balanced mixing module, and when more than one expert Mamba balanced mixing module is included, the expert Mamba balanced mixing modules are connected in sequence; The expert Mamba balanced mixing module includes N hierarchical attention aggregation Mambas and a feedforward neural network, the expert Mamba balanced mixing module activates the hierarchical attention aggregation Mamba of the corresponding number according to the expert branch number, the activated hierarchical attention aggregation Mamba performs feature extraction on the sequence input to the expert Mamba balanced mixing module, the extracted features are weighted and summed according to the weight corresponding to the expert branch number, a weighted feature sequence is obtained, the weighted feature sequence and the sequence input to the expert Mamba balanced mixing module are added element by element, a new feature sequence is obtained, and the new feature sequence is added element by element after being processed by the feedforward neural network, thereby obtaining the feature sequence output by the expert Mamba balanced mixing module; the expert network is the N hierarchical attention aggregation Mambas in the expert Mamba balanced mixing module; and the expert branch is the hierarchical attention aggregation Mamba in the expert Mamba balanced mixing module.

[0059] The remote sensing image classification device based on the balanced mixed aggregation Mamba provided by the application uses the pre-trained balanced mixed aggregation Mamba classification model to perform remote sensing image classification; the balanced mixed aggregation Mamba classification model is obtained through a two-stage training method: in the first stage, all samples are first divided into N abstract categories with balanced sample sizes, sample expert labels are generated, and then the sample with the expert label is used to train the gating network; in the second stage, the parameters of the gating network are frozen, and the balanced mixed aggregation Mamba classification model is trained; by generating expert labels for samples and using the expert labels to guide the gating network training, the gating network can be forced to achieve balanced expert allocation, the problem of serious imbalance of expert load can be overcome, and thus the generalization ability and classification performance of the balanced mixed aggregation Mamba classification model are improved; and the hierarchical attention aggregation Mamba used in the application restores the physical spatial adjacency of the input sequence before feature extraction by introducing a reshaping module, thereby significantly improving the local feature extraction ability of the balanced mixed aggregation Mamba classification model, and thus the classification accuracy of the remote sensing image is improved.

[0060] It should be noted that, for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the part of the method embodiment.

[0061] It is to be understood that the terms "first", "second", and the like, used in the description and in the claims, are used as identifiers for distinguished objects and do not necessarily have an ordinal or chronological significance. It is to be understood that the use of the terms "data" and "information" interchangeably herein does not depend on whether the data or information is actually represented by a signal, and that the terms "data" and "information" are used to refer to values, ideas, or concepts that are represented by signals. The embodiments described in the following examples do not represent all the implementations consistent with the present application. Instead, they are merely examples consistent with some aspects of the present application.

[0062] In the description of the specification, the description using the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific feature or features described in connection with the embodiment or example are included in at least one embodiment or example of the present application. The illustrative expressions in the specification do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0063] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the disclosure, the figures and the appended claims. In the description of the application, the word "comprising" does not exclude other components or steps, the word "a" or "an" does not exclude a plurality, and the word "multiple" means two or more, unless expressly specified otherwise. Further, some measures are recited in mutually different embodiments, but this does not mean that these measures cannot be combined to produce a good result.

[0064] The above is a further detailed description of the present application in conjunction with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the scope of protection of the present application.

Claims

1. A remote sensing image classification method based on balanced hybrid aggregation Mamba, characterized in that: include: Acquire a remote sensing image sequence; the remote sensing image sequence includes a multispectral image sequence and a panchromatic image sequence; Inputting the remote sensing image sequence into a pre-trained balanced hybrid classification model of aggregated Mamba to obtain a remote sensing image classification result; the balanced hybrid classification model of aggregated Mamba includes a gating network, a feature extraction network, a fusion module and a classification module; The gating network is used to extract features of the remote sensing image sequence through a hierarchical attention aggregation Mamba and output routing decision information based on the extracted features; the hierarchical attention aggregation Mamba is a feature extraction network formed by introducing a hierarchical attention aggregation mechanism into the visual Mamba; The feature extraction network includes an expert network, which is used to activate expert branches in the expert network according to the routing decision information to extract multispectral image features and panchromatic image features of the remote sensing image sequence; The fusion module is used to fuse the multispectral image features and panchromatic image features extracted by the feature extraction network to obtain fused features; The classification module is used to output the remote sensing image classification result according to the fusion feature; The balanced hybrid classification model of the aggregated Mamba is trained in the following way: In the first stage, all samples are divided into N abstract categories with balanced sample sizes according to the initial categories, and expert labels are generated for the samples. The gating network is trained using the samples with expert labels; the expert labels are used to indicate the expert branches in the expert network that need to be activated for the samples. In the second stage, the parameters of the gating network trained in the first stage are frozen, and the balanced hybrid classification model of the aggregated Mamba is trained using samples with initial categories.

2. The remote sensing image classification method based on balanced hybrid aggregation Mamba according to claim 1 is characterized in that: The gated network includes a first feature extraction module, a second feature extraction module, a first feature fusion module and a first multi-layer perceptron; The input of the first feature extraction module is the multispectral image sequence, the input of the second feature extraction module is the panchromatic image sequence, the output of the first feature extraction module and the output of the second feature extraction module are both connected to the input of the first feature fusion module, the output of the first feature fusion module is connected to the input of the first multilayer perceptron, and the first multilayer perceptron outputs the routing decision information; the first feature extraction module and the second feature extraction module both include at least one hierarchical attention aggregation Mamba.

3. The remote sensing image classification method based on balanced hybrid aggregation Mamba according to claim 2 is characterized in that: The hierarchical attention aggregation Mamba includes: a reshaping module and an improved visual Mamba; the reshaping module is connected to the improved visual Mamba; The reshaping module is used to segment the input sequence of the hierarchical attention aggregation Mamba along the channel dimension, reshape the segmented input sequence into a three-dimensional form according to the spatial position, and then input it into the improved visual Mamba; the improved visual Mamba is an improved module formed by introducing a multi-head mechanism into the 2D selective scanning block of the visual Mamba. The multi-head mechanism is used to decompose the sequence input to the 2D selective scanning block along the feature dimension, so that the 2D selective scanning block independently processes the subsequences obtained by the decomposition, and splices the results of the independent processing of each subsequence along the feature dimension as the output of the 2D selective scanning block.

4. The remote sensing image classification method based on balanced hybrid aggregation Mamba according to claim 3 is characterized in that: The routing decision information includes: an expert branch number and a weight corresponding to the expert branch number; The feature extraction network includes a third feature extraction module and a fourth feature extraction module; The input of the third feature extraction module is the multispectral image sequence and the routing decision information, and the input of the fourth feature extraction module is the panchromatic image sequence and the routing decision information. The output of the third feature extraction module and the output of the fourth feature extraction module are both connected to the input of the fusion module; the third feature extraction module and the fourth feature extraction module each include at least one expert Mamba balanced mixing module. When more than one expert Mamba balanced mixing module is included, the expert Mamba balanced mixing modules are connected sequentially; The balanced mixing module of the expert Mamba includes N hierarchical attention aggregation Maambas and a feedforward neural network. The balanced mixing module of the expert Mamba activates the hierarchical attention aggregation Maamba with the corresponding number according to the expert branch number. The activated hierarchical attention aggregation Maamba extracts features from the sequence input to the balanced mixing module of the expert Mamba, and performs weighted summation on the extracted features according to the weights corresponding to the expert branch numbers to obtain a weighted feature sequence. The weighted feature sequence and the sequence input to the balanced mixing module of the expert Mamba are added element by element to obtain a new feature sequence. The new feature sequence is processed by the feedforward neural network and then added element by element to the new feature sequence to obtain a feature sequence output by the balanced mixing module of the expert Mamba. The expert network is the N hierarchical attention aggregation Maambas in the balanced mixing module of the expert Mamba. The expert branch is the hierarchical attention aggregation Maamba in the balanced mixing module of the expert Mamba.

5. The remote sensing image classification method based on balanced hybrid aggregation Mamba according to claim 1 is characterized in that: The method divides all samples into N abstract categories with balanced sample sizes according to the initial categories, and generates expert labels for the samples, including: Divide the total number of samples by the number of expert branches to get the ideal load of each expert branch; According to the ideal load, all samples are mapped to N abstract categories with balanced sample sizes according to the initial categories, and the abstract categories to which the samples are mapped are used as expert labels of the samples.

6. A remote sensing image classification device based on balanced hybrid aggregation Mamba, characterized in that: include: An acquisition module is used to acquire a remote sensing image sequence; the remote sensing image sequence includes a multispectral image sequence and a panchromatic image sequence; A classification module is used to input the remote sensing image sequence into a pre-trained balanced hybrid classification model of the aggregated Mamba to obtain a remote sensing image classification result; the balanced hybrid classification model of the aggregated Mamba includes a gating network, a feature extraction network, a fusion module and a classification module; The gating network is used to extract features of the remote sensing image sequence through a hierarchical attention aggregation Mamba and output routing decision information based on the extracted features; the hierarchical attention aggregation Mamba is a feature extraction network formed by introducing a hierarchical attention aggregation mechanism into the visual Mamba; The feature extraction network includes an expert network, which is used to activate expert branches in the expert network according to the routing decision information to extract multispectral image features and panchromatic image features of the remote sensing image sequence; The fusion module is used to fuse the multispectral image features and panchromatic image features extracted by the feature extraction network to obtain fused features; The classification module is used to output the remote sensing image classification result according to the fusion feature; The balanced hybrid classification model of the aggregated Mamba is trained in the following way: In the first stage, all samples are divided into N abstract categories with balanced sample sizes according to the initial categories, and expert labels are generated for the samples. The gating network is trained using the samples with expert labels; the expert labels are used to indicate the expert branches in the expert network that need to be activated for the samples. In the second stage, the parameters of the gating network trained in the first stage are frozen, and the balanced hybrid classification model of the aggregated Mamba is trained using samples with initial categories.

7. The remote sensing image classification device based on balanced hybrid aggregation Mamba according to claim 6 is characterized in that: The method divides all samples into N abstract categories with balanced sample sizes according to the initial categories, and generates expert labels for the samples, including: Divide the total number of samples by the number of expert branches to get the ideal load of each expert branch; According to the ideal load, all samples are mapped to N abstract categories with balanced sample sizes according to the initial categories, and the abstract categories to which the samples are mapped are used as expert labels of the samples.

Images