Multispectral multi-label classification method based on low confusion and spatial spectral self-balancing

Abstract

The application discloses a multispectral multi-label classification method based on low confusion and spatial spectrum self-balancing, belongs to the technical field of meteorological remote sensing image classification, and relates to a spectral convolution network used for efficiently extracting spectral features, a query generation module used for effectively improving the expression capability of label query, a multilayer perceptron used for mapping label embedding into a query vector, and cross-correlation constraints used for reducing the similarity between label queries; in order to sufficiently mine spatial and spectral features, a three-branch network is designed, including a spatial branch, a spectral branch and a comprehensive branch; each branch independently carries out feature learning, and the prediction results of each branch are fused through a branch balance voting mechanism; the training effect of a weak branch is enhanced through the interaction between branches; and finally, more accurate classification results are obtained. In the meteorological multispectral image classification task, especially in the identification of complex meteorological system categories, the application can accurately classify weather systems, ground coverings and clouds.

Description

Technical Field

[0001] This invention belongs to the field of meteorological remote sensing image classification technology, specifically involving a multispectral multi-label classification method based on low confusion and spatial spectral self-balancing. Background Technology

[0002] Weather forecasting is crucial in daily life, and weather changes are caused by atmospheric motion systems known as weather systems. Remote sensing images can be used to observe the generation and development of various weather systems in the Earth's atmosphere, playing a positive role in weather forecasting. Different weather systems are typically composed of different cloud types, and observers can predict weather conditions and infer atmospheric composition by observing cloud types. The Earth's surface cover also directly interacts with and is closely connected to various weather systems and clouds. Therefore, multi-label classification of weather systems, surface layer types, and cloud types within a specific region is of great significance. The Earth's surface cover can be further divided into oceans, deserts, vegetation, etc.; weather systems can be divided into tropical cyclones, subtropical cyclones, etc.; and cloud types can be classified into nine categories according to the ISCCP standard, such as cirrus clouds and cirrostratus clouds. Using satellite remote sensing image data, multi-label classification tasks can predict the existence of these categories based on image information. Therefore, the establishment of a multi-label classification system for remote sensing cloud images is essential.

[0003] Compared to ordinary images, remote sensing images are more unique. Specifically, multispectral remote sensing images contain multiple channels that record light of different wavelengths from the Earth's surface. These different spectra reflect the distribution of different components on the Earth's surface, containing important and rich information. How to effectively utilize the spectral information of multispectral images is a crucial issue. General deep learning models focus more on spatial feature extraction, while there is room for improvement in spectral feature extraction. Besides comprehensively considering spatial and spectral features, effectively integrating the two is also a valuable problem. In addition to effectively utilizing the spatial and spectral features in multispectral data, there are also some problems with the label detection paradigm in multi-label classification problems, such as insufficient semantic encoding of query vectors and susceptibility to inter-class confusion. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a multispectral multi-label classification method based on low confusion and spatial spectral self-balancing. It uses a query vector generation mechanism to represent richer semantic information and limits the similarity between two query vectors to alleviate inter-class confusion. It integrates classification results by combining spatial branch, spectral branch and branch voting mechanism through a three-branch network structure, thereby making full use of the advantages of spatial and spectral features.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multispectral multi-label classification method based on low confusion and spatial spectral self-balancing includes the following steps:

[0007] Step 1: Construct a meteorological multispectral multi-label classification network framework;

[0008] Step 2: Dataset preprocessing: Slice and normalize the multispectral data; divide the data into training, validation, and test sets according to a set ratio; label the data.

[0009] Step 3, Network Training, the process is as follows: Initialize the model parameters with random weights. When the evaluation metric mAP on the validation set no longer increases, the learning rate will be reduced to the original set multiple. Set the initial learning rate of the Adam optimizer and the batch size during training. Perform a set number of rounds of complete training and set the loss weights for cross-correlation constraints. Train the network model until convergence and save the optimal model during training as the training output.

[0010] Step 4: Accuracy testing. Input the preprocessed test set into the optimal model obtained from training, compare the true values ​​of the dataset with the predicted results, and calculate various evaluation indicators.

[0011] Furthermore, in step one, a spatial feature extraction network and a spectral feature extraction network are first constructed; then a query generation module and a three-branch structure are constructed, which includes a spatial branch, a spectral branch, and a comprehensive branch. The labeled features obtained by the self-attention layer in each branch are concatenated with the label query vector obtained by the query generation module, and then passed through a normalization layer and another self-attention layer to obtain the output vector of each branch; finally, a branch balancing voting mechanism is established, which accepts the output vectors of the spatial branch, the spectral branch, and the comprehensive branch as input, and then uses a set of learnable parameters for voting as a reference for weights to perform a weighted average of the predictions among the branches to obtain the final prediction result.

[0012] Preferably, step one includes the following sub-steps:

[0013] Step 1.1: Construct a spatial feature extraction network: Input the meteorological multispectral image into the VGG network to initially extract spatial features. The VGG network contains at least six feature extraction units. In the first five feature extraction units, the input multispectral image features are first processed through a max pooling layer, then through two convolutional layers. Each convolutional layer contains a two-dimensional convolution and a LeakyReLU activation function. Finally, the features are processed through the last feature extraction unit, which consists of three fully connected layers, to obtain the preliminary spatial features.

[0014] Step 1.2: Constructing a spectral feature extraction network: The spectral feature extraction network includes at least four spectral convolutional blocks. Each spectral convolutional block contains two data streams: a feature stream and a global feature stream. The feature stream is responsible for extracting specific features within each channel, while the global feature stream is responsible for collecting feature features at each stage and fusing them with the global features.

[0015] More preferably, step one further includes the following sub-steps:

[0016] Step 1.3: Build the query generation module: In the query generation module, a set of label embeddings is first obtained through the label embedding layer, and then the set of label embeddings is sent to a multilayer perceptron for further processing to obtain the embedded representation of the label. The multilayer perceptron includes a fully connected layer, a GELU activation function and a linear layer. Then, a normalization layer is used to standardize the mapped label embedding representation to obtain the label query vector.

[0017] Step 1.4: Construct a three-branch structure, including a spatial branch, a spectral branch, and a synthesis branch. The spatial and spectral branches accept the spatial features or spectral representations obtained in Steps 1.1 and 1.2 as input. The synthesis branch is composed of the spatial features after passing through a linear layer and the spectral representation after passing through a linear layer. Each branch first passes through a normalization layer, then inputs the features into a Transformer self-attention layer for self-attention calculation, and returns the labeled features and attention weights. At the same time, the labeled features are also concatenated with the label query vector obtained in Step 1.3, then pass through a normalization layer and another Transformer self-attention layer, finally obtaining the output vector of each branch.

[0018] More preferably, step one further includes the following sub-steps:

[0019] Step 1.5: Establish a branch balancing voting mechanism, accept the prediction results of the spatial branch, spectral branch and comprehensive branch as input, and then use a set of learnable parameters for voting as a reference for weights to perform a weighted average of the predictions among the branches to obtain the final prediction result.

[0020] Preferably, in step 1.2, in the feature stream of each spectral convolution block, the features of the previous stage will first pass through a grouped convolution with a kernel size of 4×4, followed by at least four nonlinear layers connected by residual structures. Each nonlinear layer consists of two other grouped convolutions and a LeakyReLU activation function, with the LeakyReLU activation function located in the middle of the two grouped convolutions.

[0021] More preferably, in step 1.2, in the global feature stream, the multispectral image data first passes through a two-dimensional convolution with a kernel size of 1×1, and then enters the spectral convolution block; in the global feature stream of each spectral convolution block, the feature data from the previous stage is first fused with the global features from the previous stage, and then passed through a two-dimensional convolution with a kernel size of 4×4; in the final stage of the spectral feature extraction network, the output of the previous spectral convolution block in the global feature stream is mapped to a spectral representation as the output using another two-dimensional convolution with a kernel size of 4×4.

[0022] Furthermore, in step two, the original multispectral data is processed into several slices according to the settings, these slices are normalized to their maximum and minimum values, and then scaled to the set size; training, validation, and test sets are divided according to the set ratio; and data labels are manually or automatically labeled according to the set requirements.

[0023] Furthermore, step three includes the following sub-steps:

[0024] Step 3.1: Initialize the model parameters using random weights;

[0025] Step 3.2: Use cross-association to limit the degree of association between different query vectors. The cross-association limit calculation method includes:

[0026] First, the inner product between the tag query vectors is calculated to obtain the autocorrelation matrix. Then, an identity matrix is ​​created, where the diagonal elements are 1 and the off-diagonal elements are 0. Next, the magnitude of each tag embedding vector is calculated by squaring the elements of each tag embedding vector, summing them, and then taking the square root. The autocorrelation matrix is ​​then normalized using the product of these magnitudes. The absolute difference between the normalized autocorrelation matrix and the identity matrix is ​​calculated and used as the cross-association constraint loss term.

[0027] Step 3.3: Use binary cross-entropy as the original loss function;

[0028] Step 3.4: Set the initial learning rate for the Adam optimizer;

[0029] Step 3.5: Optimize model parameters.

[0030] Preferably, the process of step 3.5 is as follows:

[0031] Step 3.5.1: Input the training data and validation data slices into the network, and obtain the final output probability of each category through the constructed network;

[0032] Step 3.5.2: For the training data, the loss function in 3.3 is used to measure the error between the true value and the predicted value in the training set. It is also used as the original loss function for each branch in Step 1.6. After that, the branch balancing voting mechanism in Step 1.6 is used to reweight each branch, penalizing weak branches. Then, the reweighted loss value of each branch is added to the loss obtained in Step 3.2 and the loss between the true value and the final output of the model to obtain the final model loss value. The optimizer in Step 3.3 is used to optimize this loss. Then, the classification evaluation index is calculated using the validation data.

[0033] Step 3.5.3: Feed the error generated by the training slices back into the network, adjust the network parameters, and optimize the prediction performance;

[0034] Step 3.5.4: Repeat steps 3.5.1-3.5.3 to set the number of rounds. During the repetition, if the evaluation index on the validation data no longer increases, reduce the learning rate to the original set ratio and save the optimal model in the training process as the training output.

[0035] Step four includes the following sub-steps:

[0036] Step 4.1: Input the preprocessed test set into the optimal model obtained through training;

[0037] Step 4.2: Compare the true values ​​of the dataset with the predicted results and calculate various evaluation indicators.

[0038] The technical effect of this invention is that it can accurately classify weather systems, land cover, and clouds in meteorological multispectral image classification tasks, especially in the identification of complex meteorological system categories. Attached Figure Description

[0039] Figure 1 This is a flowchart of a multispectral multi-label classification method based on low confusion and spatial spectral self-balancing.

[0040] Figure 2 This is a schematic diagram of the forecast network model structure.

[0041] Figure 3 This is a schematic diagram of the spectral feature extraction network structure.

[0042] Figure 4 This is a schematic diagram of the processing flow for each branch.

[0043] Figure 5 This is a schematic diagram of the branch balancing voting mechanism. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0046] Reference Figures 1-5 A multispectral multi-label classification method based on low confusion and spatial spectral self-balancing includes the following steps:

[0047] Step 1: Construct a meteorological multispectral multi-label classification network framework. First, construct a spatial feature extraction network and a spectral feature extraction network. Then, construct a query generation module and a three-branch structure, which includes a spatial branch, a spectral branch, and a comprehensive branch. The labeled features obtained from the self-attention layer in each branch are concatenated with the label query vector obtained from the query generation module. After passing through a normalization layer and another self-attention layer, the output vector of each branch is obtained. Finally, establish a branch balancing voting mechanism, which accepts the output vectors of the spatial branch, spectral branch, and comprehensive branch as inputs. Then, a set of learnable parameters for voting is used as a reference for weights to perform a weighted average of the predictions among the branches to obtain the final prediction result.

[0048] Step one includes the following sub-steps:

[0049] Step 1.1: Constructing a Spatial Feature Extraction Network: The meteorological multispectral image is input into a VGG network for initial spatial feature extraction. The VGG network contains at least six feature extraction units, each consisting of multiple two-dimensional convolutional layers, a LeakyReLU activation function, and a max-pooling layer (MaxPooling2D). Specifically, the first-stage feature extraction unit has 16 input channels. After the first 3×3 convolutional layer, the number of channels expands to 64. Following LeakyReLU activation, a second 3×3 convolutional layer maintains the 64 channels. Then, a 2×2 max-pooling operation halves the feature map size, resulting in an output of 128×128×64. The second-stage feature extraction unit receives the feature map from the first stage as input. It first increases the number of channels to 128 through a 3×3 convolutional operation, then performs LeakyReLU activation, followed by another 3×3 convolutional layer to maintain the same number of channels, and then performs max-pooling to further reduce the feature map size to 64×64×128. The third-stage feature extraction unit continues to perform 3×3 convolution transformations on the feature map from the second stage, increasing the number of channels from 128 to 256. After LeakyReLU activation, it performs two more 3×3 convolution operations to maintain 256 channels, and then reduces the feature map size to 32×32×256 using a 2×2 max pooling layer. The fourth-stage feature extraction unit further performs 3×3 convolution operations on the feature map output from the previous stage, increasing the number of channels to 512. After LeakyReLU activation, it performs two more 3×3 convolutions to maintain 512 channels, and finally performs 2×2 max pooling to reduce the feature map size to 16×16×512. The fifth-stage feature extraction unit continues to use 3×3 convolutions to maintain the number of feature channels at 512. After LeakyReLU activation, it performs two more 3×3 convolutions to ensure the continuity of feature representation. Finally, it further reduces the size to 8×8×512 using 2×2 max pooling, forming the final deep spatial features of the meteorological multispectral data.

[0050] Step 1.2: Constructing a spectral feature extraction network: The spectral feature extraction network includes at least four spectral convolutional blocks. Each spectral convolutional block contains two data streams: a feature stream and a global feature stream. The feature stream is responsible for extracting the specific features within each channel, while the global feature stream is responsible for collecting feature features at each stage and fusing them with the global features.

[0051] In this embodiment, the spectral features of multispectral meteorological remote sensing images can be extracted through convolution and other operations in a spectral convolutional network. These spectral features will then be input into subsequent spectral branches and synthesis branches for further spectral feature extraction.

[0052] Specifically, in the feature flow of each spectral convolution block, the features of the previous stage will first pass through a grouped convolution with a kernel size of 4×4, followed by at least four nonlinear layers connected by residual structures. Each nonlinear layer consists of two other grouped convolutions and a LeakyReLU activation function, with the LeakyReLU activation function located in the middle of the two grouped convolutions.

[0053] Specifically, in the global feature flow, the multispectral image data first passes through a 1×1 2D convolution before entering the spectral convolution block. Within the global feature flow of each spectral convolution block, the feature data from the previous stage is fused with the global features from the previous stage, and then passed through a 4×4 2D convolution. In the final stage of the spectral feature extraction network, the output from the previous spectral convolution block in the global feature flow is mapped to a spectral representation as the output using another 4×4 2D convolution.

[0054] In one specific embodiment, before entering the spectral convolutional block, the meteorological multispectral image in the global feature flow first undergoes convolutions with the following input channel number, output channel number, kernel size, and stride: 16, 1, 1, 1. In the feature flow of the first spectral convolutional block, the intermediate layer channel number, output channel number, kernel size, stride, and number of groups in the first grouped convolution are 16, 64, 4, 4, 16. Then, it passes through a LeakyReLU activation function, followed by four nonlinear layers with residual connections before entering the feature flow of the next spectral convolutional block. In the global feature flow of the first spectral convolutional block, the feature flow of the previous stage is first concatenated with the global feature flow of the previous stage along the channel dimension. Then, the input channel number, output channel number, kernel size, and stride of the convolutional block it passes through are 17, 4, 4, 4. Then, after three more spectral convolutional blocks as described above, the feature output from the fourth spectral convolutional block is processed through a grouped convolution with the following parameters: input channels, output channels, kernel size, stride, and number of groups: 4096, 8192, 1, 1, 16. The resulting new feature is then sliced ​​along dimension one (channel dimension), with each slice consisting of 512 segments, and stored in a list. The global feature output from the fourth spectral convolutional block is processed through another grouped convolution with the same parameters: input channels, output channels, kernel size, stride, and number of groups: 4096, 512, 1, 1, 16. In terms of the specific input and output dimensions, the original meteorological multispectral data has a dimension of 16×256×256, where 16 represents the number of channels. The first spectral convolutional block has 64 feature channels, the second has 256, the third has 1024, and the first has 4096. Finally, the spectral feature network yields a feature list containing 16 features with 512 channels each, and the global feature has 512 channels.

[0055] Specifically, each nonlinear layer in the feature flow of the first spectral convolutional block contains one grouped convolution operation, a ReLU activation function, and another grouped convolution operation. The first nonlinear layer has 16 input channels. Within it, the first grouped convolution operation has the following parameters: input channels, output channels, kernel size, stride, and number of groups: 16, 32, 7, 16. The second grouped convolution operation has the same parameters: input channels, output channels, kernel size, stride, and number of groups: 32, 16, 7, 16. The second nonlinear layer also has 16 input channels. Within it, the first grouped convolution operation has the same parameters: input channels, output channels, kernel size, stride, and number of groups: 16, 32, 5, 16. The second grouped convolution operation has the same parameters: input channels, output channels, kernel size, stride, and number of groups: 32, 16, 3, 16. The third nonlinear layer has 16 input channels. Within it, the first grouped convolution has the following parameters: input channels, output channels, kernel size, stride, and number of groups: 16, 32, 3, 16. The second grouped convolution has the same parameters: input channels, output channels, kernel size, stride, and number of groups: 32, 16, 3, 16. The fourth nonlinear layer is the same as the third.

[0056] In this embodiment, the spatial and spectral features of meteorological multispectral data are extracted through operations such as convolution and pooling. These features will be used in the subsequent label query process, laying a good foundation for subsequent feature processing.

[0057] Step 1.3: Build the query generation module: In the query generation module, a set of label embeddings is first obtained through the label embedding layer, and then the set of label embeddings is sent to a multilayer perceptron for further processing to obtain the embedded representation of the label. The multilayer perceptron includes a fully connected layer, a GELU activation function and a linear layer. Then, a normalization layer is used to standardize the mapped label embedding representation to obtain the label query vector.

[0058] To optimize the model's category representation ability and improve classification performance, we introduce a cross-association constraint during label embedding to ensure stronger discriminative power between feature representations of different categories. Let any two query vectors in the label query vector be denoted as... as well as Then the cross-association constraint loss L can be formally expressed as:

[0059] ;

[0060] Where C is the total number of categories, which is 17 in this case; and the values ​​of i and j range from... .

[0061] The calculated cross-association term loss will be used as part of the final model loss. This orthogonality constraint effectively reduces redundant information between class embedding vectors, making class distinctions clearer and improving the model's discriminative ability in multi-label classification tasks.

[0062] Step 1.4: Construct a three-branch structure, including a spatial branch, a spectral branch, and a synthesis branch. The spatial and spectral branches are used to receive the spatial features or spectral representations obtained in Steps 1.1 and 1.2 as input. The input of the synthesis branch is composed of the spatial features after passing through a linear layer and the spectral representation after passing through a linear layer. Each branch first passes through a normalization layer, then inputs the features into a Transformer self-attention layer for self-attention calculation, and returns the labeled features and attention weights. At the same time, the labeled features are also concatenated with the label query vector obtained in Step 1.3, then pass through a normalization layer and another Transformer self-attention layer, and finally the output vector of each branch is obtained.

[0063] The category labels of meteorological multispectral data are transformed into a 512-dimensional feature representation through an embedding layer, with a shape of [17, 512], where 17 represents the number of categories. This feature representation is then normalized by a LayerNorm layer and subjected to two linear transformations, mapping it to a high-dimensional feature space to ensure the separability and stability of category information. The generated label query vector is used for both spatial feature interaction and spectral feature interaction. During spatial feature extraction, specifically in the spatial branch, the spatial features of the obtained meteorological multispectral data are transformed from a shape of [64, C, H, W] to generate spatial semantic labels with a shape of [64, N, 512], where C represents the number of channels, H represents the feature height, W represents the feature width, and N = (C × H × W) / 512. These labels are then normalized by a LayerNorm layer and input into a self-attention layer to calculate multi-head self-attention, modeling long-range dependencies in spatial information and accumulating the attention matrix for analysis. Subsequently, the tag query vector is copied 64 times, where 64 represents the training batch size. This copy is then concatenated with the spatial semantic tags obtained above along the token count dimension, resulting in a shape of [64, N + 17, 512]. This concatenated shape is then passed through a normalization layer and another self-attention layer for category feature interaction, allowing category information to further guide the spatial feature classification learning during the Transformer computation. Finally, the last 17 tokens are extracted as category prediction features for subsequent classification decisions. This process enhances the expressive power of category information through tag embedding and introduces global information into spatial features using the Transformer mechanism, improving the model's classification performance and generalization ability.

[0064] Step 1.5: Establish a branch balancing voting mechanism, accept the prediction results of the spatial branch, spectral branch and comprehensive branch as input, and then use a set of learnable parameters (denoted as V) for voting as a reference for weights to perform a weighted average of the predictions among the branches to obtain the final prediction result.

[0065] Reference Figure 5 If we denote the predicted probability of the model for the i-th class as... Then it can be formally represented as:

[0066] ;

[0067] in e It is the natural logarithm. , , Indicates different branches, Indicates branch The voting weight for class i, Indicates branch The voting weight for class i, Representative branch For the predicted probability of class i, any value in the learnable parameters is initialized to 1 / 3, since the number of branches in this method is 3. To ensure that the voting weights have a stable weight distribution during training, they are normalized using the softmax function, so that different feature branches are normalized in the direction of dim=0, ensuring that the sum of the voting weights of the three branches for each class is 1.

[0068] Furthermore, the original loss for each branch for the i-th class can be written as:

[0069] ;

[0070] After obtaining the original loss, this loss term needs to be reweighted using voting weights. The reweighting parameters on branch b, category i... It can be represented as:

[0071] ;

[0072] in It is a hyperparameter greater than 1. k This is the number of branches, which is 3 in this case; log The base of the function is e Therefore, the loss value after reweighting a branch can be written as:

[0073] ;

[0074] Therefore, during training, the voting weights V become aware of the capability of each branch in each category. The core idea of ​​this design is: if a feature branch contributes less to a category, its voting weight is also smaller, resulting in a larger penalty factor. This increases the penalty and leads to a larger gradient adjustment for that feature in the loss function, prompting the model to learn a more reasonable weight allocation. Since the goal of the reweighting process is to penalize weak branches, the gradient of the reweighting operation itself is stopped and backpropagated.

[0075] This mechanism ensures a dynamic balance among spatial, spectral, and overall features, and can adaptively adjust the importance of each branch, thereby improving the stability and accuracy of the final classification decision.

[0076] Step 2, Dataset Preprocessing: Slice and normalize the multispectral data (Sunflower 8 data); divide the data into training, validation, and test sets according to the ratio of one day's data for validation, one day's data for testing, and the rest for training, and label the data accordingly.

[0077] Step two includes the following sub-steps:

[0078] Step 2.1: The data source was selected from the Himawari-8 satellite at UTC0020 every day of 2019. Northern Hemisphere portion of the raw multispectral data for pixel L1;

[0079] Step 2.2: Use each slice The original multispectral data is processed into several slices with a size of 200 between each two slices. These slices are then normalized to a minimum-maximum value to limit the values ​​to the range [-1, 1], and then the size is scaled down. The size of the pixels facilitates model training and testing;

[0080] Step 2.3: Divide the data into training, validation, and test sets according to the ratio of one day's data for validation, one day's data for testing, and the rest for training every ten days;

[0081] Step 2.4: Data labeling adopts a combination of manual and automatic methods. Tropical cyclones, subtropical cyclones, fronts, westerly jet streams, snow, oceans, deserts, and vegetation are labeled manually, while cirrus, cirrostratus, deep convection, altocumulus, altostratus, nimbostratus, cumulus, stratocumulus, and stratus are labeled automatically. Automatic labeling uses the L2-level satellite cloud classification (CTYPE) product of the Himawari-8 satellite. If the proportion of a certain type of cloud in the cloud product corresponding to the slice is >5%, then the label is considered to be sufficiently significant in the slice and the corresponding label needs to be labeled on the slice.

[0082] Step 3: Network training, the process is as follows: Initialize the model parameters with random weights. When the validation set evaluation metric mAP no longer increases (stops increasing for 3 rounds), the learning rate will be reduced to 1 / 5 of its original value. Set the initial learning rate of the Adam optimizer to 1 / 5. During training, the batch size was set to 64, and a total of 100 rounds of training were performed. The cross-correlation constraint loss weight θ was set to 0.1. The network model was trained until convergence, and the optimal model during the training process was saved as the training output.

[0083] Step three includes the following sub-steps:

[0084] Step 3.1: Initialize the model parameters using random weights;

[0085] Step 3.2: Use cross-association to limit the degree of association between different query vectors. The cross-association limit calculation method includes:

[0086] First, calculate the inner product between the tag query vectors to obtain the autocorrelation matrix. Then, create an identity matrix, where the diagonal elements are 1 and the off-diagonal elements are 0. Next, calculate the magnitude of each tag embedding vector by squaring the elements of each tag embedding vector, summing them, and then taking the square root. Use the product of these magnitudes to normalize the autocorrelation matrix. Calculate the absolute difference between the normalized autocorrelation matrix and the identity matrix as the cross-association constraint loss term. The specific formula is described in step 1.3.

[0087] Step 3.3: Use binary cross-entropy as the original loss function;

[0088] Step 3.4: Set the initial learning rate of the Adam optimizer. ;

[0089] Step 3.5: Optimize model parameters.

[0090] Step 3.5.1: Input the training data and validation data slices into the network, and obtain the final output probability of each category through the constructed network;

[0091] Step 3.5.2: For the training data, the loss function in 3.3 is used to measure the error between the true value and the predicted value in the training set. It is also used as the original loss function for each branch in Step 1.6. After that, the branch balancing voting mechanism in Step 1.6 is used to reweight each branch, penalizing weak branches. Then, the reweighted loss value of each branch is added to the loss obtained in Step 3.2 and the loss between the true value and the final output of the model to obtain the final model loss value. The optimizer in Step 3.3 is used to optimize this loss. Then, the classification evaluation index is calculated using the validation data.

[0092] Step 3.5.3: Feed the error generated by the training slices back into the network, adjust the network parameters, and optimize the prediction performance;

[0093] Step 3.5.4: Repeat steps 3.5.1-3.5.3 for one hundred rounds. During the repetition, if the evaluation metric on the validation data no longer increases, reduce the learning rate to 1 / 5 of the original value and save the optimal model from the training process as the training output.

[0094] Step 4: Accuracy Testing. Input the pre-processed test set into the trained optimal model, compare the true values ​​in the dataset with the predicted results, and calculate various evaluation metrics. This includes the following sub-steps:

[0095] Step 4.1: Input the preprocessed test set into the optimal model obtained through training;

[0096] Step 4.2: Compare the true values ​​of the dataset with the predicted results and calculate various evaluation indicators.

[0097] This embodiment presents experimental data on the dataset used for training the prediction network model. The technical solution of this invention is compared with traditional deep learning classification methods (ResNet-50, ResNet-101, ResNet-152, VGG16, VGG19) and recently published state-of-the-art methods (ADD-GCN, C-Tran, Q2L, TSFormer, IDA, and MS-GoGo). The performance metrics used for comparison are mean precision (mAP), subset accuracy (Subacc), class precision (CP), class recall (CR), class F1 score (CF1), overall precision (OP), overall recall (OR), and overall F1 score (OF1). The comparison results are shown in Table 1.

[0098] Table 1

[0099]

[0100] As can be seen from the comparison results in Table 1, the method of this application is superior to all previous methods in almost all indicators.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A multispectral multi-label classification method based on low confusion and spatial spectral self-balancing, characterized in that, The method comprises the following steps: Step one, build a weather multispectral multi-label classification network framework; Step two, data set preprocessing: slice and normalize the multispectral data; divide the training, verification and test sets according to the set proportion; label the labels; Step three, network training, the process is as follows: initialize the model parameters with random weights, when the evaluation index mAP of the verification set no longer rises, the learning rate will be reduced to the set multiple of the original; set the initial learning rate of the Adam optimizer and the batch size in training, a total of complete training for a set number of rounds, and set the loss weight of the cross-correlation limit, train the network model to convergence, and save the optimal model in the training process as the output of the training; Step four, precision test, input the preprocessed test set into the optimal model obtained by training, compare the true value of the data set with the predicted result, and calculate various evaluation indexes; In step one, first, a spatial feature extraction network and a spectral feature extraction network are built; then, a query generation module and a three-branch structure are built, the three-branch structure includes a spatial branch, a spectral branch and a comprehensive branch, the labeled features obtained by the self-attention layer in each branch are spliced with the label query vectors obtained by the query generation module, and then the output vectors of each branch are obtained after passing through a normalization layer and another self-attention layer; finally, a branch balance voting mechanism is established, the output vectors of the spatial branch, the spectral branch and the comprehensive branch are accepted as inputs, and then a set of learnable parameters used for voting are used as a reference for the weight to make a weighted average of the predictions among the branches to obtain the final prediction result; Step one comprises the following substeps: Step 1.1, build a spatial feature extraction network: input the weather multispectral image into the VGG network to preliminarily extract spatial features; Step 1.2, build a spectral feature extraction network: the spectral feature extraction network comprises at least four spectral convolution blocks, each spectral convolution block comprises two data streams, a characteristic feature stream and a global feature stream, the characteristic feature stream is responsible for extracting special features in each channel, and the global feature stream is responsible for collecting characteristic features at each stage and fusing with global features; Step 1.3, build a query generation module: in the query generation module, first, a set of label embeddings are obtained through a label embedding layer, then the set of label embeddings are sent into a multilayer perceptron for further processing to obtain label embedding representations, and then a normalization layer is used to standardize the mapped label embedding representations to obtain label query vectors; Step 1.4, a three-branch structure is built, including a spatial branch, a spectral branch, and a comprehensive branch, the spatial branch and the spectral branch are used to accept the spatial features or spectral representations obtained in steps 1.1 and 1.2 as inputs, and the input of the comprehensive branch is composed of the splicing of the spatial features after a linear layer and the spectral representations after a linear layer, each branch is first subjected to a normalization layer, then the features are input into a Transformer self-attention layer for self-attention calculation, and the labeled features and attention weights are returned, and the labeled features are also spliced with the label query vector obtained in step 1.3, then subjected to a normalization layer and another Transformer self-attention layer, and finally the output vector of each branch is obtained.

2. The low confusion and spatial spectral self-balanced based multispectral multi-label classification method of claim 1, wherein, The step one further comprises the following sub-steps: Step 1.5, a branch balance voting mechanism is established, which accepts the prediction results of the spatial branch, the spectral branch, and the comprehensive branch as inputs, and then performs weighted averaging on the predictions of each branch with a set of learnable parameters for voting as a reference weight to obtain the final prediction result.

3. The low confusion and spatial spectral self-balanced based multispectral multi-label classification method of claim 1, wherein, In the step 1.2, in the characteristic feature stream of each spectral convolution block, the characteristic features of the previous stage are first subjected to a group convolution with a convolution kernel size of 4x4, followed by at least four nonlinear layers connected by a residual structure, each nonlinear layer is composed of two group convolutions and a LeakyReLU activation function in the middle.

4. The low confusion and spatial spectral self-balanced based multispectral multi-label classification method of claim 1, wherein, In the step 1.2, in the global feature stream, the multispectral image data is first subjected to a two-dimensional convolution with a convolution kernel size of 1x1, and then enters the spectral convolution block; in the global feature stream of each spectral convolution block, the characteristic features of the previous stage are first fused with the global features of the previous stage, and then subjected to a two-dimensional convolution with a convolution kernel size of 4x4; in the last stage of the spectral feature extraction network, the output in the previous spectral convolution block in the global feature stream is mapped into a spectral representation as output using another two-dimensional convolution with a convolution kernel size of 4x4.

5. The low confusion and spatial-spectral self-balanced based multi-spectral multi-label classification method according to any one of claims 1-4, wherein, In the step two, the original multispectral data is processed into several slices according to the setting, the slices are normalized by maximum and minimum value, and then scaled to a set size; the training, validation, and test sets are divided according to the set proportion; the data labels are manually or automatically labeled according to the set requirements.

6. The low confusion and spatial spectral self-balanced based multi-spectral multi-label classification method according to any one of claims 1-4, wherein, The step three comprises the following sub-steps: Step 3.1, initializing the model parameters with random weights; Step 3.2, using cross-correlation constraints to limit the correlation degree of different query vectors, the cross-correlation constraint calculation method comprises: First, the inner product between the label query vectors is calculated to obtain the autocorrelation matrix, then the unit matrix is created, where the diagonal elements of the unit matrix are 1 and the non-diagonal elements are 0, then the length of each label embedding vector is calculated by squaring, summing and square rooting the elements of each label embedding vector, and the product of the length is used to normalize the autocorrelation matrix, the absolute value difference between the normalized autocorrelation matrix and the unit matrix is calculated as the cross-correlation constraint loss term; Step 3.3, use binary cross-entropy as the original loss function; Step 3.4, set the initial learning rate of the Adam optimizer; Step 3.5, optimize the model parameters.

7. The low confusion and spatial-spectral self-balanced based multispectral multi-label classification method of claim 6, wherein, The process of step 3.5 is: Step 3.5.1, input the training data and validation data slices into the network to obtain the final output probability of each category through the network built; Step 3.5.2, for the training data, use the loss function in 3.3 to measure the error between the true value and the predicted value of the training set, and also as the original loss function of each branch in step 1.6, after that, use the branch balance voting mechanism in step 1.6 to reweight each branch and punish weak branches, then add the loss value of each branch after reweighting to the loss between the true value and the final output of the model obtained in step 3.2, to get the final model loss value, use the optimizer in step 3.3 to optimize the loss, then use the validation data to calculate the classification evaluation index; Step 3.5.3, feedback the error generated by the training slice to the network to adjust the parameters of the network and optimize the prediction performance; Step 3.5.4, repeat 3.5.1-3.5.3 for a certain number of times, during the repetition, if the evaluation index on the validation data no longer rises, reduce the learning rate to the original set proportion, save the optimal model in the training process as the output of the training.

Images