Remote sensing scene classification method and device based on global and local multi-instance comparison learning
The remote sensing scene classification method, which utilizes global and local multi-example comparison learning, uses global and local feature extraction modules to filter out positive and negative features, and combines them with a loss function to train the network. This solves the problem of poor remote sensing scene classification performance and improves classification accuracy and model generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUCHANG UNIV OF TECH
- Filing Date
- 2026-01-26
- Publication Date
- 2026-06-02
Smart Images
Figure CN122135067A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and more particularly to a remote sensing scene classification method and device based on global and local multi-instance contrastive learning. BACKGROUND
[0002] Remote sensing scene classification technology is a rapidly developing technology in the field of remote sensing processing in recent years. It takes complex and variable remote sensing scene images as input, extracts deep feature information through traditional machine learning or deep learning network model, and classifies different types of scenes quickly to improve the theoretical and application level of automatic interpretation of remote sensing features.
[0003] Traditional machine learning methods use low-level operators such as color, texture, shape, space, and spectral attributes, and more traditional methods use intermediate features such as bag-of-words and enhanced scale-invariant feature transform. However, they still have a large gap with semantic features. With the rapid development of deep learning, deep network models such as dense network, residual network, squeeze-and-excitation network, and visual transformer have been widely studied and applied. The dense network connects each layer directly to all subsequent layers through dense connection, which promotes feature reuse and gradient propagation, improves network performance and training efficiency, but the network occupies high memory and has more hyperparameters. The residual network introduces residual modules and skip connections, which allow information and gradients to flow smoothly between different layers, alleviating the gradient vanishing problem. The squeeze-and-excitation network consists of a squeeze module and an excitation module. The former compresses the two-dimensional spatial information of each feature channel into a one-dimensional channel weight through global average pooling to capture the global information between channels. The excitation module generates a channel weight through a fully connected layer and an activation function, and then weights the feature channels to model the dependency between channels and improve network performance. The visual transformer divides the image into small blocks, then processes these small blocks as a sequence, and captures the global dependency of the image through a self-attention mechanism, which needs to be balanced in terms of dataset size and visual attention model size.
[0004] Patent application CN115019123A discloses a self-distillation contrastive learning method for remote sensing image scene classification. It uses a long-range dependency capture backbone network module composed of a teacher network and a student network with identical structures. Global slice images and scale-aligned local slice images are fed into the student network for self-distillation contrastive learning, obtaining multiple image representations. This invention does not rely on negative samples and learns multi-scale features. Patent application CN117636080A discloses a scene classification method that identifies scene images by jointly using object localization features and image color features. It utilizes a Swin-Transformer network to extract features and calculate the cosine similarity between feature vectors, and finally predicts the scene category using a DS evidence fusion model. However, this network is too fragmented and not trained end-to-end, and the image set used is not a remote sensing scene dataset.
[0005] Therefore, it can be seen that the existing methods still have certain limitations and are not effective for image classification in remote sensing scenes. Summary of the Invention
[0006] In view of this, the present invention provides a remote sensing scene classification method and apparatus based on global and local multi-instance contrastive learning, which solves or at least partially solves the technical problem of poor classification performance in the prior art.
[0007] To address the aforementioned technical problems, the first aspect of this invention provides a remote sensing scene classification method based on global and local multi-instance contrastive learning, comprising: Acquire remote sensing scene image data and randomly divide it into training and test sets; A transformer network for global and local multi-instance learning is constructed. The transformer network includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module. The global feature extraction module uses a convolutional network to extract global features from the input image. The local feature extraction module uses a visual encoder to extract local block features from the input image. The feature selection module uses a multi-head attention mechanism to select discriminant block features as positive examples from the local block features and selects block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The classification module is used to obtain the classification prediction result of the input image based on the global features, positive examples, and negative examples. The transformer network for global and local multi-instance learning is trained using the training set. During the training process, positive examples are used to extract and optimize features for similarity loss, triplet comparison loss between positive and negative examples, and reconstruction loss, so as to obtain the optimal classification model. The optimal classification model is used to predict the images in the test set, and the prediction results of the test set images are obtained.
[0008] In one implementation, the local feature extraction module uses a visual encoder to extract local block features from the input image, including: Divide the input image into blocks; Image patches are aggregated using multi-head attention to obtain aggregated features; Local block features are obtained based on the aggregated features and the features after normalization and MLP transformation of the aggregated features.
[0009] In one implementation, selecting discriminant block features as positive example features from local block features includes: The weights of multi-head self-attention are accumulated along the layers to obtain the accumulated attention weights; Candidate local feature blocks are selected from each attention head in each layer based on the accumulated attention weights, and the index number of the candidate local feature blocks is calculated based on the accumulated attention weights. Discriminant block features are selected from each layer based on the index number of the candidate local feature blocks; The selected discriminant block features are stacked to obtain positive example features.
[0010] In one implementation, candidate local feature blocks are selected from each attention head in each layer based on the accumulated attention weights, and the index number of the candidate local feature blocks is calculated based on the accumulated attention weights, including: Based on the magnitude of the accumulated attention weights, the local feature blocks with the largest accumulated attention weights are selected from each layer as candidate local feature blocks; Calculate the index number of the candidate local feature block based on the accumulated attention weights:
[0011] in, This represents the accumulated attention weights from the h-th head in the (L-1)-th layer. This represents the index number of the local feature block selected from the h-th head of layer L-1. This function represents the parameters used to find the maximum value of a function or set. It is used to find the index number of the corresponding candidate local feature block when the attention weights accumulated after the (L-1)th layer of the h-th head reach a large value.
[0012] In one implementation, the selected discriminant block features are stacked to obtain positive example features, including:
[0013] Wherein, Concate(.) indicates that features are stacked along the channel direction. This represents the index number of the local feature block selected from the h-th head of layer L-1. Represents all block features of layer L-1. This is a characteristic of positive examples.
[0014] In one implementation, block features are selected as negative examples based on the distance between the remaining local block features and the discriminant block features, including: Calculate the cosine distance between the remaining local block features and the discriminant block features; Based on the calculated cosine distance, a preset number of blocks are selected from the remaining local block features as negative examples.
[0015] In one implementation, a transformer network for global and local multi-instance learning is trained using a training set. During training, positive examples are used to extract and optimize features for similarity loss, triplet contrast loss between positive and negative examples, and reconstruction loss, resulting in an optimal classification model, including: The global feature extraction module and the local feature extraction module extract global features and local block features from the training samples, respectively. The feature selection module is used to select discriminant block features from the local block features of the training samples as positive examples using a multi-head attention mechanism. Based on the distance between the remaining local block features and the discriminant block features, block features are selected as negative examples. The transformer network is trained using a combination of cross-entropy loss, positive pair similarity loss, triplet contrast loss, and reconstruction loss to obtain the optimal classification model. The total loss is:
[0016] Where L represents the total loss, Represents cross-entropy loss, This represents the loss for positive examples based on similarity. Indicates the comparison loss of the triplet. Indicates the reconstruction loss. These are the weighting coefficients for balancing the loss;
[0017]
[0018]
[0019]
[0020] These represent the number of training samples, the true class of the i-th training sample, and the predicted class, respectively. , These are the features of the first and second positive examples, respectively. express Norm, Let represent the global features of the j-th input image in a batch. Let represent the positive feature of the j-th input image in a batch. Let represent the negative example feature of the j-th input image in a batch. Let represent the training samples and the reconstructed image samples, respectively, and j represent the index of the input image. b Batch size, which is the total number of input images in a batch. This is an adjustable parameter.
[0021] Based on the same inventive concept, a second aspect of the present invention provides a remote sensing scene classification device for global and local multi-instance contrastive learning, comprising: The data acquisition module is used to acquire remote sensing scene image data and randomly divide it into training and test sets; The model building module is used to construct a transformer network for global and local multi-instance learning. The transformer network includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module. The global feature extraction module uses a convolutional network to extract global features from the input image. The local feature extraction module uses a visual encoder to extract local block features from the input image. The feature selection module uses a multi-head attention mechanism to select discriminant block features as positive examples from the local block features and selects block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The classification module is used to obtain the classification prediction result of the input image based on the global features, positive examples, and negative examples. The model training module is used to train the constructed global and local multi-instance learning transformer network using the training set. During the training process, positive examples are used to extract and optimize features for similarity loss, triplet comparison loss between positive and negative examples, and reconstruction loss to obtain the optimal classification model. The classification test module is used to predict the images in the test set using the optimal classification model, and obtain the prediction results for the test set images.
[0022] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, provides the remote sensing scene classification method of global and local multi-instance contrastive learning described in the first aspect.
[0023] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing scene classification method of global and local multi-instance contrastive learning described in the first aspect.
[0024] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: This invention provides a remote sensing scene classification method based on global and local multi-instance contrastive learning. It constructs a transformer network for global and local multi-instance learning. The global feature extraction module uses a convolutional network to extract global features from the input image, while the local feature extraction module uses a visual encoder to extract local block features. The feature selection module uses a multi-head attention mechanism to select discriminant block features as positive examples from the local block features and selects block features as negative examples based on the distance between the remaining local block features and the discriminant block features. By introducing the ViT block selection strategy and combining it with positive examples to extract and optimize features for similarity loss, triplet contrast loss between positive and negative examples, and reconstruction loss, multiple positive and negative example blocks can be accurately located, achieving good classification performance on remote sensing scene classification tasks and improving the generalization performance of the network model. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a flowchart of the remote sensing scene classification method based on global and local multi-instance comparison learning in an embodiment of the present invention; Figure 2 This is a framework diagram of network training and testing in an embodiment of the present invention; Figure 3 This is a schematic diagram of the comparison loss calculation process in an embodiment of the present invention. Detailed Implementation
[0027] The purpose of this invention is to address the technical problem of poor remote sensing scene classification performance caused by the inability of existing technologies to effectively locate the feature representations of positive and negative examples for complex objects in remote sensing scenes. This invention provides a remote sensing scene classification method based on global and local multi-example comparative learning, thereby improving the classification accuracy and generalization ability of the model.
[0028] To achieve the above-mentioned technical effects, the main concept of this invention is as follows: This paper proposes a remote sensing scene classification method based on global and local multi-example contrastive learning. First, global features are extracted. Then, a visual transformer is introduced to extract multi-head self-attention features (local block features) for each image patch, and multiple discriminant block features are selected as positive features. Based on the distance between the feature block features and the positive feature blocks, local block features are selected as negative features. Next, the similarity between positive pairs is calculated, and a triplet contrastive loss is introduced to calculate the distance between global features and positive and negative features. Then, the reconstruction loss between images is calculated. Finally, the entire network is trained end-to-end.
[0029] The remote sensing scene classification method based on global and local multi-example contrastive learning provided in this invention, by introducing a ViT block selection strategy and jointly extracting and optimizing features for similarity loss, triplet contrast loss between positive and negative examples, and reconstruction loss using positive examples, can accurately locate multiple positive and negative example blocks, achieving good classification performance on remote sensing scene classification tasks and improving the generalization performance of the network model. Example 1 This embodiment provides a remote sensing scene classification method based on global and local multi-instance contrastive learning. Please refer to [link to relevant documentation]. Figure 1 ,include: S1: Acquire remote sensing scene image data and randomly divide it into training and test sets.
[0030] In practice, remote sensing scene images can be obtained from existing image datasets or real-world scenarios. For example, remote sensing scene images can be obtained from public datasets such as AID and NWPU. The dataset partitioning ratio can be set according to the situation, for example, the training set can use 50% and 20% respectively, and the image size can be set to 224×224.
[0031] S2: Construct a transformer network for global and local multi-instance learning. The transformer network includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module. The global feature extraction module uses a convolutional network to extract global features from the input image. The local feature extraction module uses a visual encoder to extract local block features from the input image. The feature selection module is used to select discriminant block features as positive examples from the local block features and to select block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The classification module is used to obtain the classification prediction result of the input image based on the global features, positive examples, and negative examples.
[0032] Specifically, S2 is the network construction, which includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module.
[0033] In the specific implementation process, the global feature extraction module can use a lightweight convolutional network, such as a 2D convolutional layer, and set the output channels, convolutional filter size, stride and padding of the 2D convolutional layer; set activation layers and linear layers, and use the set 2D convolutional layer to extract global features from the input image.
[0034] For example, set the number of 2D convolution output channels to 64, the convolution filter size to 3×3, the stride to 2, and the padding to 1; set the activation layer to ReLU() and the linear layer to Linear(256×7×7, 768).
[0035] The local feature extraction module uses the visual encoder ViT. The process of extracting local block features includes: Divide the input image into blocks; Image patches are aggregated using multi-head attention to obtain aggregated features; Local block features are obtained based on the aggregated features and the features after normalization and MLP transformation of the aggregated features.
[0036] In the specific implementation process, the input image is divided into blocks, and the total number of image blocks after division is N. A block number is added as its sequence number in the original image. , These represent the height and width of the image, respectively, and P represents the size of the block. As an example, , .
[0037] Aggregating image patches using multi-head attention means using a transformer with a multi-head attention mechanism to aggregate the features of multiple image patches in each layer to obtain the aggregated features of that layer.
[0038] Local block features are obtained based on aggregated features and the features obtained after normalization and MLP transformation of aggregated features, specifically: (1) in, Indicates the first layer, The representation layer normalization function, MLP( ) represents the multilayer perceptron transformation. Indicates the first Layer aggregation characteristics, Indicates the first All block features of the layer, i.e., the first Local block features of the layer.
[0039] Discriminant block features are selected from local block features as positive examples, including: The weights of multi-head self-attention are accumulated along the layers to obtain the accumulated attention weights; Candidate local feature blocks are selected from each attention head in each layer based on the accumulated attention weights, and the index number of the candidate local feature blocks is calculated based on the accumulated attention weights. Discriminant block features are selected from each layer based on the index number of the candidate local feature blocks; The selected discriminant block features are stacked to obtain positive example features.
[0040] In the specific implementation process, the formula for calculating the accumulated attention weight is as follows: (2) in, L Indicates the number of layers in the ViT encoder, for example , This represents the attention weight of the h-th head obtained after multiplying through multiple layers. Indicates the first Layer from the first h Attention weight based on size.
[0041] Based on maximum attention from the first In each head of the layer, the feature block corresponding to the maximum accumulated attention weight is selected as a candidate local feature block, and its index number is calculated. The index number of the candidate local feature block corresponding to the maximum attention of the h-th head is denoted as... The calculation is as follows: (3) in, The accumulated attention weights from the h-th head in layer L-1. This represents the index number of the local feature block selected from the h-th head of layer L-1. A function representing the parameters when finding the maximum value of a function or set, used to find the index number of the corresponding candidate local feature block when the attention weights accumulated after the (L-1)th layer of the h-th head reach a large value; (4) Wherein, Concate(.) indicates that features are stacked along the channel direction. This represents the index number of the local feature block selected from the h-th head of layer L-1. Represents all block features of layer L-1. This indicates the characteristics of positive examples.
[0042] Based on the distance between the remaining local block features and the discriminant block features, block features are selected as negative examples, including: Calculate the cosine distance between the remaining local block features and the discriminant block features; Based on the calculated cosine distance, a preset number of blocks are selected from the remaining local block features as negative examples.
[0043] Specifically, the number of preset blocks can be set according to the actual situation.
[0044] S3: Train the global and local multi-instance learning transformer network using the training set. During the training process, use positive examples to extract and optimize features for similarity loss, triplet comparison loss between positive and negative examples, and reconstruction loss to obtain the optimal classification model.
[0045] Please see Figure 2 This is a framework diagram of network training and testing in an embodiment of the present invention.
[0046] S3 can be implemented in the following way: The global feature extraction module and the local feature extraction module are used to extract global features and local block features from the training samples, respectively. The feature selection module is used to select discriminant block features from the local block features of the training samples as positive examples, and select block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The transformer network is trained using a combination of cross-entropy loss, positive pair similarity loss, triplet contrast loss, and reconstruction loss to obtain the optimal classification model. The total loss is: (5) Where L represents the total loss, Represents cross-entropy loss, This represents the loss for positive examples based on similarity. The triplet comparison loss is calculated as follows: Figure 3 As shown; Indicates the reconstruction loss. These are the weighting coefficients for balancing the loss.
[0047] (6) (7) (8) (9) These represent the number of training samples, the true class of the i-th training sample, and the predicted class, respectively. , These are the features of the first and second positive examples, respectively. express Norm, Let represent the global features of the j-th input image in a batch. Let represent the positive feature of the j-th input image in a batch. Let represent the negative example feature of the j-th input image in a batch. Let represent the training samples and the reconstructed image samples, respectively, and j represent the index of the input image. b Batch size, which is the total number of input images in a batch. This is an adjustable parameter.
[0048] S4: Use the optimal classification model to predict the images in the test set and obtain the prediction results for the test set images.
[0049] Specifically, the test set is input into the trained optimal classification model to obtain prediction results, which are then interpreted and evaluated. S4 specifically includes: Step S4.1: Input the test set into the optimal classification model, extract global features using the global feature extraction module; extract local block features using the ViT encoder, and use the feature selection module to select discriminative block features (positive and negative features) using a multi-head attention mechanism. Step S4.2: Predict the class labels of the test set using the optimal classification model; Step S4.3: Interpret the prediction results and calculate the average precision and confusion matrix.
[0050] This invention provides a remote sensing scene classification method based on global and local multi-example contrastive learning. During network training, a global feature extraction module is used to extract global features; and a local feature extraction module is used to extract multi-head self-attention features (local block features) of each block from the input image using a visual encoder, and multiple discriminant blocks are selected. Next, the similarity between positive pairs is calculated, and a triplet contrastive loss is introduced to calculate the distance between global features and positive / negative feature pairs. Then, the reconstruction loss between images is calculated; finally, the entire network is trained end-to-end. This invention's method features stable training, high classification accuracy, and effective parameters.
[0051] Therefore, this invention develops a remote sensing scene classification method based on global and local multi-instance contrastive learning, which can accurately locate positive example blocks and locate difficult negative examples based on distance metrics. It effectively establishes a global and local multi-instance learning paradigm, improves the classification accuracy of the network model while ensuring parameter effectiveness, and enhances the generalization performance of the network model. It features stable training, high classification accuracy, and effective parameters.
[0052] To verify the effectiveness of the remote sensing scene classification method based on global and local multi-instance contrastive learning proposed in this invention, two different remote sensing scene classification datasets, AID-30 and NWPU-45, were selected for experiments. The category information of the datasets can be seen in Table 1.
[0053] Table 1: Scene Dataset Category Information
[0054] This invention uses two publicly available remote sensing scene datasets, AID-30 and NWPU-45, for experimental verification, with training set ratios of 50% and 20%, respectively. The algorithm in this example is analyzed and compared with other algorithms, and the results are shown in Table 2.
[0055] Table 2: Comparison of classification accuracy and parameters of different algorithms
[0056] The ResNet algorithm is referenced in the paper "Deep Residual Learning in Image Recognition" by He K et al. (IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016); the SCCov algorithm is referenced in the paper "Remote Sensing Scene Classification with Jump-Linked Covariance Networks" by He N et al. (IEEE International Journal of Neural Networks and Learning Systems, vol. 31, no.5, pp. 1461-1474, 2019); the MI-ALexNet algorithm is referenced in the paper "Deep Multi-Instance Convolutional Neural Networks for Learning Robust Representations of Remote Sensing Scenes" by Li Z et al. (IEEE International Journal of Geographical Sciences and Remote Sensing, vol. 58, no.5, pp. 3685-3702, 2020); and the paper "Effective Networks: Rethinking Model Scaling for Convolutional Neural Networks" by Tan M et al. (IEEE International Conference on Machine Learning, pp. 6105–6114, 2019). As shown in Table 2, in the convolutional neural network algorithm, the remote sensing scene classification method based on global and local multi-instance contrastive learning proposed in this invention has better classification accuracy and parameter efficiency. This further illustrates that establishing and training a remote sensing transformer network model based on global and local multi-instance contrastive learning can accurately locate positive example blocks and locate difficult negative examples based on distance metrics. This effectively establishes a global and local multi-instance learning paradigm, improves the classification accuracy of the network model while ensuring parameter efficiency, and also improves the generalization performance of the network model.
[0057] Example 2 Based on the same inventive concept, this embodiment discloses a remote sensing scene classification device for global and local multi-instance contrastive learning, including: The data acquisition module is used to acquire remote sensing scene image data and randomly divide it into training and test sets; The model building module is used to construct a transformer network for global and local multi-instance learning. The transformer network includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module. The global feature extraction module uses a convolutional network to extract global features from the input image. The local feature extraction module uses a visual encoder to extract local block features from the input image. The feature selection module uses a multi-head attention mechanism to select discriminant block features as positive examples from the local block features and selects block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The classification module is used to obtain the classification prediction result of the input image based on the global features, positive examples, and negative examples. The model training module is used to train the constructed global and local multi-instance learning transformer network using the training set. During the training process, positive examples are used to extract and optimize features for similarity loss, triplet comparison loss between positive and negative examples, and reconstruction loss to obtain the optimal classification model. The classification test module is used to predict the images in the test set using the optimal classification model, and obtain the prediction results for the test set images.
[0058] Since the apparatus in Embodiment 2 of this invention is the same apparatus used in the remote sensing scene classification method of global and local multi-instance comparison learning in Embodiment 1, those skilled in the art can understand the specific structure and variations of this apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.
[0059] Example 3 Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the remote sensing scene classification method of global and local multi-instance contrastive learning according to Embodiment 1.
[0060] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the remote sensing scene classification method of global and local multi-instance comparison learning in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.
[0061] Example 4 Based on the same inventive concept, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in Embodiment 1.
[0062] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the remote sensing scene classification method of global and local multi-instance comparison learning in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.
[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0065] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.
Claims
1. A remote sensing scene classification method based on global and local multi-instance contrastive learning, characterized in that, include: Acquire remote sensing scene image data and randomly divide it into training and test sets; A transformer network for global and local multi-instance learning is constructed. The transformer network includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module. The global feature extraction module uses a convolutional network to extract global features from the input image. The local feature extraction module uses a visual encoder to extract local block features from the input image. The feature selection module uses a multi-head attention mechanism to select discriminant block features as positive examples from the local block features and selects block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The classification module is used to obtain the classification prediction result of the input image based on the global features, positive examples, and negative examples. The transformer network for global and local multi-instance learning is trained using the training set. During the training process, positive examples are used to extract and optimize features for similarity loss, triplet comparison loss between positive and negative examples, and reconstruction loss, so as to obtain the optimal classification model. The optimal classification model is used to predict the images in the test set, and the prediction results of the test set images are obtained.
2. The remote sensing scene classification method based on global and local multi-instance contrastive learning as described in claim 1, characterized in that, The local feature extraction module uses a visual encoder to extract local block features from the input image, including: Divide the input image into blocks; Image patches are aggregated using multi-head attention to obtain aggregated features; Local block features are obtained based on the aggregated features and the features after normalization and MLP transformation of the aggregated features.
3. The remote sensing scene classification method based on global and local multi-instance contrastive learning as described in claim 2, characterized in that, Discriminant block features are selected from local block features as positive examples, including: The weights of multi-head self-attention are accumulated along the layers to obtain the accumulated attention weights; Candidate local feature blocks are selected from each attention head in each layer based on the accumulated attention weights, and the index number of the candidate local feature blocks is calculated based on the accumulated attention weights. Discriminant block features are selected from each layer based on the index number of the candidate local feature blocks; The selected discriminant block features are stacked to obtain positive example features.
4. The remote sensing scene classification method based on global and local multi-instance contrastive learning as described in claim 3, characterized in that, Candidate local feature blocks are selected from each attention head in each layer based on the accumulated attention weights. The index number of the candidate local feature block is calculated based on the accumulated attention weights, including: Based on the magnitude of the accumulated attention weights, the local feature blocks with the largest accumulated attention weights are selected from each layer as candidate local feature blocks; Calculate the index number of the candidate local feature block based on the accumulated attention weights: in, This represents the accumulated attention weights from the h-th head in the (L-1)-th layer. This represents the index number of the local feature block selected from the h-th head of layer L-1. This function represents the parameters used to find the maximum value of a function or set. It is used to find the index number of the corresponding candidate local feature block when the attention weights accumulated after the (L-1)th layer of the h-th head reach a large value.
5. The remote sensing scene classification method based on global and local multi-instance contrastive learning as described in claim 3, characterized in that, The selected discriminant block features are stacked to obtain positive example features, including: Wherein, Concate(.) indicates that features are stacked along the channel direction. This represents the index number of the local feature block selected from the h-th head of layer L-1. Represents all block features of layer L-1. This indicates the characteristics of positive examples.
6. The remote sensing scene classification method based on global and local multi-instance contrastive learning as described in claim 1, characterized in that, Based on the distance between the remaining local block features and the discriminant block features, block features are selected as negative examples, including: Calculate the cosine distance between the remaining local block features and the discriminant block features; Based on the calculated cosine distance, a preset number of blocks are selected from the remaining local block features as negative examples.
7. The remote sensing scene classification method based on global and local multi-instance contrastive learning as described in claim 1, characterized in that, The transformer network, which employs global and local multi-instance learning, is trained using the training set. During training, positive examples are used to extract and optimize features for similarity loss, triplet contrast loss between positive and negative examples, and reconstruction loss, resulting in the optimal classification model, including: The global feature extraction module and the local feature extraction module extract global features and local block features from the training samples, respectively. The feature selection module is used to select discriminant block features from the local block features of the training samples as positive examples using a multi-head attention mechanism. Based on the distance between the remaining local block features and the discriminant block features, block features are selected as negative examples. The transformer network is trained using a combination of cross-entropy loss, positive pair similarity loss, triplet contrast loss, and reconstruction loss to obtain the optimal classification model. The total loss is: Where L represents the total loss, Represents cross-entropy loss, This represents the loss for positive examples based on similarity. Indicates the comparison loss of the triplet. Indicates the reconstruction loss. These are the weighting coefficients for balancing the loss; These represent the number of training samples, the true class of the i-th training sample, and the predicted class, respectively. , These are the features of the first and second positive examples, respectively. express Norm, Let represent the global features of the j-th input image in a batch. Let represent the positive feature of the j-th input image in a batch. Let represent the negative example feature of the j-th input image in a batch. Let represent the training samples and the reconstructed image samples, respectively, and j represent the index of the input image. b The total number of input images in a batch. This is an adjustable parameter.
8. A remote sensing scene classification device based on global and local multi-instance contrastive learning, characterized in that, include: The data acquisition module is used to acquire remote sensing scene image data and randomly divide it into training and test sets; The model building module is used to construct a transformer network for global and local multi-instance learning. The transformer network includes a global feature extraction module, a local feature extraction module, a feature selection module, and a classification module. The global feature extraction module uses a convolutional network to extract global features from the input image. The local feature extraction module uses a visual encoder to extract local block features from the input image. The feature selection module uses a multi-head attention mechanism to select discriminant block features as positive examples from the local block features and selects block features as negative examples based on the distance between the remaining local block features and the discriminant block features. The classification module is used to obtain the classification prediction result of the input image based on the global features, positive examples, and negative examples. The model training module is used to train the constructed global and local multi-instance learning transformer network using the training set. During the training process, positive examples are used to extract and optimize features for similarity loss, triplet comparison loss between positive and negative examples, and reconstruction loss to obtain the optimal classification model. The classification test module is used to predict the images in the test set using the optimal classification model, and obtain the prediction results for the test set images.
9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the remote sensing scene classification method based on global and local multi-instance contrastive learning as described in any one of claims 1 to 7.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the remote sensing scene classification method based on global and local multi-instance contrastive learning as described in any one of claims 1 to 7.