A remote sensing scene classification method based on a visual language large model and knowledge distillation
By constructing a large visual language model and a remote sensing scene classification method based on knowledge distillation, the problem of insufficient modeling of large-scale structures and long-distance semantic dependencies in remote sensing image classification is solved. This method achieves high-precision remote sensing scene classification using a lightweight network, which is suitable for remote sensing equipment with limited resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-09
AI Technical Summary
Existing remote sensing image classification methods are insufficient in capturing large-scale spatial structures and long-distance semantic dependencies. Traditional CNNs are difficult to model effectively, ViT models are computationally complex and difficult to deploy, and large-scale visual language models have a large number of parameters and feature alignment is difficult.
We construct a remote sensing scene classification method based on a large visual language model and knowledge distillation. Through a heterogeneous feature alignment mechanism and a cross-modal semantic guidance strategy, we utilize a visual knowledge distillation network and a semantic alignment module, combined with the ViT teacher model and the VGG student model, to achieve high-precision classification using a lightweight network.
Without increasing inference costs, it significantly improves the robustness of feature extraction and the generalization performance of the model, achieving high-precision classification of complex remote sensing scenes with a lightweight network, and is suitable for resource-constrained spaceborne or UAV platforms.
Smart Images
Figure CN122176383A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an improvement in remote sensing scene classification and recognition technology, specifically to a remote sensing scene classification method based on a large visual language model and knowledge distillation, applicable to intelligent interpretation and automated classification tasks of remote sensing images, and belonging to the fields of remote sensing image processing and computer vision technology. Background Technology
[0002] Remote sensing image scene classification aims to automatically identify the semantic category of an image based on its visual content. Early classification methods mainly relied on manually designed feature extraction operators, such as Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), and color and texture features, combined with Support Vector Machines (SVM) for classification. However, these methods depend on prior knowledge, have limited feature representation capabilities, and struggle to adapt to the complex intra- and inter-class differences in remote sensing images.
[0003] With the development of deep learning, Convolutional Neural Networks (CNNs) have become the mainstream feature extraction tool. Classic CNN architectures, such as VGG, deepen the network by stacking small convolutional kernels to extract rich texture features, while ResNet (Residual Network) solves the degradation problem of deep networks through residual connections. In the field of remote sensing, CNN-based methods have significantly improved classification accuracy by learning multi-level visual features. However, convolutional operations mainly focus on pixel correlations within local neighborhoods, making it difficult to effectively capture large-scale spatial structures and long-distance semantic dependencies in remote sensing images (such as narrow rivers or widely distributed residential areas), resulting in insufficient modeling ability for global contextual information.
[0004] To overcome the limitations of CNNs, the Vision Transformer (ViT) architecture has been introduced for remote sensing image classification. ViT segments images into sequential patches and uses a self-attention mechanism to directly model the global interactions of the image patch sequence. For example, improved models such as SCViT (Spatial-Contextual Vision Transformer) further enhance their ability to represent complex scenes globally by introducing a spatial-channel joint feature preservation mechanism. Although ViT performs excellently in capturing global information, its high computational complexity and slow inference speed make it difficult to deploy on edge devices such as spaceborne or drone-borne systems; and it lacks local inductive bias, making it difficult to capture fine-grained local texture features.
[0005] In recent years, large-scale visual-language models (VLMs) have provided a new technological paradigm for image classification. These models align the visual and textual feature spaces through contrastive learning on large-scale image-text pairs. In the field of remote sensing, methods such as RS-CLIP (Remote Sensing CLIP) attempt to use text encoders to transform scene labels into semantic vectors, guiding visual models to learn and achieving zero-shot or few-shot classification. However, large-scale visual-language models involve a huge number of parameters, and the inherent gap between visual and semantic modalities makes achieving efficient feature alignment between the two still a challenge. Summary of the Invention
[0006] To address the aforementioned shortcomings of existing technologies, the present invention aims to propose a remote sensing scene classification method based on a large visual language model and knowledge distillation. By constructing a heterogeneous feature alignment mechanism and a cross-modal semantic guidance strategy, the present invention not only solves the problem of structural differences between the ViT teacher model and the VGG student model, but also breaks through the limitations of a single visual modality, enabling lightweight networks to achieve high-precision and strong generalization classification of complex remote sensing scenes.
[0007] The technical solution of this invention is implemented as follows:
[0008] A remote sensing scene classification method based on a large visual language model and knowledge distillation, comprising the following steps:
[0009] 1) Training of remote sensing scene classification model;
[0010] 1.1) Pre-acquire remote sensing image datasets with different scene categories and known scene categories in advance; preprocess the remote sensing image datasets and divide the preprocessed remote sensing image datasets into training sets and test sets; use the text encoder of the pre-trained visual language large model to convert the scene category labels of the datasets into text semantic label vectors;
[0011] 1.2) Construct a training remote sensing scene classification network. The remote sensing scene classification network architecture includes a visual knowledge distillation network and a semantic alignment module. The visual knowledge distillation network includes a teacher model, a student model, and a knowledge distillation attention module set between the two. The student model includes multiple convolutional layers, a global average pooling layer, and a fully connected layer connected in sequence. The semantic alignment module includes a nonlinear projection module and the text semantic label vector obtained in step 1.1). The nonlinear projection module is set after the global average pooling layer of the student model and is used to receive the feature vector after global average pooling of the student model.
[0012] 1.3) Input the remote sensing images corresponding to the training set into the remote sensing scene classification network. While keeping the teacher model parameters unchanged, perform dual optimization training: In the visual dimension, use the knowledge distillation attention module to align student features with teacher features, and minimize the visual distillation loss to enable the student model to learn the spatial attention pattern of the teacher model; In the semantic dimension, use the nonlinear projection module to map the global visual features of the student model to the target semantic space, and minimize the semantic alignment loss to ensure that the mapped features of the student model are consistent with the corresponding text semantic label vector; At the same time, combine the classification loss corresponding to the classification result output by the fully connected layer of the student model to jointly update the parameters of the student model and the nonlinear projection module until the set iteration conditions are met, the training ends, and the trained student model is the remote sensing scene classification model.
[0013] 2) In actual remote sensing image scene classification, the remote sensing image to be classified is preprocessed in the same way as in step 1.1) and then input into the trained remote sensing scene classification model. The remote sensing scene classification model then outputs the scene classification and recognition result of the remote sensing image.
[0014] Further, in step 1.1), the preprocessing includes uniform scaling of the remote sensing image size, random cropping and flipping, and pixel value standardization.
[0015] Further, in step 1.1), the conversion process of the text semantic label vector is as follows: the scene category label of each remote sensing image is expanded into a natural language description using the prompt template and input into the text encoder of the pre-trained visual language large model, and a fixed-dimensional feature vector is extracted and normalized as the text semantic label vector.
[0016] Further, in step 1.2), the knowledge distillation attention module is set between the corresponding levels of the teacher model and the student model, with each level sharing one knowledge distillation attention module. The knowledge distillation attention module includes a feature projection unit and an attention reconstruction unit. The feature projection unit uses a 1×1 convolutional layer to map the number of student feature channels to be consistent with the number of teacher feature channels. The attention reconstruction unit first normalizes the teacher features and the projected student features, then calculates the affinity matrix between the two and normalizes it to obtain the attention weight matrix. Finally, the attention weight matrix is used to linearly weight the projected student features to generate fitted teacher features for comparative learning with the real teacher features.
[0017] Further, in step 1.2), the nonlinear projection module consists of a first fully connected layer, a ReLU activation function layer, and a second fully connected layer connected in sequence, used to map the visual features extracted by the student model to the same semantic feature space as the text semantic label vector.
[0018] Further, in step 1.3), the visual distillation loss uses mean squared error as the loss function to calculate the Euclidean distance between the fitted teacher features generated by the student model through the knowledge distillation attention module and the real teacher features; the specific formula is as follows:
[0019] ;
[0020] in, The length of the feature sequence. For feature dimension, The true feature vector extracted for the teacher model. To generate a fitted feature vector using student characteristics weighted by their features; This represents the square of the L2 norm.
[0021] Further, in step 1.3), the semantic alignment loss adopts a contrastive cross-entropy loss function based on temperature coefficient scaling, which is used to maximize the cosine similarity between the mapped student features and the corresponding real category text semantic label vectors. Its mathematical formula is:
[0022]
[0023] in, Represents the L2-normalized semantic features of students. With the Text semantic tags of each category Cosine similarity between them; For indicator functions, if category If it is a real category, then ,otherwise ; This represents the total number of scene categories. This is the temperature coefficient.
[0024] Further, in step 1.3), the classification loss uses the cross-entropy loss function to measure the difference between the class probability distribution predicted by the student model and the true class label. Its mathematical formula is:
[0025]
[0026] in, This represents the total number of scene categories. For indicator functions, if category If it is a real category, then ,otherwise ; and These are the unnormalized predicted Logit values output by the fully connected layers of the student model, corresponding to the... The and the first One category.
[0027] Furthermore, in step 1.3), the Adam optimizer is used to optimize and update the parameters in the student model and the nonlinear projection module. Through continuous iterative training, the total loss function is minimized. At the same time, the learning rate scheduler is used to dynamically adjust the learning rate, and the learning rate is reduced proportionally after a set number of iterations to improve the convergence stability and generalization ability of the model.
[0028] Furthermore, in step 1.2), the student model adopts a truncated VGG16 network structure; the teacher model adopts a ViT-Base model pre-trained on the ImageNet dataset.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] 1. This invention constructs a heterogeneous knowledge distillation architecture combining a ViT teacher model and a VGG student model. While retaining the advantages of lightweight and fast inference speed of convolutional neural networks, it effectively integrates the global context modeling capabilities of ViT. The knowledge distillation attention module designed in this invention delves into the intermediate layers of the model, extracts intermediate process quantities for alignment, and accurately transfers the spatial attention pattern of the teacher model to the student model through feature projection and attention reconstruction mechanisms. This significantly improves the robustness of feature extraction without increasing inference costs.
[0031] 2. This invention utilizes a large visual language model to achieve cross-modal semantic guidance, overcoming the limitations of traditional "closed-set distillation." Through a semantic alignment module and nonlinear projection technology, this invention injects rich textual semantic knowledge from the CLIP model into the student model. This dual supervision mechanism of "visual + semantic" enables the model to no longer merely mechanically memorize image textures, but truly understand the semantic connotations of the scene, thereby significantly enhancing the model's recognition ability in complex scenes and its generalization performance to new scenes.
[0032] 3. This invention employs a dual training strategy that jointly optimizes visual, semantic, and classification loss, achieving a perfect balance between model performance and efficiency. By jointly optimizing visual distillation loss, semantic alignment loss, and classification loss, the model achieves classification accuracy approaching that of large-scale ViT models while maintaining the advantages of VGG networks such as lightweight design, fast inference speed, and ease of deployment. This design is particularly suitable for monitoring tasks in resource-constrained environments such as spaceborne platforms and UAVs. Attached Figure Description
[0033] Figure 1 This is a flowchart of the remote sensing scene classification method of the present invention.
[0034] Figure 2This is a diagram of the remote sensing scene classification network architecture used for training in this invention.
[0035] Figure 3 This is a graph showing the change in the loss function trained on the UCM dataset according to the present invention.
[0036] Figure 4 This is a confusion matrix diagram of the present invention in the UCM dataset.
[0037] Figure 5 This is a diagram showing the validation results of the present invention on the UCM dataset.
[0038] Figure 6 This is a graph showing the change in the loss function during training on the OPTIMAL-31 dataset.
[0039] Figure 7 This is a confusion matrix diagram of the present invention in the OPTIMAL-31 dataset.
[0040] Figure 8 This is a diagram showing the validation results of the present invention on the OPTIMAL-31 dataset. Detailed Implementation
[0041] This invention proposes a remote sensing scene classification method based on a large visual language model (CLIP) and knowledge distillation. The inventive concept involves introducing the cross-modal semantic prior of the large visual language model (CLIP) into a heterogeneous knowledge distillation framework. A knowledge distillation attention module is constructed to address the feature adaptation challenge between the ViT teacher model and the VGG student model. Simultaneously, a semantic alignment module enhances the student model's semantic understanding of remote sensing scenes, thereby achieving high-precision and lightweight remote sensing scene classification. The core of this invention lies in constructing and training a remote sensing scene classification model based on the above inventive concept. After the model is trained, inputting the actual remote sensing image to be identified into the model yields the classification result.
[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0043] See Figure 1 This invention discloses a remote sensing scene classification method based on a large visual language model and knowledge distillation, the main steps of which are as follows:
[0044] 1) Training of remote sensing scene classification model;
[0045] 1.1) Data Acquisition and Text Semantic Label Construction. A dataset of remote sensing images with different scene categories, all of which are known, is pre-acquired and divided into training and testing sets. The acquired remote sensing images are preprocessed, including: uniform image scaling, random cropping and flipping, and pixel value standardization. Simultaneously, text semantic label vectors are constructed. Specifically, each scene category label in the dataset is expanded into a natural language description using a prompt template (e.g., "A satellite image of {category}"). These descriptions are then input into the text encoder of a pre-trained visual language large model (CLIP - Contrastive Language-Image Pre-Training) to extract fixed-dimensional text feature vectors. Finally, the extracted vectors are L2-norm normalized to obtain the text semantic label vector for that category.
[0046] 1.2) Constructing a large-scale visual language model and a remote sensing scene classification network based on knowledge distillation. (Reference) Figure 2The remote sensing scene classification network architecture comprises two parts: a visual knowledge distillation network and a semantic alignment module. In the visual knowledge distillation network, a teacher model and a student model are constructed. The teacher model uses a ViT-Base model pre-trained on the ImageNet dataset, which includes a patch embedding layer, a positional encoding layer, and 12 Transformer encoder layers. To extract multi-level global features, the embodiment extracts intermediate feature maps from the 3rd, 6th, 9th, and 12th Transformer encoder layers as the target features for distillation. The student model includes multiple convolutional layers, a global average pooling layer, and a fully connected layer connected in sequence. In this embodiment, the student model uses a truncated VGG16 network structure as the backbone network, retaining the first four stages of its feature extraction part and removing the original fully connected layers. Each stage contains several convolutional layers, a ReLU activation function, and a max-pooling layer. The student model extracts local texture features of the image through convolutional operations and outputs feature maps of different scales at the corresponding layers. To compensate for the student model's deficiency in global context modeling capabilities, a knowledge distillation attention module (SAB) is set between corresponding layers of the teacher and student models. This SAB includes a feature projection unit and an attention reconstruction unit. The semantic alignment module mainly comprises a multi-layer perceptron projector (MLP projector) connected to the end of the student model. This MLP projector consists of a first fully connected layer, a ReLU (Rectified Linear Unit) activation function layer, and a second fully connected layer connected sequentially. This module receives the feature vector after global average pooling from the student model, aiming to map the visual features extracted by the student to a shared semantic space consistent with the text semantic label vector in step 1).
[0047] 1.3) Model Training Phase. The remote sensing images corresponding to the training set are input into the remote sensing scene classification network. While keeping the teacher model parameters frozen (not participating in gradient updates), the parameters of the student model and the nonlinear projection module are iteratively updated using a multi-task joint loss function. Upon completion of training, the trained student model becomes the remote sensing scene classification model. The specific training process includes optimization in the following three dimensions:
[0048] First Dimension: Visual Feature Distillation Based on the SAB Module. To enable the student model to learn the spatial attention pattern of the teacher model, a visual distillation loss is calculated. The feature projection unit uses a 1×1 convolutional layer to map the number of channels in the student feature map to match the number of channels in the teacher feature map. To prevent numerical overflow and enhance gradient stability, the projected student and teacher features are normalized using the attention reconstruction unit using the L2 norm. Subsequently, the affinity matrix between the teacher and student features is calculated, and a temperature coefficient is introduced to adjust the sharpness of the attention distribution. The attention weights are then normalized using the Softmax function. Finally, the projected student features are reconstructed using these attention weights (linear weighted combination) to generate fitted teacher features. The mean squared error (MSE) between the reconstructed features and the true teacher features is calculated as the visual distillation loss. Its mathematical expression is:
[0049] ;
[0050] in, The length of the feature sequence. For feature dimension, The real feature vector extracted from the teacher network. To generate a fitted feature vector using student characteristics weighted by their features; This represents the square of the L2 norm.
[0051] The second dimension: Cross-modal constraints based on the semantic alignment module. To endow the student model with the ability to understand scene semantics, a semantic alignment loss is calculated. The visual features output by the student model are mapped through a nonlinear projection module (containing a "linear layer-ReLU-linear layer" structure) to obtain the student's semantic features. .Will Compared with the text semantic label vectors of all categories pre-computed in step 1), Matching is performed. A contrastive loss based on temperature coefficient scaling is used to maximize the cosine similarity between student features and the text labels corresponding to the true categories, thereby bridging the gap between visual features and text semantics in the feature space. Semantic alignment loss. The formula is as follows:
[0052]
[0053] in, Represents the L2-normalized semantic features of students. With the Text semantic tags of each category Cosine similarity between them; For indicator functions, if category If it is a real category, then ,otherwise ; This represents the total number of scene categories. This is the temperature coefficient.
[0054] The third dimension: Label prediction based on the classification head. The feature vector of the student model is directly input into the fully connected classification layer, and the output is the predicted class probability. The classification loss is calculated using the standard cross-entropy loss function. This is used to measure the difference between the class probability distribution predicted by the student model and the true class label, to ensure that the model has basic classification and discrimination ability. Its mathematical formula is:
[0055]
[0056] in, This represents the total number of scene categories. For indicator functions, if category If it is a real category, then ,otherwise . and These are the unnormalized predicted Logit values output by the fully connected layers of the student model, corresponding to the... The and the first One category.
[0057] The total loss function is obtained by weighted summation of the above three parts:
[0058]
[0059] in, , , These are the weight coefficients for classification loss, visual distillation loss, and semantic alignment loss, respectively. In this implementation, to enhance the effect of semantic alignment and prevent classification loss from dominating the gradient, the weights are appropriately increased. The weight, and reduce The weights are determined. The Adam (Adaptive Moment Estimation) optimizer is used to update the parameters in the student model and the nonlinear projection module. Through iterative training, the total loss function is minimized. In addition, a learning rate scheduler is used to dynamically adjust the learning rate during training. When the validation set loss no longer decreases, the learning rate is decayed to promote the model to converge to a better local minimum.
[0060] 2) Online Detection and Application Stage. High-resolution cameras or remote sensing satellites are used to acquire remote sensing images to be detected. The same preprocessing operations as in step 1.1) are performed on the images to be detected. The preprocessed remote sensing images are then input into a trained scene classification student model. The student model undergoes forward propagation, outputting probability scores for each scene category through fully connected layers. The category with the highest probability is selected as the final scene classification result.
[0061] The remote sensing images used for training and testing in this invention belong to only one scene category. In actual testing, the model ultimately only provides one scene category for the image to be identified.
[0062] As can be seen from the above introduction, this invention has made improvements and innovations in the following aspects to improve the accuracy of the model in classifying remote sensing image scenes:
[0063] 1. In terms of network architecture design and visual feature distillation, a heterogeneous knowledge distillation framework based on multi-level intermediate feature alignment is constructed, extending the depth of feature alignment down to the hidden layers of the network, significantly improving feature utilization efficiency. This invention establishes multiple sets of one-to-one feature mapping relationships between the Transformer encoder of the ViT teacher model and the convolutional stage of the VGG student model, and utilizes a knowledge distillation attention module to perform fine-grained feature alignment at these intermediate levels. This design enables the student model not only to mimic the final classification decision of the teacher model, but also to fully mimic the teacher model's feature extraction and reasoning process from shallow texture to deep semantics. By mining and transmitting the rich semantic information of the intermediate layers, the problem of feature space mismatch between heterogeneous models is effectively solved, achieving deep transfer and efficient utilization of the teacher model's global context awareness capabilities.
[0064] 2. In terms of cross-modal semantic fusion and alignment, this invention utilizes the CLIP (Visual Language Processing) large-scale model to introduce textual semantic priors, breaking the limitation of traditional single-modal classification that relies solely on visual features. This invention uses the CLIP text encoder to convert scene labels into high-dimensional textual semantic label vectors, and designs a non-linear projection module at the end of the student model to map the extracted visual features to a semantic space consistent with the text labels. By minimizing the semantic alignment loss, the model is forced to learn the deep association between image content and natural language description. This dual supervision mechanism of "visual + semantic" not only enhances the model's understanding of the semantic connotation of the scene, but also significantly improves the model's generalization performance and discrimination accuracy when facing scenarios with high inter-class similarity or scarce data by introducing an external knowledge base.
[0065] To verify the effectiveness of this application, the inventors conducted the following verification experiments:
[0066] The hardware environment for this experiment is as follows: Intel(R) Core(TM) 7th generation processor, NVIDIA Corporation GP102 [TITAN X] GPU with 12GB GDDR5X dedicated video memory; the software environment is as follows: Python 3.8.20 interpreter, PyTorch 2.4.1 deep learning framework, and CUDA version Cuda_12.1.
[0067] The method of this invention was tested using the UCM_Land_Use (UCM) and OPTIAML-31 remote sensing scene image datasets. The UCM dataset is a widely used benchmark dataset in the field of remote sensing image scene classification, containing 2100 remote sensing images divided into 21 scene categories, with 100 images in each category and a spatial resolution of 1 foot. The validation scheme was set as follows: 80% of the dataset was used as the training set, and 20% as the test set. An initial learning rate of 0.0001 was set, and the Adam optimizer and cosine annealing scheduling algorithm were used to iteratively update the model parameters to ensure stable convergence within 100 epochs. The loss function change curve during model training is shown in the figure below. Figure 3 As shown.
[0068] To further verify the discriminative ability and class distinction stability of the method of this invention in high-resolution remote sensing scene classification tasks, the normalized confusion matrix results on the UCM dataset are presented, as follows: Figure 4 As shown, the model achieved near-ideal recognition results in the vast majority of categories. The predicted probabilities on the main diagonal remained between 0.95 and 1.00, including categories such as agricultural land, baseball diamond, and beach, all of which achieved a recognition probability of 1.00. This indicates that the method of this invention can fully exploit the structural differences and semantic features between different remote sensing scenes to achieve high-precision discrimination.
[0069] While the method proposed in this invention achieves high accuracy in identifying most categories, slight confusion exists between a few. For example, the correct identification rate for categories such as airplane, intersection, mobile home park, overpass, sparse residential area, and tennis court is 0.95; the dense residential area category shows approximately 0.05 misclassifications to buildings, medium residential area, sparse residential area, and storage tanks. This slight confusion primarily stems from the similarity in spatial layout, feature composition, and texture structure among some categories. For example, dense residential areas, medium residential areas, and sparse residential areas, along with buildings, all contain dense building and road networks, and their scale distribution and texture patterns overlap to some extent. Storage tanks and dense residential areas also share similarities in local geometric structure and grayscale distribution features, making them prone to neighboring distributions in the feature representation space. However, even in categories where confusion exists, the method of this invention maintains a recognition probability of no less than 0.80, with the overall error distribution being relatively concentrated and the magnitude small.
[0070] In summary, the confusion matrix of the proposed method on the UCM dataset exhibits a highly concentrated diagonal distribution, with low and dispersed values for off-diagonal elements. This indicates that the model can effectively learn discriminative representations of different scene categories and significantly reduce the interference caused by feature overlap between categories. This result further validates the stability and generalization ability of the proposed feature fusion and representation learning mechanism in complex remote sensing scenes.
[0071] The training accuracy, overall validation accuracy, and average validation accuracy of the method of this invention on the UCM dataset are as follows: Figure 5 As shown in Table 1, the experimental results demonstrate that the method of this invention (ViT-VGG-CLIP) achieves an overall accuracy (OA) of 96.67% on the validation set, outperforming several existing comparative algorithms. This indicates that the measurement accuracy of this invention meets the standards for practical remote sensing image recognition applications.
[0072] Table 1. Comparison of the method of this invention with other methods on the UCM dataset.
[0073]
[0074] Based on this, comparative experiments were conducted on the OPTIMAL-31 remote sensing image dataset. The OPTIMAL-31 dataset contains 31 typical remote sensing scene categories, with 60 images of 256×256 pixels per category, totaling 1860 images. Compared to the UCM dataset, this dataset has more refined categories and fewer samples per category, placing higher demands on the model's feature extraction and overfitting prevention capabilities. The validation scheme used 80% of the dataset as the training set and 20% as the test set (372 images). An initial learning rate of 0.0001 was set, and the Adam optimizer and cosine annealing scheduling algorithm were used to iteratively update the model parameters, ensuring stable convergence within 100 epochs. The loss function variation curve during model training is shown in the figure below. Figure 6 As shown.
[0075] To further verify the discriminative ability and class differentiation stability of the method of this invention in high-resolution remote sensing scene classification tasks, the normalized confusion matrix results on the OPTIMAL-31 dataset are presented, as follows: Figure 7 As shown, although the number of categories in OPTIMAL-31 has increased from 21 in UCM to 31, and the semantic overlap between categories is more severe, the model of this invention still achieves ideal recognition results in the vast majority of categories. The prediction probabilities on the main diagonal remain at a high level overall, with more than 10 categories, including airplane, basketball court, and bridge, achieving a recognition probability of 1.00. This indicates that the method of this invention effectively extracts the global structure and key semantic features of the scene, and can achieve accurate discrimination of typical features even under multi-class interference.
[0076] While the method proposed in this invention demonstrates extremely high classification accuracy in some categories, confusion still exists between some categories. The correct recognition rate for "church" and "industrial area" is only 0.5. The misclassification is not random but has an underlying feature logic: the confusion matrix shows that "church" has a 0.17 probability of being misclassified as "circular farmland". Analysis of the geometric features of the image reveals that because some church buildings adopt an arched dome design, their closed curve features are similar to the edge contour of circular farmland when viewed from above. Meanwhile, there was a 0.17% chance of misclassifying it as a commercial area. This is because churches widely use glass walls and roofs, which are composite building materials, and their shapes are similar to those of buildings in modern commercial areas, causing the model to make a recognition bias during feature extraction. On the other hand, industrial areas often contain large factories, office buildings, and employee dormitories. Their complex combination of functions makes them easy to confuse with categories such as church and dense residential. Essentially, this is because the regular shadows of a large number of factories and the textures in these categories tend to be consistent at a certain downsampling scale, causing spatial perception illusions in the model.
[0077] In other categories, the correct recognition rates for bridge, desert, freeway, island, mobile home park, and railway all ranged from 0.75 to 0.83. This is because while these scenes possess a certain degree of macroscopic recognizability, they exhibit strong correlations with adjacent categories in key semantic features: bridge and railway are easily misclassified as freeway, mainly because all three have long, linear topological structures, which easily lead to structural confusion in two-dimensional images lacking height dimension; desert is easily misclassified as mountain because the hue and texture features of both are highly similar in arid regions; freeway is easily misclassified as railway or runway because the apparent color and texture features of these linear transportation facilities are highly similar, making it difficult for the model to extract high-level semantics. The system accurately identifies subtle differences in road surface materials and functional attributes. Islands are easily misidentified as lakes because, from a local perspective, islands and lakes exhibit complementary symmetry in their spatial structure, making it difficult for the model to accurately determine the hierarchical relationship between land and water, leading to a "mirror image" error in the recognition results. Mobile home parks are easily misidentified as dense residential buildings because both exhibit high-density rectangular geometric arrangements and have similar building colors and spatial distribution patterns. Railways are easily misidentified as freeways because both appear as directional, elongated linear structures in the image, with very similar geometric outlines.
[0078] The training accuracy, overall validation accuracy, and average validation accuracy of the method of this invention on the OPTIMAL-31 dataset are as follows: Figure 8 As shown in Table 2, the experimental results demonstrate that the method of this invention (ViT-VGG-CLIP) achieves an overall accuracy (OA) of 87.90% on the validation set, outperforming several existing comparative algorithms. This indicates that the measurement accuracy of this invention meets the standards for practical remote sensing image recognition applications.
[0079] Table 2. Comparison of the method of this invention with other methods on the OPTIMAL-31 dataset.
[0080]
[0081] The remote sensing scene classification method proposed in this invention, based on a large visual language model and knowledge distillation, introduces the cross-modal semantic prior of the large visual language model CLIP into a heterogeneous knowledge distillation framework. By constructing a knowledge distillation attention module, it solves the feature adaptation problem between the ViT teacher model and the VGG student model, breaking the limitations of a single visual modality. At the same time, it uses a semantic alignment module to enhance the student model's semantic understanding of remote sensing scenes, thereby achieving high-precision and strong generalization classification of complex remote sensing scenes by a lightweight network.
[0082] Finally, it should be noted that the above examples of the present invention are merely illustrative and not intended to limit the implementation of the invention. Although the applicant has described the present invention in detail with reference to preferred embodiments, those skilled in the art can make other variations and modifications based on the above description. It is impossible to exhaustively list all possible implementations here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A remote sensing scene classification method based on a large visual language model and knowledge distillation, characterized in that, The steps are as follows: 1) Training of remote sensing scene classification model; 1.1) Pre-acquire remote sensing image datasets with different scene categories and known scene categories in advance; preprocess the remote sensing image datasets and divide the preprocessed remote sensing image datasets into training sets and test sets; use the text encoder of the pre-trained visual language large model to convert the scene category labels of the datasets into text semantic label vectors; 1.2) Construct a training remote sensing scene classification network. The remote sensing scene classification network architecture includes a visual knowledge distillation network and a semantic alignment module. The visual knowledge distillation network includes a teacher model, a student model, and a knowledge distillation attention module set between the two. The student model includes multiple convolutional layers, a global average pooling layer, and a fully connected layer connected in sequence. The semantic alignment module includes a nonlinear projection module and the text semantic label vector obtained in step 1.1). The nonlinear projection module is set after the global average pooling layer of the student model and is used to receive the feature vector after global average pooling of the student model. 1.3) Input the remote sensing images corresponding to the training set into the remote sensing scene classification network. While keeping the teacher model parameters unchanged, perform dual optimization training: In the visual dimension, use the knowledge distillation attention module to align student features with teacher features, and minimize the visual distillation loss to enable the student model to learn the spatial attention pattern of the teacher model; In the semantic dimension, use the nonlinear projection module to map the global visual features of the student model to the target semantic space, and minimize the semantic alignment loss to ensure that the mapped features of the student model are consistent with the corresponding text semantic label vector; At the same time, combine the classification loss corresponding to the classification result output by the fully connected layer of the student model to jointly update the parameters of the student model and the nonlinear projection module until the set iteration conditions are met, the training ends, and the trained student model is the remote sensing scene classification model. 2) In actual remote sensing image scene classification, the remote sensing image to be classified is preprocessed in the same way as in step 1.1) and then input into the trained remote sensing scene classification model. The remote sensing scene classification model then outputs the scene classification and recognition result of the remote sensing image.
2. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.1), the preprocessing includes uniform scaling of the remote sensing image size, random cropping and flipping, and pixel value standardization.
3. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.1), the conversion process of the text semantic label vector is as follows: the scene category label of each remote sensing image is expanded into a natural language description using the prompt template and input into the text encoder of the pre-trained visual language large model. The fixed-dimensional feature vector is extracted and normalized as the text semantic label vector.
4. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.2), the knowledge distillation attention module is set between the corresponding levels of the teacher model and the student model, and each level shares one knowledge distillation attention module; the knowledge distillation attention module includes a feature projection unit and an attention reconstruction unit, and the feature projection unit uses a 1×1 convolutional layer to map the number of student feature channels to be consistent with the number of teacher feature channels; The attention reconstruction unit first normalizes the teacher features and the projected student features, then calculates and normalizes the affinity matrix of the two to obtain the attention weight matrix; finally, it uses the attention weight matrix to perform a linear weighted combination of the projected student features to generate fitted teacher features, which are used for comparative learning with the real teacher features.
5. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.2), the nonlinear projection module consists of a first fully connected layer, a ReLU activation function layer, and a second fully connected layer connected in sequence, which is used to map the visual features extracted by the student model to the same semantic feature space as the text semantic label vector.
6. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.3), the visual distillation loss uses mean squared error as the loss function to calculate the Euclidean distance between the fitted teacher features generated by the student model through the knowledge distillation attention module and the real teacher features; the specific formula is as follows: ； in, The length of the feature sequence. For feature dimension, The true feature vector extracted for the teacher model. To generate a fitted feature vector using student characteristics weighted by their features; This represents the square of the L2 norm.
7. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.3), the semantic alignment loss employs a contrastive cross-entropy loss function scaled based on a temperature coefficient, used to maximize the cosine similarity between the mapped student features and the corresponding real-class text semantic label vectors. Its mathematical formula is: in, Represents the L2-normalized semantic features of students. With the Text semantic tags of each category Cosine similarity between them; For indicator functions, if category If it is a real category, then ,otherwise ; This represents the total number of scene categories. This is the temperature coefficient.
8. The remote sensing scene classification method based on a large visual language model and knowledge distillation as described in claim 1, characterized in that: In step 1.3), the classification loss uses the cross-entropy loss function to measure the difference between the class probability distribution predicted by the student model and the true class label. Its mathematical formula is: in, This represents the total number of scene categories. For indicator functions, if category If it is a real category, then ,otherwise ; and These are the unnormalized predicted Logit values output by the fully connected layers of the student model, corresponding to the... The and the first One category.
9. The remote sensing scene classification method based on a large visual language model and knowledge distillation according to claim 1, characterized in that: In step 1.3), the Adam optimizer is used to optimize and update the parameters in the student model and the nonlinear projection module. Through continuous iterative training, the total loss function is minimized. At the same time, the learning rate scheduler is used to dynamically adjust the learning rate. After a set number of iterations, the learning rate is reduced proportionally to improve the convergence stability and generalization ability of the model.
10. The remote sensing scene classification method based on a large visual language model and knowledge distillation according to claim 1, characterized in that: In step 1.2), the student model adopts a truncated VGG16 network structure; the teacher model adopts a ViT-Base model pre-trained on the ImageNet dataset.