Unified framework incremental learning remote sensing scene classification method and system

Abstract

The invention provides a unified framework incremental learning remote sensing scene classification method and system, and belongs to the technical field of image processing. Comprising the steps of unified database construction, unified backbone network, classification header network model construction, unified hyper-parameter construction, linear programming incremental learning supervised training normal form network training, classification prediction result output and unified model verification. According to the method, a unified remote sensing image classification model framework normal form can be constructed, the framework supports embedding of various models, so that the models are accurate, universal and easy to verify, various modal data are supported to serve as model input, and the universality of the framework is remarkably enhanced; and meanwhile, a supervised learning training normal form based on linear programming incremental learning is designed, so that the model can be subjected to continuous fully supervised training on a large-scale heterogeneous data set, and the generalization of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a unified framework incremental learning remote sensing scene classification method and system. Background Art

[0002] Most of the current research on remote sensing image classification models is limited to data training of specific data sets, which consumes a lot of time and resources and fails to achieve universality across different data sources. When faced with new remote sensing data sets, these deep models must be trained to adapt to new classes, so they cannot truly adapt to new classes. At the same time, these methods are developed and evaluated individually, and there is no general model framework that can be used to uniformly carry and compare remote sensing classification algorithms.

[0003] Jilin University disclosed a remote sensing image classification method based on hyperspectral modality in its patent application "A hyperspectral remote sensing image classification method and related equipment" (patent application number: 2023113853497). This method first obtains the hyperspectral remote sensing image to be classified, and performs dimensionality reduction processing on the hyperspectral remote sensing image to be classified through principal component analysis to obtain the reduced-dimensional image data; secondly, the reduced-dimensional image data is input into the attention and multi-scale dense network to extract the spatial-spectral features of the image to obtain the spatial-spectral feature map of the image; then the obtained spatial-spectral feature map is input into the fully connected layer, the three-dimensional feature map output by the convolution is converted into a one-dimensional feature vector, and the feature space calculated by the previous layer is mapped to the sample space; finally, the extracted one-dimensional feature vector is optimized using the Adam optimizer, and the optimized one-dimensional feature vector is classified using the Sofmax classifier to obtain the classification result; this method is only applicable to model training for specific data sets and cannot be generalized to other data sets.

[0004] Beijing Institute of Technology disclosed a remote sensing image classification method based on hyperspectral modality in its patent application "A hyperspectral image classification method based on full-spectral correlation learning network" (patent application number: 2023110735747). This method first constructs a three-dimensional data cube and its spectral grouping for each ground object target based on the attribute characteristics of rich spatial spectral (spatent spectrum) information, strong spectral correlation and multimodal characteristics in hyperspectral remote sensing images; secondly, the three-dimensional data spectral grouping is used as input data, and the group convolution and spatial spectral convolution long short-term memory networks are combined to design a full-spectral correlation adaptive learning module, and dynamically Learn the spectral correlation of hyperspectral images; then use the full-spectral correlation adaptive learning module as the basic structural unit to build a backbone feature extraction network to extract deep semantic features that enhance spectral information and retain the intrinsic geometric structure; then use the three-dimensional data cube as input data to build an asymmetric fusion module based on a gating structure to align and integrate deep semantic features and traditional manual features, while suppressing noise and abnormal data interference; finally, integrate the above two modules to build a full-spectral correlation learning network to achieve hyperspectral image classification; this method is trained and evaluated separately for a specific model, and the number of models that can be adapted for embedding is small.

[0005] It can be seen that the current research on remote sensing image classification models is mostly limited to data training of specific data sets, which consumes a lot of time and resources and fails to achieve universality across different data sources. When faced with new remote sensing data sets, these deep models must be trained to adapt to new classes and cannot truly adapt to new classes. Secondly, the existing neural network algorithms are all developed and evaluated separately for specific scenarios, and there is no general model framework that can be used to unify compatible embedding and compare remote sensing classification algorithms. Summary of the invention

[0006] The purpose of the present invention is to provide a unified framework incremental learning remote sensing scene classification method and system to overcome the problems existing in the prior art. The present invention can construct a unified remote sensing image classification model framework paradigm. The framework supports the embedding of multiple models, making the model accurate, universal and easy to verify, and supports various types of modal data as model input, which significantly enhances the versatility of the framework; at the same time, a supervised learning training paradigm based on linear programming incremental learning is designed, so that the model can perform continuous fully supervised training on large-scale heterogeneous data sets, thereby enhancing the generalization of the model.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A unified framework incremental learning remote sensing scene classification method, comprising the following steps: (1) Unified database construction: Unified database construction: Build a unified database containing several image datasets, where each image dataset consists of a training set, a validation set, and a test set, and then perform data enhancement on the unified database; (2) Unified backbone network and classification head network model construction: The backbone network and classification head network are combined into a unified model; (3) Unified hyperparameter construction: Unify the optimizer, learning rate, and loss function hyperparameters; (4) Network training using the linear programming incremental learning supervised training paradigm: The training sets created in (1) are sequentially used as training data input into the unified backbone network and classification head network model constructed in (2). The hyperparameters constructed in (3) are applied to the linear programming incremental learning supervised training paradigm to train the unified backbone network and classification head network model. During the training process, the validation set created in (1) is used to calculate the classification performance of the unified backbone network and classification head network model in real time. Then, the weights of the unified backbone network and classification head network model are back-propagated using the loss function, optimizer, and learning rate hyperparameters to obtain the trained unified backbone network and classification head network model weights. (5) Output of classification prediction results: Call the test_model_builder function to load the unified backbone network and classification head network model, then use the weights obtained in (4) as the parameters of the unified backbone network and classification head network model obtained in (2), input the test set constructed in (1) into the unified backbone network and classification head network model, and output the classification prediction results; (6) Unified model verification: The classification prediction results output by (5) are used for performance evaluation to verify the performance of the unified backbone network and classification head network model in terms of classification accuracy and generalization ability; Furthermore, (1) is specifically as follows: Indian Pines, Pavia University, Pavia Centre, Salinas and Houston 2013 datasets are used for training, the datasets are randomly divided into training set and validation set in a ratio of 9:1, and then data augmentation is performed on the validation set; Furthermore, the data enhancement methods include: horizontal flipping, normalization, resizing, color jittering, color blurring, image scale enhancement, image cropping enhancement or image translation enhancement; Furthermore, the above (2) is specifically: (2-1) Segmentation and embedding: Each image in the dataset is segmented into non-overlapping patches, and then the non-overlapping patches are linearly embedded into high-dimensional space vectors through the PatchEmbed linear layer; (2-2) Position code addition: Add the position code to the high-dimensional space vector of each tile; (2-3) Input Dropout layer: Input the high-dimensional space vector with position encoding into the Dropout layer and output the result vector A; (2-4) Enter the linear layer of the neural network classification model: Input the result vector A into the linear layer of the neural network classification model to obtain the result vector D; (2-5) Merge: Downsample the result vector D and merge the downsampled data as the feature extracted by the backbone network; (2-6) Normalization and pooling: Normalize and pool the features extracted by the backbone network in turn, output the result vector, and then input the result vector into the classification head network to complete the construction of the unified backbone network and classification head network model; Furthermore, (2-4) is specifically: (2-4-1) Construct a multi-head self-attention mechanism at the image block level: divide the result vector A into several non-overlapping image blocks, and calculate the attention feature information within each non-overlapping image block and between adjacent image blocks; (2-4-2) Input the attention feature information into the multi-layer perceptron and output the result vector B; (2-4-3) Normalize the result vector B and output the result vector C; (2-4-4) Add result vector A to result vector C and output result vector D; Furthermore, the constructing of the optimizer in (3) is to update the parameters according to the gradient of the loss function, and the optimizer is SGD, AdamW or Adam; constructing the learning rate includes polynomial decay, cosine annealing or step decay, polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, cosine annealing smoothly adjusts the learning rate through a cosine function, and step decay decreases the learning rate at a preset number of iterations or training cycles; Furthermore, the loss function constructed in (3) is specifically: When the loss function type is the cross entropy loss function, a standard cross entropy loss function is returned; When the type of the loss function is not the cross entropy loss function, the loss function is dynamically imported and the specified loss function class is instantiated. The loss function is constructed by setting parameters. Furthermore, the above (4) is specifically: (4-1) The training set constructed in (1) is used as training data in turn Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the old linear programming incremental learning classifier ; (4-2) The training set constructed in (1) Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the new linear programming incremental learning classifier ; (4-3) Use and The unified backbone network constructed by (2) and Perform adjustment. During the adjustment process, the classification performance of the unified backbone network and classification head network model is calculated in real time using the validation set created in (1). Then, the weights of the model are back-propagated using the loss function, optimizer, and learning rate hyperparameters. (4-4) Updated to .

[0008] A classification method for remote sensing scenes based on a unified framework incremental learning in a hyperspectral modal data environment, based on the above-mentioned unified framework incremental learning remote sensing scene classification method.

[0009] A unified framework incremental learning remote sensing scene classification system, based on the above-mentioned unified framework incremental learning remote sensing scene classification method, is characterized by comprising: A unified data set construction module is used to construct a unified database containing several image data sets and perform data enhancement on the unified database; Unified network model building module, used to adjust the model parameters in the experimental configuration file, realize the selection of different types of backbone networks, and combine the backbone network and the classification head network into a unified model; Unified hyperparameter building block for updating model parameters according to the gradient of the loss function to minimize the loss during training; The linear programming incremental learning supervised training paradigm network training module is used to train heterogeneous supervised data sets and introduce the linear programming incremental learning classifier between each stage of incremental learning to adapt the deep learning classification model to the new data set while solving the problem of catastrophic forgetting of the model. The classification prediction result output module is used to input the test set data into the unified backbone network and classification head network model loaded with weights, and the unified backbone network and classification head network model perform classification prediction on each image in the test set; The unified model verification module is used to predict category labels and evaluate performance on the verification set to verify the performance of the unified model in terms of classification accuracy and generalization ability.

[0010] The above technical solution has the following advantages or beneficial effects: The present invention provides a unified framework incremental learning remote sensing scene classification method. By constructing a unified image database (including training set, validation set and test set) and a unified backbone network and classification head network model, the method provides a standardized and reusable solution, which greatly improves the efficiency of experiments and deployments, and also enables the model to flexibly adapt to different remote sensing scene classification tasks; data enhancement is performed on the unified database, and by introducing more variants and complexities into the validation data, the generalization ability of the model can be effectively improved when dealing with unseen or highly variable remote sensing scenes; by unifying the hyperparameter configuration (including optimizer, learning rate and loss function), the model training process can be systematically adjusted and optimized, thereby maximizing the classification accuracy and training efficiency of the model, which helps to find the optimal configuration that best suits the current task and reduces human intervention. and trial-and-error costs; the adopted linear programming incremental learning supervised training paradigm allows the model to gradually learn new data sets while retaining previously learned knowledge. It is suitable for processing large-scale or continuously updated remote sensing data, and there is no need to train the entire model from scratch, thus saving computing resources and time, and enabling existing deep learning classification models to adapt to new data sets, thereby achieving rapid learning of new classification capabilities, and also helping to avoid the catastrophic forgetting problem, that is, the model forgets old tasks when learning new tasks; the unified model verification step provides quantitative indicators for the classification accuracy and generalization ability of the model by predicting the category labels of the verification set and performing performance evaluation, which helps researchers or developers to clearly understand the performance of the model and make further adjustments and optimizations as needed.

[0011] Furthermore, by using different data sets for training, the model can learn a wider range of richer scene features, and the fusion of multi-source data helps the model to have stronger adaptability when dealing with actual remote sensing scenes; different data sets represent different geographical conditions, lighting conditions, and ground cover types. The model can learn these changing factors during the training process, so that it can more accurately classify new and unseen remote sensing images; the data set is divided into training set and validation set in a 9:1 ratio, ensuring that the model learns on enough samples, and there are also enough samples to verify the performance of the model, which helps to adjust the model parameters in time during the training process and improve the final classification accuracy; data enhancement of the validation set can simulate various deformations and interferences that may be encountered in actual applications, so that the model can still maintain stable performance when facing these situations.

[0012] Furthermore, by introducing a variety of data enhancement methods, the model can be exposed to more diverse data variants during training, thereby learning more robust feature representations, which helps the model make more accurate classification judgments when faced with complex and changeable remote sensing scenes in practical applications, thereby improving the generalization ability of the model; through data enhancement, the model can learn more comprehensive and rich feature information, which helps to improve the accuracy of the model in classification tasks, especially for some subtle category differences or complex scene changes, data enhancement can enhance the model's ability to capture these details, thereby improving classification accuracy.

[0013] Furthermore, through segmentation and embedding, the remote sensing image data is effectively converted into a high-dimensional space vector, which not only reduces the complexity of the data, but also improves the efficiency of data processing. Non-overlapping tile segmentation and PatchEmbed linear embedding ensure that each tile can be fully utilized while retaining the key information of the image; by adding position encoding, adding position encoding to the high-dimensional space vector solves the problem of position information loss that may be encountered by convolutional neural networks when processing sequence data, which helps the model to better understand the relationship between different areas in the image, thereby improving the accuracy of classification; inputting the Dropout layer effectively prevents the overfitting of the model and enhances the generalization ability and robustness of the model; the result vector A is input into the linear layer of the neural network classification model and after merging, it is normalized and pooled, making the model structure more flexible and scalable.

[0014] Furthermore, by constructing a multi-head self-attention mechanism at the image block level, the complex relationship within each non-overlapping image block and between adjacent image blocks can be captured, which helps the model to understand the image content more deeply and extract richer and more accurate feature information; inputting the attention feature information into the multi-layer perceptron can further refine and fuse these features, so that the output result vector B contains both the information of the original image block and the association information between adjacent blocks, which helps to improve the accuracy of classification; normalizing the result vector B can stabilize the training process of the model and prevent the gradient disappearance or explosion problem caused by excessively large or small eigenvalues, which helps the model to better learn and adapt to new data; adding the result vector A to the result vector C not only retains the feature information of the original image block, but also incorporates the enhanced features processed by the self-attention mechanism and the multi-layer perceptron, so that the output result vector D has both the integrity of the original information and stronger expression ability.

[0015] Furthermore, by providing the choice of SGD, AdamW or Adam optimizer, users can choose the most suitable optimization algorithm according to specific tasks and data characteristics; learning rate configuration strategies such as polynomial decay, cosine annealing and step decay enable the learning rate to be dynamically adjusted as the training process progresses, which helps to use a larger learning rate in the early stage of training to quickly reduce the loss, and gradually reduce the learning rate in the later stage of training to fine-tune the model parameters, thereby avoiding overfitting and improving the generalization ability of the model; it supports dynamic generation and customization of loss functions through configuration, so that users can choose or design the most suitable loss function according to specific task requirements. Cross entropy loss, as a common loss function in classification tasks, can directly measure the difference between the probability distribution predicted by the model and the true label; by reasonably configuring the optimizer, learning rate and loss function, the training stability and convergence speed of the model can be significantly improved; by continuously optimizing the model parameters to minimize the loss function, the model can gradually learn the inherent laws and feature representations of the data, which not only improves the performance of the model on the training set, but more importantly, enhances the generalization ability of the model on the unseen test set, so that the model can better cope with complex scenarios in practical applications.

[0016] Furthermore, when the cross entropy loss function is used, the standard cross entropy loss function is directly returned, which simplifies the operation; when other types of loss functions are needed, the loss function can be customized by dynamically importing and instantiating the specified loss function class and setting the corresponding parameters, so that the method of the present invention has higher flexibility and scalability when processing different types of remote sensing scene classification tasks.

[0017] Furthermore, by inputting the training set as training data into the unified backbone network in turn, the new category data can be processed efficiently, which not only reduces the amount of data required for each training, but also enables the model to gradually learn and adapt to the characteristics of the new category, thereby improving the training efficiency and classification accuracy; when the old category data is accessible, the current category and the new category data are used to jointly fine-tune the feature extractor and classifier, which helps the model to better adapt to the new category data while maintaining the recognition ability of the old category, thereby improving the stability and generalization ability of the model; when the old category data is inaccessible, the linear programming method of updating the classifier weight matrix is ​​adopted to avoid the need to retrain the entire model, which not only saves computing resources but also shortens the update time of the model, making it more suitable for practical application scenarios; through the incremental learning method, the model can be gradually updated and optimized to maintain a high performance in the classification task of the ever-changing remote sensing scene; compared with other incremental learning methods, the linear programming incremental learning classifier reduces the forgetting of the old category information by directly adding the feature vector of the new category to the weight matrix. The weight of the old category remains unchanged during the update process, and the model can continue to use the information of these old categories to make classification decisions.

[0018] The present invention also provides a classification method for remote sensing scenes based on a unified framework incremental learning in a hyperspectral modal data environment. Hyperspectral data has extremely high spectral resolution and contains rich ground object information. Through a unified framework and incremental learning strategy, hyperspectral data can be efficiently processed and analyzed, and feature information of great value for remote sensing scene classification can be extracted. Due to the complexity and diversity of hyperspectral data, traditional classification methods often find it difficult to obtain ideal classification effects. This method, by constructing a flexible unified framework and combining it with an incremental learning strategy, can continuously adapt to new data and scenes, thereby improving the generalization ability of the model and enabling it to more accurately classify different remote sensing scenes.

[0019] The present invention also provides a unified framework incremental learning remote sensing scene classification system, which provides high configurability and flexibility through a unified data set construction module, a unified network model construction module and a unified hyperparameter configuration module. Users can easily adjust data sets, select different types of backbone networks, configure model parameters and hyperparameters according to specific task requirements to adapt to different remote sensing scene classification tasks; the linear programming incremental learning supervised training module can effectively train heterogeneous supervised data sets, and introduce the linear programming incremental learning classifier between each stage of incremental learning, which not only enables the deep learning classification model to quickly adapt to the new data set, but also effectively alleviates the catastrophic forgetting problem that the model may encounter during the incremental learning process, thereby maintaining The stability and continuous performance of the model are improved; the classification prediction result output module uses a unified backbone network and classification head network model loaded with weights to make accurate classification predictions for each image in the test set; the unified model verification module can comprehensively verify the performance of the unified model in classification accuracy and generalization ability by predicting category labels and evaluating the performance of the verification set. The comprehensive verification and evaluation mechanism helps users to understand the performance characteristics of the model more accurately, and then carry out targeted optimization and improvement; the system adopts a unified framework and modular design, with strong adaptability and scalability. With the continuous development of remote sensing technology and the emergence of new task requirements, the system can easily add new modules or expand the functions of existing modules to adapt to new application scenarios and task requirements. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 The present invention is a flowchart of a unified framework incremental learning remote sensing scene classification method and system. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it. In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention. It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] Embodiment 1: like Figure 1 As shown, the present invention provides a unified framework incremental learning remote sensing scene classification method, comprising the following steps: (1) Unified database construction: A unified database containing several image datasets is constructed, where each image dataset consists of a training set, a validation set, and a test set. The unified database is then augmented. Specifically, the Indian Pines, Pavia University, Pavia Centre, Salinas, and Houston 2013 datasets are used for training. The datasets are randomly divided into training and validation sets in a ratio of 9:1. The validation set is then augmented. Preferably, the data enhancement methods include: horizontal flipping, normalization, resizing, color jittering, color blurring, image scale enhancement, image cropping enhancement or image translation enhancement; (2) Unified backbone network and classification head network model construction: The backbone network and classification head network are combined into a unified model, specifically: (2-1) Segmentation and embedding: Each image in the dataset is segmented into non-overlapping patches, and then the non-overlapping patches are linearly embedded into high-dimensional space vectors through the PatchEmbed linear layer; (2-2) Position code addition: Add the position code to the high-dimensional space vector of each tile; (2-3) Input Dropout layer: Input the high-dimensional space vector with position encoding into the Dropout layer and output the result vector A; (2-4) Input the linear layer of the neural network classification model: Input the result vector A into the linear layer of the neural network classification model to obtain the result vector D; (2-4-1) Construct a multi-head self-attention mechanism at the image block level: divide the result vector A into several non-overlapping image blocks, and calculate the attention feature information within each non-overlapping image block and between adjacent image blocks; (2-4-2) Input the attention feature information into the multi-layer perceptron and output the result vector B; (2-4-3) Normalize the result vector B and output the result vector C; (2-4-4) Add result vector A to result vector C and output result vector D; (2-5) Merge: Downsample the result vector D and merge the downsampled data as the feature extracted by the backbone network; (2-6) Normalization and pooling: Normalize and pool the features extracted by the backbone network in turn, output the result vector, and then input the result vector into the classification head network to complete the construction of the unified backbone network and classification head network model; (3) Unified hyperparameter configuration: build unified optimizer, learning rate, and loss function hyperparameters; Preferably, constructing an optimizer to update model parameters according to the gradient of the loss function, the optimizer is SGD, AdamW or Adam; Preferably, constructing the learning rate includes polynomial decay, cosine annealing or step decay, polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, cosine annealing smoothly adjusts the learning rate through a cosine function, and step decay decreases the learning rate at a preset number of iterations or training cycles; Preferably, the loss function is constructed as follows: When the loss function type is the cross entropy loss function, a standard cross entropy loss function is returned; When the type of the loss function is not the cross entropy loss function, the loss function is dynamically imported and the specified loss function class is instantiated. The loss function is constructed by setting parameters. (4) Network training using the linear programming incremental learning supervised training paradigm: The training sets created in (1) are sequentially used as training data input into the unified backbone network and classification head network model constructed in (2). The hyperparameters constructed in (3) are applied to the linear programming incremental learning supervised training paradigm to train the unified backbone network and classification head network model. During the training process, the validation set created in (1) is used to calculate the classification performance of the unified backbone network and classification head network model in real time. Then, the weights of the unified backbone network and classification head network model are back-propagated using the loss function, optimizer, and learning rate hyperparameters to obtain the trained unified backbone network and classification head network model weights, which are as follows: (4-1) The training set constructed in (1) is used as training data in turn Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the old linear programming incremental learning classifier ; (4-2) The training set constructed in (1) Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the new linear programming incremental learning classifier ; (4-3) Use and The unified backbone network constructed by (2) and Perform adjustment. During the adjustment process, the classification performance of the unified backbone network and classification head network model is calculated in real time using the validation set created in (1). Then, the weights of the model are back-propagated using the loss function, optimizer, and learning rate hyperparameters. (4-4) Updated to ; (5) Output of classification prediction results: Call the test_model_builder function to load the unified backbone network and classification head network model, then use the weights obtained in (4) as the parameters of the unified backbone network and classification head network model obtained in (2), input the test set constructed in (1) into the unified backbone network and classification head network model, and output the classification prediction results; (6) Unified model verification: The classification prediction results output by (5) are used for performance evaluation to verify the performance of the unified backbone network and classification head network model in terms of classification accuracy and generalization ability.

[0022] The present invention also provides a unified framework incremental learning remote sensing scene classification system, based on the above-mentioned unified framework incremental learning remote sensing scene classification method, comprising: A unified data set construction module is used to construct a unified database containing several image data sets and perform data enhancement on the unified database; Unified network model building module, used to adjust the model parameters in the experimental configuration file, realize the selection of different types of backbone networks, and combine the backbone network and the classification head network into a unified model; Unified hyperparameter building block for updating model parameters according to the gradient of the loss function to minimize the loss during training; The linear programming incremental learning supervised training paradigm network training module is used to train heterogeneous supervised data sets and introduce the linear programming incremental learning classifier between each stage of incremental learning to adapt the deep learning classification model to the new data set while solving the problem of catastrophic forgetting of the model. The classification prediction result output module is used to input the test set data into the unified backbone network and classification head network model loaded with weights, and the unified backbone network and classification head network model perform classification prediction on each image in the test set; The unified model verification module is used to predict category labels and evaluate performance on the verification set to verify the performance of the unified model in terms of classification accuracy and generalization ability.

[0023] Embodiment 2: like Figure 1 As shown, the present invention provides a unified framework incremental learning remote sensing scene classification method, comprising the following steps: (1) Unified database construction: A unified database containing several image datasets is constructed, where each image dataset consists of a training set, a validation set, and a test set. The unified database is then augmented. Specifically, the Indian Pines, Pavia University, Pavia Centre, Salinas, and Houston 2013 datasets are used for training. The datasets are randomly divided into training and validation sets in a ratio of 9:1. The validation set is then augmented. Preferably, the dataset can cover RGB modality, SAR modality, infrared, multispectral and hyperspectral modalities, as well as corresponding ground object category labels; Preferably, the data enhancement methods include: horizontal flipping, normalization, resizing, color jittering, color blurring, image scale enhancement, image cropping enhancement or image translation enhancement; (2) Unified backbone network and classification head network model construction: The backbone network and classification head network are combined into a unified model, specifically: (2-1) Segmentation and embedding: Each image in the dataset is segmented into non-overlapping patches, and then the non-overlapping patches are linearly embedded into high-dimensional space vectors through the PatchEmbed linear layer; (2-2) Position code addition: Add the position code to the high-dimensional space vector of each tile; (2-3) Input Dropout layer: Input the high-dimensional space vector with position encoding into the Dropout layer and output the result vector A; (2-4) Input the linear layer of the neural network classification model: Input the result vector A into the linear layer of the neural network classification model to obtain the result vector D; (2-4-1) Construct a multi-head self-attention mechanism at the image block level: divide the result vector A into several non-overlapping image blocks, and calculate the attention feature information within each non-overlapping image block and between adjacent image blocks; (2-4-2) Input the attention feature information into the multi-layer perceptron and output the result vector B; (2-4-3) Normalize the result vector B and output the result vector C; (2-4-4) Add result vector A to result vector C and output result vector D; (2-5) Merge: Downsample the result vector D and merge the downsampled data as the feature extracted by the backbone network; (2-6) Normalization and pooling: Normalize and pool the features extracted by the backbone network in turn, output the result vector, and then input the result vector into the classification head network to complete the construction of the unified backbone network and classification head network model; Preferably, by adjusting the model parameters in the experimental configuration file, different types of backbone networks can be selected to adapt to different data inputs and task requirements; Preferably, the classification head network is responsible for converting the feature map extracted by the backbone network into the final classification result. According to the parameters in the experimental configuration file, different types of classification head networks can be selected and configured as needed; Preferably, this framework supports embedding almost all open source neural network classification models in the industry. The models that have been connected include: MiT, PVTv2, Vision Transformer (ViT), Swin Transformer, InternImage, VMamba, ConvNeXt, LSKNet, UniRepLKNet, FocalNet; (3) Unified hyperparameter configuration: build unified optimizer, learning rate, and loss function hyperparameters; Preferably, an optimizer is constructed to update parameters according to the gradient of the loss function, and the optimizer is SGD, AdamW or Adam, and the specific choice depends on the complexity of the model and the characteristics of the data set; Preferably, constructing the learning rate includes polynomial decay, cosine annealing or step decay, polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, cosine annealing smoothly adjusts the learning rate through a cosine function, and step decay decreases the learning rate at a preset number of iterations or training cycles; the learning rate determines the amplitude of parameter update in each iteration, and is an important hyperparameter for adjusting the training speed and convergence effect. The learning rate can be set in a constant manner or dynamically adjusted according to the progress of the training; Preferably, the present invention supports three learning rate scheduling strategies: polynomial decay (poly), cosine annealing (cosine) and step decay (step); polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, cosine annealing smoothly adjusts the learning rate through a cosine function, and step decay decreases the learning rate at a preset number of iterations or training cycles. Different strategies and parameter configurations can adapt to different training requirements and optimization goals; Preferably, the loss function is constructed as follows: When the loss function type is the cross entropy loss function, a standard cross entropy loss function is returned; When the type of loss function is not the cross entropy loss function, the loss function is dynamically imported and the specified loss function class is instantiated. The loss function is constructed by setting parameters to customize the loss function, allowing flexible selection of different loss functions to meet various training requirements. Preferably, the loss function is an indicator for evaluating the difference between the predicted result and the actual label, and is the objective function in the training process of the unified backbone network and the classification head network model. For different tasks and model designs, a suitable loss function can be selected to optimize the training effect; Preferably, the specific experimental hyperparameter design scheme is as follows: the training will last for 50 epochs, the size of each batch is 32 (on 4 GPUs, the total batch size is 128), the optimizer selects AdamW, the learning rate is set to 0.00006, the weight decay is 0.01, and there is a learning rate multiplication factor of 10.0; the learning rate scheduler uses a polynomial decay method (poly), the update type is the number of iterations (iter); the loss function is the cross entropy loss (CELoss), and the category label with index 255 is ignored; (4) Network training using the linear programming incremental learning supervised training paradigm: The training sets created in (1) are sequentially used as training data input into the unified backbone network and classification head network model constructed in (2). The hyperparameters constructed in (3) are applied to the linear programming incremental learning supervised training paradigm to train the unified backbone network and classification head network model. During the training process, the validation set created in (1) is used to calculate the classification performance of the unified backbone network and classification head network model in real time. Then, the weights of the unified backbone network and classification head network model are back-propagated using the loss function, optimizer, and learning rate hyperparameters to obtain the trained unified backbone network and classification head network model weights, which are as follows: (4-1) The training set constructed in (1) is used as training data in turn Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the old linear programming incremental learning classifier ; (4-2) The training set constructed in (1) Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the new linear programming incremental learning classifier ; (4-3) Use and The unified backbone network constructed by (2) and Perform adjustment. During the adjustment process, the classification performance of the unified backbone network and classification head network model is calculated in real time using the validation set created in (1). Then, the weights of the model are back-propagated using the loss function, optimizer, and learning rate hyperparameters. (4-4) Updated to ; (5) Output of classification prediction results: Call the test_model_builder function to load the unified backbone network and classification head network model, then use the weights obtained in (4) as the parameters of the unified backbone network and classification head network model obtained in (2), input the test set constructed in (1) into the unified backbone network and classification head network model, and output the classification prediction results; (6) Unified model verification: The classification prediction results output by (5) are used for performance evaluation to verify the performance of the unified backbone network and classification head network model in terms of classification accuracy and generalization ability. Specifically: Put the trained unified backbone network and classification head network models into evaluation mode, initialize the evaluation indicators, load the validation set data and input the images into the unified backbone network and classification head network models for inference. The prediction results are used to obtain the classification labels through argmax. The intersection and union of the prediction results and the true labels are calculated using the intersectionAndUnion function to generate indicator data. The indicator data on each GPU is synchronized through torch.distributed to ensure that all processes are consistent in the distributed environment. The evaluation indicators of the semantic segmentation task, change detection task, classification task, and visual localization task are updated. The overall evaluation indicators are calculated and the various evaluation results and corresponding labels are returned. Preferably, the evaluation indicators of the semantic segmentation task are IoU, precision, recall rate and F1 score; the evaluation indicators of the change detection task are IoU, precision, recall rate and F1 score; the evaluation indicators of the classification task are Overall Accuracy, Average Accuracy and Kappa Coefficient; the evaluation indicators of the visual localization task are PR and IoU.

[0024] The present invention also provides a unified framework incremental learning remote sensing scene classification system, based on the above-mentioned unified framework incremental learning remote sensing scene classification method, comprising: A unified data set construction module is used to construct a unified database containing several image data sets and perform data enhancement on the unified database; Unified network model building module, used to adjust the model parameters in the experimental configuration file, realize the selection of different types of backbone networks, and combine the backbone network and the classification head network into a unified model; Unified hyperparameter building block for updating model parameters according to the gradient of the loss function to minimize the loss during training; The linear programming incremental learning supervised training paradigm network training module is used to train heterogeneous supervised data sets and introduce the linear programming incremental learning classifier between each stage of incremental learning to adapt the deep learning classification model to the new data set while solving the problem of catastrophic forgetting of the model. The classification prediction result output module is used to input the test set data into the unified backbone network and classification head network model loaded with weights, and the unified backbone network and classification head network model perform classification prediction on each image in the test set; The unified model verification module is used to predict category labels and evaluate performance on the verification set to verify the performance of the unified model in terms of classification accuracy and generalization ability.

[0025] The system of the present invention adopts a unified framework and modular design, and has strong adaptability and scalability. With the continuous development of remote sensing technology and the emergence of new mission requirements, the system can easily add new modules or expand the functions of existing modules to adapt to new application scenarios and mission requirements.

[0026] Table 1 Unified dataset construction details

[0027] Embodiment 3: The present invention provides a unified framework incremental learning remote sensing scene classification method, comprising the following steps: (1) Unified database construction: A unified database containing several hyperspectral modal image datasets is constructed. Each hyperspectral modal image dataset consists of a training set, a validation set, and a test set. The unified database is then enhanced. Specifically, the Indian Pines, Pavia University, Pavia Centre, Salinas, and Houston2013 datasets are used for training. The datasets are randomly divided into training and validation sets in a ratio of 9:1. The validation set is then enhanced. The patch size of the hyperspectral image is 7x7. Preferably, the data enhancement methods include: horizontal flipping, normalization, resizing, color jittering, color blurring, image scale enhancement, image cropping enhancement or image translation enhancement; (2) Unified backbone network and classification head network model construction: The backbone network and classification head network are combined into a unified model. The Swin Transformer is used as the model backbone network as an example. Specifically: (2-1) Segmentation and embedding: Swin Transformer divides each image in the hyperspectral modality image dataset into non-overlapping patches, and then linearly embeds the non-overlapping patches into high-dimensional space vectors through the PatchEmbed linear layer; (2-2) Position coding addition: Adding the position coding to the high-dimensional space vector of each tile helps capture the location information of the tile; (2-3) Input Dropout layer: Input the high-dimensional space vector with position encoding into the Dropout layer and output the result vector A; (2-4) Input the linear layer of the neural network classification model: Input the result vector A into the linear layer of the neural network classification model to obtain the result vector D; (2-4-1) Construct a multi-head self-attention mechanism at the image block level: divide the result vector A into several non-overlapping image blocks, and calculate the attention feature information within each non-overlapping image block and between adjacent image blocks; (2-4-2) Input the attention feature information into the multi-layer perceptron and output the result vector B; (2-4-3) Normalize the result vector B and output the result vector C; (2-4-4) Add result vector A to result vector C and output result vector D; (2-5) Merge: Downsample the result vector D and merge the downsampled data as the features extracted by SwinTransformer; (2-6) Normalization and pooling: Normalize and pool the features extracted by Swin Transformer in turn, output the result vector, and then input the result vector into the classification head network to complete the construction of the unified backbone network and classification head network model; (3) Unified hyperparameter configuration: build unified optimizer, learning rate, and loss function hyperparameters; Preferably, an optimizer is constructed to update parameters according to the gradient of the loss function, and the optimizer is selected as AdamW, and the specific selection depends on the complexity of the model and the characteristics of the data set; Preferably, constructing the learning rate includes polynomial decay, cosine annealing or step decay, polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, cosine annealing smoothly adjusts the learning rate through a cosine function, and step decay decreases the learning rate at a preset number of iterations or training cycles; the learning rate determines the amplitude of parameter update in each iteration, and is an important hyperparameter for adjusting the training speed and convergence effect. The learning rate can be set in a constant manner or dynamically adjusted according to the progress of the training; Preferably, the present invention supports three learning rate scheduling strategies: polynomial decay (poly), cosine annealing (cosine) and step decay (step); polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, cosine annealing smoothly adjusts the learning rate through a cosine function, and step decay decreases the learning rate at a preset number of iterations or training cycles. Different strategies and parameter configurations can adapt to different training requirements and optimization goals; Preferably, the loss function is constructed as follows: When the loss function type is the cross entropy loss function, a standard cross entropy loss function is returned; When the type of loss function is not the cross entropy loss function, the loss function is dynamically imported and the specified loss function class is instantiated. The loss function is constructed by setting parameters to customize the loss function, allowing flexible selection of different loss functions to meet various training requirements. Preferably, the loss function is an indicator for evaluating the difference between the predicted result and the actual label, and is the objective function in the training process of the unified backbone network and the classification head network model. For different tasks and model designs, a suitable loss function can be selected to optimize the training effect; Preferably, the specific experimental hyperparameter design scheme is as follows: the training will last for 50 epochs, the size of each batch is 32 (on 4 GPUs, the total batch size is 128), the optimizer selects AdamW, the learning rate is set to 0.00006, the weight decay is 0.01, and there is a learning rate multiplication factor of 10.0; the learning rate scheduler uses a polynomial decay method (poly), the update type is the number of iterations (iter); the loss function is the cross entropy loss (CELoss), and the category label with index 255 is ignored; (4) Network training using the linear programming incremental learning supervised training paradigm: The training sets created in (1) are sequentially used as training data input into the unified backbone network and classification head network model constructed in (2). The hyperparameters constructed in (3) are applied to the linear programming incremental learning supervised training paradigm to train the unified backbone network and classification head network model. During the training process, the validation set created in (1) is used to calculate the classification performance of the unified backbone network and classification head network model in real time. Then, the weights of the unified backbone network and classification head network model are back-propagated using the loss function, optimizer, and learning rate hyperparameters to obtain the trained unified backbone network and classification head network model weights, which are as follows: (4-1) The training set constructed in (1) is used as training data in turn Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the old linear programming incremental learning classifier ; (4-2) The training set constructed in (1) Input (2) to construct the unified backbone network and extract the feature vector list ,Will Input the classification head network constructed by (2) to obtain the weights of the new linear programming incremental learning classifier ; (4-3) Use and The unified backbone network constructed by (2) and Perform adjustment. During the adjustment process, the classification performance of the unified backbone network and classification head network model is calculated in real time using the validation set created in (1). Then, the weights of the model are back-propagated using the loss function, optimizer, and learning rate hyperparameters. (4-4) Updated to ; (5) Output of classification prediction results: Call the test_model_builder function to load the unified backbone network and classification head network model, then use the weights obtained in (4) as the parameters of the unified backbone network and classification head network model obtained in (2), input the test set constructed in (1) into the unified backbone network and classification head network model, and output the classification prediction results; (6) Unified model verification: The classification prediction results output by (5) are used for performance evaluation to verify the performance of the unified backbone network and classification head network model in terms of classification accuracy and generalization ability. Specifically: Put the trained unified backbone network and classification head network models into evaluation mode, initialize the evaluation indicators, load the validation set data and input the images into the unified backbone network and classification head network models for inference. The prediction results are used to obtain the classification labels through argmax. The intersection and union of the prediction results and the true labels are calculated using the intersectionAndUnion function to generate indicator data. The indicator data on each GPU is synchronized through torch.distributed to ensure that all processes are consistent in the distributed environment. The evaluation indicators of the semantic segmentation task, change detection task, classification task, and visual localization task are updated. The overall evaluation indicators are calculated and the various evaluation results and corresponding labels are returned. Preferably, the evaluation indicators of the semantic segmentation task are IoU, precision, recall rate and F1 score; the evaluation indicators of the change detection task are IoU, precision, recall rate and F1 score; the evaluation indicators of the classification task are Overall Accuracy, Average Accuracy and Kappa Coefficient; the evaluation indicators of the visual localization task are PR and IoU.

[0028] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unified framework incremental learning remote sensing scene classification method, characterized in that: The following steps are involved: S1, unified database construction: construct a unified database containing several image datasets, where each image dataset consists of a training set, a validation set, and a test set, and then perform data enhancement on the unified database; S2, unified backbone network and classification head network model construction: the backbone network and classification head network are combined into a unified model; S3, unified hyperparameter construction: build unified optimizer, learning rate and loss function hyperparameters; S4, linear programming incremental learning supervised training paradigm network training: several training sets created in S1 are input as training data in turn into the unified backbone network and classification head network model constructed in S2, and the hyperparameters constructed in S3 are applied to the linear programming incremental learning supervised training paradigm to train the unified backbone network and classification head network model. During the training process, the validation set created in S1 is used to calculate the classification performance of the unified backbone network and classification head network model in real time, and then the weights of the unified backbone network and classification head network model are back-propagated using the loss function, optimizer and learning rate hyperparameters to obtain the trained unified backbone network and classification head network model weights; S5, classification prediction result output: call the test_model_builder function to load the unified backbone network and classification head network model, then use the weights obtained in S4 as the unified backbone network and classification head network model parameters obtained in S2, input the test set built in S1 into the unified backbone network and classification head network model, and output the classification prediction result; S6, unified model verification: The classification prediction results output by S5 are used for performance evaluation to verify the performance of the unified backbone network and classification head network model in terms of classification accuracy and generalization ability.

2. The unified framework incremental learning remote sensing scene classification method according to claim 1, characterized in that: Specifically, S1 uses Indian Pines, Pavia University, Pavia Centre, Salinas and Houston2013 datasets for training, randomly divides the datasets into training sets and validation sets in a ratio of 9:1, and then performs data augmentation on the validation set.

3. A unified framework incremental learning remote sensing scene classification method according to claim 2, characterized in that: The data enhancement methods include: horizontal flipping, normalization, size adjustment, color jittering, color blurring, image scale enhancement, image cropping enhancement or image translation enhancement.

4. The unified framework incremental learning remote sensing scene classification method according to claim 1, characterized in that: The S2 is specifically: S2-1, segmentation and embedding: Each image in the dataset is segmented into non-overlapping patches, and then the non-overlapping patches are linearly embedded into high-dimensional space vectors through the PatchEmbed linear layer; S2-2, position code addition: add the position code to the high-dimensional space vector of each tile; S2-3, input Dropout layer: input the high-dimensional space vector with position encoding into the Dropout layer and output the result vector A; S2-4, inputting the linear layer of the neural network classification model: inputting the result vector A into the linear layer of the neural network classification model to obtain the result vector D; S2-5, merging: downsampling the result vector D, and merging the downsampled data as features extracted by the backbone network; S2-6, normalization and pooling: normalize and pool the features extracted by the backbone network in turn, output the result vector, and then input the result vector into the classification head network to complete the construction of the unified backbone network and classification head network model.

5. A unified framework incremental learning remote sensing scene classification method according to claim 4, characterized in that: The S2-4 is specifically: S2-4-1, construct a multi-head self-attention mechanism at the image block level: divide the result vector A into several non-overlapping image blocks, and calculate the attention feature information within each non-overlapping image block and between adjacent image blocks; S2-4-2, input the attention feature information into the multi-layer perceptron and output the result vector B; S2-4-3, normalize the result vector B and output the result vector C; S2-4-4, add result vector A and result vector C, and output result vector D.

6. The unified framework incremental learning remote sensing scene classification method according to claim 1, characterized in that: The constructing optimizer in S3 is to update the parameters according to the gradient of the loss function, and the optimizer is SGD, AdamW or Adam; the constructing learning rate includes polynomial decay, cosine annealing or step decay, the polynomial decay adjusts the learning rate according to the total number of training iterations or cycles, the cosine annealing smoothly adjusts the learning rate through the cosine function, and the step decay decreases the learning rate at a preset number of iterations or training cycles.

7. The unified framework incremental learning remote sensing scene classification method according to claim 1, characterized in that: The loss function constructed in S3 is specifically: When the loss function type is the cross entropy loss function, a standard cross entropy loss function is returned; When the type of the loss function is not the cross entropy loss function, the loss function is dynamically imported and the specified loss function class is instantiated. The loss function is constructed by setting custom parameters.

8. The unified framework incremental learning remote sensing scene classification method according to claim 1, characterized in that: The S4 is specifically: S4-1, the training set constructed in S1 is used as training data in turn Input the unified backbone network constructed by S2 and extract the feature vector list ,Will Input the classification head network constructed by S2 to obtain the weights of the old linear programming incremental learning classifier ; S4-2, the training set constructed in S1 Input the unified backbone network constructed by S2 and extract the feature vector list ,Will Input the classification head network constructed by S2 to obtain the weights of the new linear programming incremental learning classifier ; S4-3, use and The unified backbone network built by S2 and Adjustment is performed, during which the classification performance of the unified backbone network and classification head network model is calculated in real time using the validation set created in S1, and then the weights of the model are back-propagated using the loss function, optimizer, and learning rate hyperparameters; S4-4, will Updated to .

9. A classification method for remote sensing scenes based on a unified framework incremental learning in a hyperspectral modality data environment, characterized in that: A unified framework incremental learning remote sensing scene classification method based on any one of claims 1-8.

10. A unified framework incremental learning remote sensing scene classification system, based on a unified framework incremental learning remote sensing scene classification method according to any one of claims 1 to 9, characterized in that: include: A unified data set construction module is used to construct a unified database containing several image data sets and perform data enhancement on the unified database; Unified network model building module, used to adjust the model parameters in the experimental configuration file, realize the selection of different types of backbone networks, and combine the backbone network and the classification head network into a unified model; Unified hyperparameter building block for updating model parameters according to the gradient of the loss function to minimize the loss during training; The linear programming incremental learning supervised training paradigm network training module is used to train heterogeneous supervised data sets and introduce the linear programming incremental learning classifier between each stage of incremental learning to adapt the deep learning classification model to the new data set while solving the problem of catastrophic forgetting of the model. The classification prediction result output module is used to input the test set data into the unified backbone network and classification head network model loaded with weights, and the unified backbone network and classification head network model perform classification prediction on each image in the test set; The unified model verification module is used to predict category labels and evaluate performance on the verification set to verify the performance of the unified model in terms of classification accuracy and generalization ability.

Images