A continuous learning method, device, electronic device and storage medium for a cross-scene remote sensing image classification model

Abstract

The present application provides a continuous learning method, device, electronic device and storage medium for a cross-scene remote sensing image classification model, including: extracting features of a new task remote sensing image and an old task remote sensing image based on a multidimensional convolutional neural network layer to determine the features of the new remote sensing image and the old remote sensing image; determining the cosine similarity between the features of the new remote sensing image and the old remote sensing image based on a vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature measurement learning based on cosine similarity; constructing a loss function based on maximizing the consistency of the new and old models to iteratively optimize the network parameters of the initial remote sensing image classification model, generate a new cross-scene remote sensing image classification model, so that the new task remote sensing image can be automatically classified. Thereby improving the cross-scene robustness and practicality of the remote sensing image classification model, so that the remote sensing classification model can achieve continuous learning and updating under the massive emergence of remote sensing data sets.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a continuous learning method, device, electronic device and storage medium for a cross-scene remote sensing image classification model. Background Art

[0002] Remote sensing images obtained by remote sensing earth observation sensors have been widely used in mining exploration, agricultural monitoring, environmental management, etc. Remote sensing image classification is not only one of the basic problems in remote sensing information processing, but also the key to the practical application of remote sensing technology. Deep learning methods have made important progress in remote sensing image classification due to their strong autonomous feature learning and category judgment capabilities. However, existing remote sensing classification models are usually trained and predicted on given data sets. When faced with different data sets or classification scenarios with obvious spectral differences and category differences, the trained classification model cannot be directly applied to the new data set. The old model needs to be retrained and adjusted, resulting in the loss of the original knowledge of the model and waste of computing resources, which greatly limits the application scope of the remote sensing image classification model. As a result, the classification model based on historical (old task) remote sensing images is difficult to use for current or even future classification tasks (new tasks). Based on the massive accumulation of earth observation data, especially remote sensing data, the current remote sensing data processing paradigm can no longer meet the major needs of rapid acquisition of remote sensing information. Therefore, how to continuously update the classification model of remote sensing images has become a technical issue that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a continuous learning method, device, electronic device and storage medium for a cross-scene remote sensing image classification model, so as to improve the robustness and practicality of the remote sensing image classification model across scenes, and enable the remote sensing classification model to achieve continuous learning and updating under the massive emergence of remote sensing data sets.

[0004] The present application embodiment provides a continuous learning method for a cross-scene remote sensing image classification model, and the continuous learning method includes:

[0005] The multidimensional convolutional neural network layer based on the old cross-scene remote sensing image classification model extracts the features of the sample new task remote sensing image and the old task remote sensing image, and determines the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image;

[0006] Determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on the vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric;

[0007] A loss function is constructed based on maximizing the consistency between the new and old models, and the target loss value of the loss function is used to iteratively optimize the network parameters of the initial remote sensing image classification model to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task.

[0008] In a possible implementation, for the sample new remote sensing image features, the multidimensional convolutional neural network layer based on the old cross-scene remote sensing image classification model extracts features of the sample new task remote sensing image and the old task remote sensing image, and determines the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image, including:

[0009] Inputting the spectral vectors in the sample new task remote sensing image into the spectral feature extraction module of the multi-dimensional convolutional neural network layer, performing multiple convolution processing and multiple maximum pooling processing on each of the spectral vectors to obtain multiple spectral features;

[0010] Inputting the spatial slices of the spatial background information in the sample new task remote sensing image into the spatial feature extraction module of the multi-dimensional convolutional neural network layer, performing convolution processing and multiple maximum pooling processing on each of the spatial slices to obtain multiple spatial features;

[0011] The multiple spatial features and the multiple spectral features are fused to obtain the new remote sensing image feature.

[0012] In a possible implementation manner, the iterative optimization of the network parameters of the initial remote sensing image classification model using the target loss value of the loss function to generate a new cross-scene remote sensing image classification model includes:

[0013] Detecting whether the target loss value is less than or equal to a preset loss value;

[0014] If yes, stop changing the network parameters of the initial remote sensing image classification model and generate the new cross-scene remote sensing image classification model;

[0015] If not, continue to change the network parameters of the initial remote sensing image classification model, and use the old remote sensing image features and the new remote sensing image features to continue training the initial remote sensing image classification model after the changed parameters, until the target loss value is less than or equal to the preset loss value, then stop training and generate the new cross-scene remote sensing image classification model.

[0016] In a possible implementation manner, the target loss value is determined by the following steps:

[0017] The features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model are calculated based on the feature consistency loss function to determine the feature consistency loss value between the new and old models;

[0018] Based on the category loss function, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated to determine the category loss value between the new and old models;

[0019] Calculating the classification result of the initial remote sensing image classification model for the current task remote sensing image based on the cross entropy loss function, and determining the cross entropy loss value of the initial remote sensing image classification model;

[0020] The feature consistency loss value, the category loss value, and the cross entropy loss value are added to determine the target loss value.

[0021] In a possible implementation manner, the feature consistency loss function is used to calculate the features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model to determine the feature consistency loss value between the new and old models, including:

[0022] Performing vector cosine angle calculation on spatial features in the new remote sensing image features extracted by the initial remote sensing image classification model and spatial features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a first similarity value;

[0023] Performing vector cosine angle calculation on the spectral features in the new remote sensing image features extracted by the initial remote sensing image classification model and the spectral features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a second similarity value;

[0024] The feature consistency loss value is determined based on the feature consistency loss function for the first similarity value and the second similarity value.

[0025] In a possible implementation manner, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated based on the category loss function to determine the category loss value between the new and old models, including:

[0026] Inputting the first task remote sensing image into the initial remote sensing image classification model, and outputting a first classification result;

[0027] Inputting the first task remote sensing image into the old cross-scene remote sensing image classification model, and outputting a second classification result;

[0028] The first classification result and the second classification result are calculated based on the category loss function to determine the category loss value.

[0029] In a possible implementation manner, the calculating the classification result of the initial remote sensing image classification model for the current task remote sensing image based on the cross entropy loss function to determine the cross entropy loss value of the initial remote sensing image classification model includes:

[0030] Inputting the current task remote sensing image into the initial remote sensing image classification model, and outputting the predicted category label of the current task remote sensing image;

[0031] The real category label and the predicted category label of the remote sensing image of the current task are calculated based on the cross entropy loss function to determine the cross entropy loss value.

[0032] The present application also provides a continuous learning device for a cross-scene remote sensing image classification model, the continuous learning device comprising:

[0033] A feature extraction module is used to extract features of the sample new task remote sensing image and the old task remote sensing image based on the multidimensional convolutional neural network layer of the old cross-scene remote sensing image classification model, and determine the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image;

[0034] A feature metric learning module is used to determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on a vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric;

[0035] The training module is used to construct a loss function based on maximizing the consistency between the new and old models, iteratively optimize the network parameters of the initial remote sensing image classification model using the target loss value of the loss function, and generate a new cross-scene remote sensing image classification model so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task.

[0036] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the continuous learning method of the cross-scene remote sensing image classification model are performed as described above.

[0037] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the continuous learning method of the cross-scene remote sensing image classification model are executed as described above.

[0038] The embodiments of the present application provide a continuous learning method, device, electronic device and storage medium for a cross-scene remote sensing image classification model. The continuous learning method includes: extracting features of a sample new task remote sensing image and an old task remote sensing image based on a multidimensional convolutional neural network layer of an old cross-scene remote sensing image classification model to determine new remote sensing image features of the sample new task remote sensing image and old remote sensing image features of the old task remote sensing image; determining the cosine similarity between the new remote sensing image features and the old remote sensing image features based on a vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric; constructing a loss function based on maximizing the consistency between the new and old models, and iteratively optimizing the network parameters of the initial remote sensing image classification model using the target loss value of the loss function to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model automatically classifies the new task remote sensing image. This will improve the robustness and practicality of remote sensing image classification models across scenarios, and enable remote sensing classification models to achieve continuous learning and updating under the massive emergence of remote sensing data sets.

[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of a continuous learning method for a cross-scene remote sensing image classification model provided in an embodiment of the present application;

[0042] Figure 2 One of the structural schematic diagrams of a continuous learning device for a cross-scene remote sensing image classification model provided in an embodiment of the present application;

[0043] Figure 3A second structural diagram of a continuous learning device for a cross-scene remote sensing image classification model provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0046] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of remote sensing image processing technology.

[0047] Research has found that remote sensing images obtained by remote sensing earth observation sensors have been widely used in mining exploration, agricultural monitoring, environmental management, etc. Remote sensing image classification is not only one of the basic problems in remote sensing information processing, but also the key to the practical application of remote sensing technology. Deep learning methods have made important progress in remote sensing image classification due to their strong autonomous feature learning and category judgment capabilities. However, existing remote sensing classification models are usually trained and predicted on given data sets. When faced with different data sets or classification scenarios with obvious spectral differences and category differences, the trained classification model cannot be directly applied to the new data set. The old model needs to be retrained and adjusted, resulting in the loss of the original knowledge of the model and the waste of computing resources, which limits the application scope of the remote sensing image classification model. As a result, the classification model based on historical (old task) remote sensing images is difficult to use for current or even future classification tasks (new tasks). Based on the massive accumulation of earth observation data, especially remote sensing data, the current remote sensing data processing paradigm can no longer meet the major needs of rapid acquisition of remote sensing information. Therefore, how to continuously update the classification model of remote sensing images has become a technical issue that cannot be underestimated.

[0048] Based on this, an embodiment of the present application provides a continuous learning method for a cross-scene remote sensing image classification model, thereby improving the robustness and practicality of the remote sensing image classification model across scenes, enabling the remote sensing classification model to achieve continuous learning and updating under massive emerging remote sensing data sets.

[0049] See also Figure 1 , Figure 1 A flowchart of a continuous learning method for a cross-scene remote sensing image classification model provided in an embodiment of the present application. Figure 1 As shown in , the continuous learning method provided by the embodiment of the present application includes:

[0050] S101: Extract features of sample new task remote sensing images and old task remote sensing images based on a multidimensional convolutional neural network layer of an old cross-scene remote sensing image classification model to determine new remote sensing image features of the sample new task remote sensing images and old remote sensing image features of the old task remote sensing images.

[0051] In this step, the features of the sample new task remote sensing image and the old task remote sensing image are extracted according to the multidimensional convolutional neural network layer of the old cross-scene remote sensing image classification model to determine the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image.

[0052] Among them, the old cross-scene remote sensing image classification model is a pre-trained neural network model that can identify old mission remote sensing images.

[0053] Among them, in this application, the multidimensional convolutional neural network layer is composed of a spatial feature extractor and a spectral feature extractor. The one-dimensional spectral vector and the three-dimensional spatial patch are used as the input of the convolutional neural network to extract spectral and spatial features, and their fusion features are used as the input of the final classification layer as the extracted robust features to further improve the performance of the classification model.

[0054] In a possible implementation, for the sample new remote sensing image features, the multidimensional convolutional neural network layer based on the old cross-scene remote sensing image classification model extracts features of the sample new task remote sensing image and the old task remote sensing image, and determines the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image, including:

[0055] A: The spectral vectors in the sample new task remote sensing image are input into the spectral feature extraction module of the multi-dimensional convolutional neural network layer, and each of the spectral vectors is subjected to multiple convolution processing and multiple maximum pooling processing to obtain multiple spectral features.

[0056] Here, the spectral vectors in the sample new task remote sensing image are input into the spectral feature extraction module of the multidimensional convolutional neural network layer, and each spectral vector is subjected to multiple convolution processing and multiple maximum pooling processing to obtain multiple spectral features.

[0057] In order to obtain the spectral domain information of the surface cover from a large number of remote sensing image bands, a deep spectral dimension CNN module (spectral feature extraction module) is designed. Usually, CNN is mainly composed of convolutional layers, pooling layers and nonlinear layers. The CNN module inputs the spectral vector To obtain the spectral features. l The spectral characteristics can be expressed as:

[0058]

[0059] in, represents the convolution operation, Indicates l -1 layer output spectral characteristics, and For the l The weights and bias parameters of the convolutional layer involved in training, represents the nonlinear activation function ReLU, k is the number of convolutional layers, represents the maximum pooling. The spectral features obtained are l It becomes more abstract and discriminative through the convolution operation.

[0060] Here, multispectral remote sensing images It can be represented by a three-dimensional data cube with a width of W, a height of H, and a number of spectral bands of B. In general, the classification task of remote sensing images mainly relies on the model to capture the distinctive spectral information to achieve accurate classification results. Assume that the spectral vector ,in i , j Represents the columns and rows of the original image, and B represents the number of channels of the remote sensing image. These parameters are used for convolution of the spectral dimension to obtain spectral features. In order to further obtain the distinguishing features of the input samples, the cropped three-dimensional tensor It is defined as a spatial slice (Patch) consisting of a central pixel and adjacent pixels, where w, h, and b represent the width, height, and number of channels of a slice, respectively. These parameters are used to capture spatial background information.

[0061] B: Input the spatial slices of the spatial background information in the sample new task remote sensing image into the spatial feature extraction module of the multi-dimensional convolutional neural network layer, and perform convolution processing and multiple maximum pooling processing on each of the spatial slices to obtain multiple spatial features.

[0062] Here, the spatial slices of the spatial background information in the sample new task remote sensing image are input into the spatial feature extraction module of the multidimensional convolutional neural network layer, and each of the spatial slices is subjected to convolution processing and multiple maximum pooling processing to obtain multiple spatial features.

[0063] In order to extract the internal spatial background information of remote sensing images, a spatial feature extraction module is designed. The spatial feature extraction module has a similar structure to the spectral feature extraction module. The module consists of a convolution layer, an activation layer, and a pooling layer. As input, a convolution kernel is used to slide the spatial slice in the spatial dimension. The spatial feature can be expressed as:

[0064]

[0065] in, represents the convolution operation, Is the current l spatial features, Represents the features of the output of the previous layer. And and It is l The trainable weights and bias parameters of the convolutional layers, represents the ReLU activation function, k is the number of convolutional layers, stands for max pooling.

[0066] C: Fusing the plurality of spatial features and the plurality of spectral features to obtain the new remote sensing image features.

[0067] Here, after the above process, the spectral features and deep features of the spatial level of the remote sensing image are obtained respectively, and the robust feature expression generated by the fusion of these two features allows more representative information to be retained, thereby further improving the classification performance. In order to simplify the calculation, the fusion of the spectral and spatial representative features extracted by the one-dimensional module and the two-dimensional module is used as the input of the softmax layer of the classification. The process of feature fusion is expressed as:

[0068]

[0069] in, represents the remote sensing image features that will be input to the classifier layer, Represents a feature concatenation operation.

[0070] Among them, the process of determining the characteristics of the old remote sensing images is consistent with the process of determining the characteristics of the new remote sensing images of the above samples, and will not be repeated in this part.

[0071] S102: Determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on the vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes spectral feature metric and spatial feature metric.

[0072] In this step, the cosine similarity between the new remote sensing image features and the old remote sensing image features is determined according to the vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model can obtain the initial remote sensing image classification model by performing feature metric learning according to the cosine similarity.

[0073] Here, in the process of continuous learning, as new data is continuously added, the model will inevitably "forget" important feature information of historical data. The most important goal of continuous learning is to retain the previously learned historical knowledge to the maximum extent while learning new classifications. Therefore, feature metric learning of the new and old models becomes crucial in order to maintain the consistency of the performance of the new and old models while minimizing the distance between the features extracted from the new and old models in the feature space. The most commonly used feature similarity metric is cosine similarity, which is calculated by the cosine angle of two vectors.

[0074] S103: Constructing a loss function based on maximizing the consistency between the new and old models, iteratively optimizing the network parameters of the initial remote sensing image classification model using the target loss value of the loss function, and generating a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task.

[0075] In this step, a loss function is constructed based on maximizing the consistency between the new and old models, and the target loss value of the loss function is used to iteratively optimize the network parameters of the initial remote sensing image classification model to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task.

[0076] In a possible implementation manner, the iterative optimization of the network parameters of the initial remote sensing image classification model using the target loss value of the loss function to generate a new cross-scene remote sensing image classification model includes:

[0077] (1): Check whether the target loss value is less than or equal to the preset loss value.

[0078] Here, it is detected whether the target loss value is less than or equal to the preset loss value.

[0079] Here, the preset loss values ​​corresponding to different tasks are different and are determined based on expert experience.

[0080] (2) If yes, stop changing the network parameters of the initial remote sensing image classification model and generate the new cross-scene remote sensing image classification model.

[0081] Here, if yes, stop changing the network parameters of the initial remote sensing image classification model and generate a new cross-scene remote sensing image classification model.

[0082] (3): If not, continue to change the network parameters of the initial remote sensing image classification model, and continue to train the initial remote sensing image classification model after the changed parameters using the old remote sensing image features and the new remote sensing image features until the target loss value is less than or equal to the preset loss value, and stop training to generate the new cross-scene remote sensing image classification model.

[0083] Here, if not, continue to change the network parameters of the initial remote sensing image classification model, and use the old remote sensing image features and the new remote sensing image features to continue training the initial remote sensing image classification model after the changed parameters until the target loss value is less than or equal to the preset loss value, and stop training to generate a new cross-scene remote sensing image classification model.

[0084] Among them, the content of a single execution of the model includes: cosine similarity of spectral features and spatial features, feature consistency loss function, category loss function and cross entropy loss function. When the threshold does not reach the set value, a loop is performed and the model parameter value is updated once each loop.

[0085] In a possible implementation manner, the target loss value is determined by the following steps:

[0086] a: Calculate the features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model based on the feature consistency loss function to determine the feature consistency loss value between the new and old models.

[0087] Here, the features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model are calculated according to the feature consistency loss function to determine the feature consistency loss value between the new and old models.

[0088] In a possible implementation manner, the feature consistency loss function is used to calculate the features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model to determine the feature consistency loss value between the new and old models, including:

[0089] I: performing vector cosine angle calculation on spatial features in the new remote sensing image features extracted by the initial remote sensing image classification model and spatial features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a first similarity value.

[0090] II: performing vector cosine angle calculation on the spectral features in the new remote sensing image features extracted by the initial remote sensing image classification model and the spectral features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a second similarity value.

[0091] Here, the feature extraction model includes both the spectral feature extraction module and the spatial feature extraction module, so it is necessary to measure the consistency of the new and old models from the perspective of the above two features. The measurement calculation of these two features is as follows:

[0092]

[0093]

[0094] in, N is the number of training samples in a batch, L is the number of intermediate model feature layers, and Represent the first l Layer features, is the first similarity value, is the second similarity value.

[0095] III: Determine the feature consistency loss value based on the feature consistency loss function for the first similarity value and the second similarity value.

[0096] Here, a feature consistency loss value is determined based on the first similarity value and the second similarity value according to the feature consistency loss function.

[0097] In order to maintain the learned knowledge in the new remote sensing image classification model, a metric loss function is constructed by maximizing the consistency of the spectral and spatial features of the new and old models. The feature consistency loss function is designed as follows:

[0098]

[0099]

[0100] in, is the feature consistency loss value, λ represents the similarity weight, which is usually set to 0.5, and the value of the loss function varies from 1 to 0, because the value of cosine similarity varies from 0 to 1. The closer the value of the feature consistency loss function is to 0, the stronger the consistency between the new and old model features.

[0101] b: Based on the category loss function, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated to determine the category loss value between the new and old models.

[0102] Here, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated according to the category loss function to determine the category loss value between the new and old models.

[0103] In a possible implementation manner, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated based on the category loss function to determine the category loss value between the new and old models, including:

[0104] The first task remote sensing image is input into the initial remote sensing image classification model to output a first classification result; the first task remote sensing image is input into the old cross-scene remote sensing image classification model to output a second classification result; the first classification result and the second classification result are calculated based on the category loss function to determine the category loss value.

[0105] Here, the first task remote sensing image is respectively input into the initial remote sensing image classification model and the old cross-scene remote sensing image classification model to obtain the first classification result and the second classification result respectively; the first classification result and the second classification result are calculated according to the category loss function to determine the category loss value.

[0106] Among them, the similarity matching of feature maps at different levels between the historical model and the new model retains historical knowledge. In addition, the prediction results between the old and new models should be equally consistent. Therefore, the category loss function is designed as follows:

[0107]

[0108] in, N is the number of training samples in a batch, and Represent the outputs of the new and old models respectively, and are the parameters corresponding to the new and old models respectively, is the class loss value. Compared with the feature-based metric loss, the class loss effectively acquires knowledge by minimizing the difference between the prediction values ​​of the new and old models.

[0109] c: Calculate the classification result of the initial remote sensing image classification model for the current task remote sensing image based on the cross entropy loss function to determine the cross entropy loss value of the initial remote sensing image classification model.

[0110] Here, the classification result of the initial remote sensing image classification model for the current task remote sensing image is calculated according to the cross entropy loss function to determine the cross entropy loss value of the initial remote sensing image classification model.

[0111] In a possible implementation manner, the calculating the classification result of the initial remote sensing image classification model for the current task remote sensing image based on the cross entropy loss function to determine the cross entropy loss value of the initial remote sensing image classification model includes:

[0112] The current task remote sensing image is input into the initial remote sensing image classification model, and the predicted category label of the current task remote sensing image is output; the real category label and the predicted category label of the current task remote sensing image are calculated based on the cross entropy loss function to determine the cross entropy loss value.

[0113] Here, the current task remote sensing image is input into the initial remote sensing image classification model, and the predicted category label of the current task remote sensing image is output; the real category label and the predicted category label of the current task remote sensing image are calculated according to the cross entropy loss function to determine the cross entropy loss value.

[0114] Among them, in remote sensing image classification tasks, the cross entropy loss function is often used to measure the probability distribution difference between the model prediction and the ground truth category. and corresponds to the true label Space slice , the softmax layer converts the final features Converted into corresponding categories, the process is expressed as follows:

[0115]

[0116] in, is the prediction result of the model, C Represents the number of categories, and are the weights and bias parameters of the softmax layer.

[0117] Here, the cross entropy loss function formula for the current task is as follows:

[0118]

[0119] in, N represents the number of training samples in a batch. Then, the category c It keeps growing as new tasks are added. For example, the model has been trained on all previous t-1 remote sensing image tasks and has corresponding categories So for the current taskt , the final category C is defined as , which is the sum of all task categories, is the true category label, is the cross entropy loss value.

[0120] d: Add the feature consistency loss value, the category loss value, and the cross entropy loss value to determine the target loss value.

[0121] Here, the continuous learning loss of the cross-scene remote sensing image classification task consists of feature consistency loss function, category loss function and cross entropy function. In a constantly changing task sequence, the continuous learning loss function used to optimize the current remote sensing image task model can be expressed as follows:

[0122]

[0123] Among them, the feature consistency loss function Hierarchical knowledge preservation for spectral and spatial features. This item enhances the stability of cross-scene category features by maintaining the consistency between the spectral and spatial features extracted by the new and old models, and retains the knowledge of the previous model. , using higher-level semantic information to easily and effectively transfer from historical models to the current model. As a typical loss function, the cross entropy loss function Used to learn new knowledge, so that the model learning process constantly measures the gap between the prediction and the ground truth of the model, is the target loss value.

[0124] In a specific embodiment, the collected remote sensing data is used to extract the ground object information of cross-scene remote sensing images. First, the remote sensing image data of new and old tasks and their training samples are input, and the spectral-spatial feature extraction framework based on a multi-dimensional convolutional neural network is used to obtain spatial features and spectral features. Then, the spectral-spatial feature measurement mechanism is used to obtain stable deep features and cross-scene information by applying feature measurement. At the same time, the model is continuously updated by constructing a feature consistency loss function, a category loss function, and a cross entropy loss function. Finally, the cross-scene remote sensing images are automatically classified according to the updated model.

[0125] In this application, a typical sample data set is selected for model training, and the model parameters are iteratively optimized according to the changes in the loss function terms. After the model training is completed, the remote sensing image in the new scene is input to achieve automatic classification. By constructing a cross-scene remote sensing image continuous learning model, the model can be iterated and the parameters updated according to the characteristics of remote sensing images in different scenes to achieve accurate classification of remote sensing images.

[0126] The embodiment of the present application provides a continuous learning method for a cross-scene remote sensing image classification model, the continuous learning method comprising: extracting features of a sample new task remote sensing image and an old task remote sensing image based on a multidimensional convolutional neural network layer of an old cross-scene remote sensing image classification model, and determining new remote sensing image features of the sample new task remote sensing image and old remote sensing image features of the old task remote sensing image; determining the cosine similarity between the new remote sensing image features and the old remote sensing image features based on a vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric; constructing a loss function based on maximizing the consistency between the new and old models, and using the target loss value of the loss function to iteratively optimize the network parameters of the initial remote sensing image classification model to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model automatically classifies the new task remote sensing images. This will improve the robustness and practicality of remote sensing image classification models across scenarios, and enable remote sensing classification models to achieve continuous learning and updating under the massive emergence of remote sensing data sets.

[0127] See also Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a continuous learning device for a cross-scene remote sensing image classification model provided in an embodiment of the present application; Figure 3 The second structural diagram of a continuous learning device for a cross-scene remote sensing image classification model provided in an embodiment of the present application. Figure 2 As shown in , the continuous learning device 200 includes:

[0128] A feature extraction module 210 is used to extract features of the sample new task remote sensing image and the old task remote sensing image based on the multidimensional convolutional neural network layer of the old cross-scene remote sensing image classification model, and determine the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image;

[0129] The feature metric learning module 220 is used to determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on the vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric;

[0130] The training module 230 is used to construct a loss function based on maximizing the consistency between the new and old models, and use the target loss value of the loss function to iteratively optimize the network parameters of the initial remote sensing image classification model to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task.

[0131] Further, when the feature extraction module 210 is used to extract features of the sample new task remote sensing image and the old task remote sensing image based on the multidimensional convolutional neural network layer of the old cross-scene remote sensing image classification model for the sample new remote sensing image features and determine the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image, the feature extraction module 210 is specifically used to:

[0132] Inputting the spectral vectors in the sample new task remote sensing image into the spectral feature extraction module of the multi-dimensional convolutional neural network layer, performing multiple convolution processing and multiple maximum pooling processing on each of the spectral vectors to obtain multiple spectral features;

[0133] Inputting the spatial slices of the spatial background information in the sample new task remote sensing image into the spatial feature extraction module of the multi-dimensional convolutional neural network layer, performing convolution processing and multiple maximum pooling processing on each of the spatial slices to obtain multiple spatial features;

[0134] The multiple spatial features and the multiple spectral features are fused to obtain the new remote sensing image feature.

[0135] Further, when the training module 230 is used to iteratively optimize the network parameters of the initial remote sensing image classification model using the target loss value of the loss function to generate a new cross-scene remote sensing image classification model, the training module 230 is specifically used to:

[0136] Detecting whether the target loss value is less than or equal to a preset loss value;

[0137] If yes, stop changing the network parameters of the initial remote sensing image classification model and generate the new cross-scene remote sensing image classification model;

[0138] If not, continue to change the network parameters of the initial remote sensing image classification model, and use the old remote sensing image features and the new remote sensing image features to continue training the initial remote sensing image classification model after the changed parameters, until the target loss value is less than or equal to the preset loss value, then stop training and generate the new cross-scene remote sensing image classification model.

[0139] Further, such as Figure 3As shown, the continuous learning device 200 further includes a loss function determination module 240, and the loss function determination module 240 determines the target loss value through the following steps:

[0140] The features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model are calculated based on the feature consistency loss function to determine the feature consistency loss value between the new and old models;

[0141] Based on the category loss function, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated to determine the category loss value between the new and old models;

[0142] Calculating the classification result of the initial remote sensing image classification model for the current task remote sensing image based on the cross entropy loss function, and determining the cross entropy loss value of the initial remote sensing image classification model;

[0143] The feature consistency loss value, the category loss value, and the cross entropy loss value are added to determine the target loss value.

[0144] Further, when the loss function determination module 240 is used to calculate the features extracted from the old cross-scene remote sensing image classification model and the initial remote sensing image classification model based on the feature consistency loss function to determine the feature consistency loss value between the new and old models, the loss function determination module 240 is specifically used to:

[0145] Performing vector cosine angle calculation on spatial features in the new remote sensing image features extracted by the initial remote sensing image classification model and spatial features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a first similarity value;

[0146] Performing vector cosine angle calculation on the spectral features in the new remote sensing image features extracted by the initial remote sensing image classification model and the spectral features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a second similarity value;

[0147] The feature consistency loss value is determined based on the feature consistency loss function for the first similarity value and the second similarity value.

[0148] Further, when the loss function determination module 240 is used to calculate the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image based on the category loss function to determine the category loss value between the new and old models, the loss function determination module 240 is specifically used to:

[0149] Inputting the first task remote sensing image into the initial remote sensing image classification model, and outputting a first classification result;

[0150] Inputting the first task remote sensing image into the old cross-scene remote sensing image classification model, and outputting a second classification result;

[0151] The first classification result and the second classification result are calculated based on the category loss function to determine the category loss value.

[0152] Further, when the loss function determination module 240 is used to calculate the classification result of the current task remote sensing image by the initial remote sensing image classification model based on the cross entropy loss function to determine the cross entropy loss value of the initial remote sensing image classification model, the loss function determination module 240 is specifically used to:

[0153] Inputting the current task remote sensing image into the initial remote sensing image classification model, and outputting the predicted category label of the current task remote sensing image;

[0154] The real category label and the predicted category label of the remote sensing image of the current task are calculated based on the cross entropy loss function to determine the cross entropy loss value.

[0155] The embodiment of the present application provides a continuous learning device for a cross-scene remote sensing image classification model, and the continuous learning device includes: a feature extraction module, which is used to extract features of a sample new task remote sensing image and an old task remote sensing image based on a multidimensional convolutional neural network layer of an old cross-scene remote sensing image classification model, and determine the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image; a feature metric learning module, which is used to determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on a vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric; a training module, which is used to construct a loss function based on maximizing the consistency between the new and old models, and iteratively optimize the network parameters of the initial remote sensing image classification model using the target loss value of the loss function to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model automatically classifies the new task remote sensing image. This will improve the robustness and practicality of remote sensing image classification models across scenarios, and enable remote sensing classification models to achieve continuous learning and updating under the massive emergence of remote sensing data sets.

[0156] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0157] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the continuous learning method of the cross-scene remote sensing image classification model in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0158] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the continuous learning method of the cross-scene remote sensing image classification model in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0163] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0164] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A continuous learning method for a cross-scene remote sensing image classification model, characterized in that: The continuous learning method includes: The multidimensional convolutional neural network layer based on the old cross-scene remote sensing image classification model extracts the features of the sample new task remote sensing image and the old task remote sensing image, and determines the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image; Determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on the vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric; A loss function is constructed based on maximizing the consistency of the new and old models, and the target loss value of the loss function is used to iteratively optimize the network parameters of the initial remote sensing image classification model to generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task; With respect to the sample new remote sensing image features, the multidimensional convolutional neural network layer based on the old cross-scene remote sensing image classification model extracts features of the sample new task remote sensing image and the old task remote sensing image, and determines new remote sensing image features of the sample new task remote sensing image and old remote sensing image features of the old task remote sensing image, including: Inputting the spectral vectors in the sample new task remote sensing image into the spectral feature extraction module of the multi-dimensional convolutional neural network layer, performing multiple convolution processing and multiple maximum pooling processing on each of the spectral vectors to obtain multiple spectral features; Among them, the spectral feature extraction module is: in, represents the convolution operation, represents the spectral characteristics of the output of the l-1th layer, and are the weights and bias parameters involved in the training of the lth convolutional layer, represents the nonlinear activation function ReLU, k is the number of convolutional layers, represents maximum pooling, is the spectrum vector; Inputting the spatial slices of the spatial background information in the sample new task remote sensing image into the spatial feature extraction module of the multi-dimensional convolutional neural network layer, performing convolution processing and multiple maximum pooling processing on each of the spatial slices to obtain multiple spatial features; Among them, the spatial feature extraction module is: in, represents the convolution operation, is the current l-th spatial feature, represents the features of the output of the previous layer, and and are the trainable weights and bias parameters of the lth convolutional layer, represents the ReLU activation function, k is the number of convolutional layers, represents maximum pooling, is a three-dimensional tensor, w , h and b Respectively represent the width, height and number of channels of a slice, For the n A spatial slice; The multiple spatial features and the multiple spectral features are fused to obtain the new remote sensing image feature.

2. The continuous learning method according to claim 1, characterized in that: The method of iteratively optimizing the network parameters of the initial remote sensing image classification model by using the target loss value of the loss function to generate a new cross-scene remote sensing image classification model includes: Detecting whether the target loss value is less than or equal to a preset loss value; If yes, stop changing the network parameters of the initial remote sensing image classification model and generate the new cross-scene remote sensing image classification model; If not, continue to change the network parameters of the initial remote sensing image classification model, and use the old remote sensing image features and the new remote sensing image features to continue training the initial remote sensing image classification model after the changed parameters, until the target loss value is less than or equal to the preset loss value, then stop training and generate the new cross-scene remote sensing image classification model.

3. The continuous learning method according to claim 1, characterized in that: The target loss value is determined by the following steps: The features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model are calculated based on the feature consistency loss function to determine the feature consistency loss value between the new and old models; Based on the category loss function, the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image are calculated to determine the category loss value between the new and old models; Calculating the classification result of the initial remote sensing image classification model for the current task remote sensing image based on the cross entropy loss function, and determining the cross entropy loss value of the initial remote sensing image classification model; The feature consistency loss value, the category loss value, and the cross entropy loss value are added to determine the target loss value.

4. The continuous learning method according to claim 3, characterized in that: The feature consistency loss function is used to calculate the features extracted by the old cross-scene remote sensing image classification model and the initial remote sensing image classification model to determine the feature consistency loss value between the new and old models, including: Performing vector cosine angle calculation on spatial features in the new remote sensing image features extracted by the initial remote sensing image classification model and spatial features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a first similarity value; Performing vector cosine angle calculation on the spectral features in the new remote sensing image features extracted by the initial remote sensing image classification model and the spectral features in the old remote sensing image features extracted by the old cross-scene remote sensing image classification model to determine a second similarity value; The first similarity value and the second similarity value are calculated based on the feature consistency loss function to determine the feature consistency loss value.

5. The continuous learning method according to claim 3, characterized in that: The method of calculating the classification results of the old cross-scene remote sensing image classification model and the initial remote sensing image classification model for the same task remote sensing image based on the category loss function to determine the category loss value between the new and old models includes: Inputting the first task remote sensing image into the initial remote sensing image classification model, and outputting a first classification result; Inputting the first task remote sensing image into the old cross-scene remote sensing image classification model, and outputting a second classification result; The first classification result and the second classification result are calculated based on the category loss function to determine the category loss value.

6. The continuous learning method according to claim 3, characterized in that: The method of calculating the classification result of the current task remote sensing image by the initial remote sensing image classification model based on the cross entropy loss function to determine the cross entropy loss value of the initial remote sensing image classification model includes: Inputting the current task remote sensing image into the initial remote sensing image classification model, and outputting the predicted category label of the current task remote sensing image; The real category label and the predicted category label of the remote sensing image of the current task are calculated based on the cross entropy loss function to determine the cross entropy loss value.

7. A continuous learning device for a cross-scene remote sensing image classification model, characterized in that: The continuous learning device comprises: A feature extraction module is used to extract features of the sample new task remote sensing image and the old task remote sensing image based on the multidimensional convolutional neural network layer of the old cross-scene remote sensing image classification model, and determine the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image; A feature metric learning module is used to determine the cosine similarity between the new remote sensing image features and the old remote sensing image features based on a vector cosine angle calculation formula, so that the old cross-scene remote sensing image classification model performs feature metric learning based on the cosine similarity to obtain an initial remote sensing image classification model; wherein the feature metric includes a spectral feature metric and a spatial feature metric; A training module is used to construct a loss function based on maximizing the consistency of the new and old models, iteratively optimize the network parameters of the initial remote sensing image classification model using the target loss value of the loss function, and generate a new cross-scene remote sensing image classification model, so that the new cross-scene remote sensing image classification model can automatically classify the remote sensing images of the new task; When the feature extraction module is used to extract features of the sample new remote sensing image and the old task remote sensing image based on the multidimensional convolutional neural network layer of the old cross-scene remote sensing image classification model for the sample new task remote sensing image and the old task remote sensing image, and determines the new remote sensing image features of the sample new task remote sensing image and the old remote sensing image features of the old task remote sensing image, the feature extraction module is specifically used to: Inputting the spectral vectors in the sample new task remote sensing image into the spectral feature extraction module of the multi-dimensional convolutional neural network layer, performing multiple convolution processing and multiple maximum pooling processing on each of the spectral vectors to obtain multiple spectral features; Among them, the spectral feature extraction module is: in, represents the convolution operation, represents the spectral characteristics of the output of the l-1th layer, and are the weights and bias parameters involved in the training of the lth convolutional layer, represents the nonlinear activation function ReLU, k is the number of convolutional layers, represents maximum pooling, is the spectrum vector; Inputting the spatial slices of the spatial background information in the sample new task remote sensing image into the spatial feature extraction module of the multi-dimensional convolutional neural network layer, performing convolution processing and multiple maximum pooling processing on each of the spatial slices to obtain multiple spatial features; Among them, the spatial feature extraction module is: in, represents the convolution operation, is the current l-th spatial feature, represents the features of the output of the previous layer, and and are the trainable weights and bias parameters of the lth convolutional layer, represents the ReLU activation function, k is the number of convolutional layers, represents maximum pooling, is a three-dimensional tensor, w , h and b Respectively represent the width, height and number of channels of a slice, For the n A spatial slice; The multiple spatial features and the multiple spectral features are fused to obtain the new remote sensing image feature.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the continuous learning method of the cross-scene remote sensing image classification model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the continuous learning method of the cross-scene remote sensing image classification model as described in any one of claims 1 to 6 are executed.

Images