A remote sensing image incremental learning method and device based on polymerization energy distillation

Abstract

The application provides a remote sensing image incremental learning method and device based on polymerization energy distillation, and relates to the technical field of machine learning and artificial intelligence. The method comprises the following steps: inputting preset remote sensing images into a convolutional neural network model, a spatial channel joint representation module and the convolutional neural network model in sequence, extracting a plurality of feature vectors, selecting and retaining distinguished prototype features, and reducing the forgetting of old task knowledge when learning a new task. In the incremental learning of the classification of the output features, the plurality of final prototype features, the plurality of target prototype features of the memory bank and the preset prototype features are input into a classifier, effective knowledge transfer between the new and old tasks is realized, and the learning efficiency of the model on the new task is higher. The adaptive feature distillation loss function is used, so that the model can stably transfer the knowledge of the old task and adapt to the new task in the new task learning process, and the generalization ability of the model on the new task is improved. The prototype features are screened in the feature vectors for storage, and the storage cost can be reduced.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and artificial intelligence, and in particular to a method and apparatus for incremental learning of remote sensing images based on polymeric energy distillation. Background Technology

[0002] Remote sensing technology acquires detailed information about the Earth's surface, such as topography, landforms, vegetation, soil, water resources, and climate, through platforms like satellites, aircraft, and drones. With the surge in remote sensing image data, Intelligent Remote Sensing Systems (IRSS) have emerged. These systems utilize artificial intelligence methods, especially deep learning, to analyze remote sensing data and perform functions such as target detection and semantic segmentation. When faced with new tasks, such as different observation angles, altitudes, or complex background interference, IRSS systems often need to adjust their models to adapt to the new data distribution.

[0003] Currently, the incremental learning approach for remote sensing image classification tasks involves designing learning sequences from simple to complex, using similarity comparisons to optimize the learning order, introducing the concept of a memory bank to store key data samples from previous tasks, fine-tuning the new task based on the existing model, and using optimization algorithms to adjust the model parameters in order to minimize the loss function between the old and new tasks.

[0004] However, existing methods fail to effectively extract and transfer knowledge from old tasks, making it difficult for models to identify and utilize key features from old tasks in new tasks. This leads to catastrophic forgetting problems in the incremental learning process of intelligent remote sensing systems. To maintain the memory of old tasks, existing methods typically require storing large amounts of old task data, which not only increases storage costs but also poses challenges in data management. Existing methods are not stable enough in the learning process of new tasks because they rely too much on data from old tasks without fully considering the characteristics of new tasks, resulting in poor model performance on new tasks. Existing methods require readjusting model parameters for each new task, which leads to a waste of computational resources and reduced learning efficiency when processing large amounts of remote sensing data. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for incremental learning of remote sensing images based on polymeric energy distillation, which solves the problem of catastrophic forgetting in the incremental learning process of intelligent remote sensing systems, as well as the problems of poor model performance, low learning efficiency and high storage cost on new tasks.

[0006] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention provides a remote sensing image incremental learning method based on polymeric energy distillation, the method comprising:

[0008] Step S1: Acquire a preset remote sensing image;

[0009] Step S2: Input the preset remote sensing image into the convolutional neural network model, the spatial channel joint representation module and the convolutional neural network model in sequence to extract multiple feature vectors. The spatial channel joint representation module is a module that determines the corresponding weighted feature map and adjusts the weighted feature map of the cascaded feature map output by the convolutional neural network model. Each feature vector includes multiple elements.

[0010] Step S3: Set filtering conditions based on the number of multiple elements and feature vectors, and filter out multiple preset prototype features from multiple feature vectors according to the filtering conditions.

[0011] Step S4: Store multiple preset prototype features in the memory bank;

[0012] Step S5: Repeat steps S1 to S3 to obtain multiple new prototype features;

[0013] Step S6: Based on the adaptive feature distillation loss function, process multiple preset prototype features and multiple new prototype features in the memory bank to obtain multiple target prototype features. The adaptive feature distillation loss function includes memory loss, correction loss, distillation loss and cross-entropy loss.

[0014] Step S7: Store multiple target prototype features into the memory bank;

[0015] Step S8: Repeat steps S1 to S6 to extract multiple final prototype features;

[0016] Step S9: Input multiple final prototype features, multiple target prototype features in the memory, and multiple preset prototype features into the classifier, and output the classification of the features.

[0017] A second aspect of this application provides a remote sensing image incremental learning device based on polymeric energy distillation, the device comprising:

[0018] The acquisition module is used to acquire preset remote sensing images;

[0019] The first extraction module is used to sequentially input the preset remote sensing image into the convolutional neural network model, the spatial channel joint representation module, and the convolutional neural network model to extract multiple feature vectors. The spatial channel joint representation module is a module that determines the corresponding weighted feature map and adjusts the weighted feature map based on the cascaded feature map output by the convolutional neural network model. Each feature vector includes multiple elements.

[0020] The filtering module is used to set filtering conditions based on the number of multiple elements and feature vectors, and to filter out multiple preset prototype features from multiple feature vectors according to the filtering conditions.

[0021] The first storage module is used to store multiple preset prototype features into the memory bank;

[0022] The repeat module is used to repeat the operations of the acquisition module, the first extraction module, and the filtering module to obtain multiple new prototype features;

[0023] The processing module is used to process multiple preset prototype features and multiple new prototype features in the memory bank according to the adaptive feature distillation loss function to obtain multiple target prototype features. The adaptive feature distillation loss function includes memory loss, correction loss, distillation loss and cross-entropy loss.

[0024] The second storage module is used to store multiple target prototype features into the memory bank;

[0025] The second extraction module is used to repeat the operations of the acquisition module, the first extraction module, the filtering module, the first storage module, the repeating module, and the processing module to extract multiple final prototype features;

[0026] The classification module is used to input multiple final prototype features, multiple target prototype features from the memory, and multiple preset prototype features into the classifier, and output the classification of the features.

[0027] Compared to existing technologies, the incremental learning method and apparatus for remote sensing images based on synergistic energy distillation provided by this invention acquires a preset remote sensing image, sequentially inputs the preset remote sensing image into a convolutional neural network model, a spatial channel joint representation module, and another convolutional neural network model, extracts multiple feature vectors, sets filtering conditions based on the number of multiple elements and feature vectors, and selects multiple preset prototype features from the multiple feature vectors according to the filtering conditions, storing the multiple preset prototype features in a memory bank. That is, through the synergistic energy distillation operation, discriminative prototype features are selected and retained. These prototype features help reduce the forgetting of knowledge from old tasks when learning new tasks. By repeatedly filtering feature vectors, multiple new prototype features are obtained. Using an adaptive feature distillation loss function, multiple preset prototype features and new prototype features in the memory are processed to obtain multiple target prototype features, which are stored in the memory. This process of obtaining multiple target prototype features using the adaptive feature distillation loss function is repeated to extract multiple final prototype features. These final prototype features, along with the target and preset prototype features in the memory, are input into a classifier, which outputs the feature classification. Throughout the incremental learning process, the old target and preset prototype features are effectively transferred to the new final prototype features, thus transferring knowledge from the old task to the new task. This achieves effective knowledge transfer between the old and new tasks, resulting in higher learning efficiency for the model on new tasks. Using an adaptive feature distillation loss function to process the multiple preset and new prototype features in the memory ensures that the model can stably utilize and transfer knowledge from the old task while adapting to the new task. This fully considers the new task, leading to better model performance on new tasks and improving the model's generalization ability. It is not necessary to store all extracted feature vectors. Instead, it stores only a number of preset prototype features selected from all feature vectors and a number of target prototype features obtained by processing with an adaptive feature distillation loss function. This reduces storage requirements and thus storage costs. Attached Figure Description

[0028] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0029] Figure 1 A flowchart illustrating an incremental learning method for remote sensing images based on polymeric energy distillation is shown schematically.

[0030] Figure 2 A schematic diagram of a remote sensing image incremental learning device based on polymer energy distillation is shown. Detailed Implementation

[0031] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0032] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0033] The methods described in the embodiments of the present invention will be explained in detail below.

[0034] Figure 1 A flowchart illustrating the incremental learning method for remote sensing images based on polymeric energy distillation in an embodiment of the present invention is shown schematically. See [link to relevant documentation]. Figure 1 As shown, the method may include:

[0035] S1. Acquire preset remote sensing images.

[0036] The preset remote sensing images are the data source to be processed. There can be multiple preset remote sensing images. The preset remote sensing images are used as input to the convolutional neural network model.

[0037] S2. Input the preset remote sensing image into the convolutional neural network model, the spatial channel joint representation module and the convolutional neural network model in sequence to extract multiple feature vectors.

[0038] The spatial channel joint representation module is a module that determines the corresponding weighted feature map and adjusts the weighted feature map of the cascaded feature map output by the convolutional neural network model. Each feature vector includes multiple elements.

[0039] The convolutional neural network model and the spatial channel joint representation module can be collectively referred to as the feature prototype pre-extraction module. The convolutional neural network model can be the VGG-VD16 model. The VGG-VD16 model is a pre-trained convolutional neural network capable of capturing abstract features of images.

[0040] Specifically, the preset remote sensing image is sequentially input into the convolutional neural network model, the spatial channel joint representation module, and the convolutional neural network model to extract multiple feature vectors, including:

[0041] Step A1: Input the preset remote sensing image into the deep convolutional neural network model to extract the cascaded feature map of the preset remote sensing image.

[0042] The cascaded feature map includes feature maps from multiple channels.

[0043] Using a pre-trained VGG-VD16 model as the basis for feature extraction, a preset remote sensing image is input into the VGG-VD16 model, and the cascaded feature map of the preset remote sensing image is extracted from the input preset remote sensing image.

[0044] The extracted cascaded feature map is input into the spatial channel joint representation module, that is, the extracted cascaded feature map is used to determine the adjusted weighted feature map through the following steps A2 to A6.

[0045] Step A2: Calculate the spatial feature map based on the feature map of each channel.

[0046] The spatial feature map is the sum of the feature maps of all channels. The spatial feature map contains multiple pixels and the pixel value corresponding to each pixel.

[0047] Specifically, based on the feature maps of each channel, a spatial feature map is calculated, including:

[0048] Based on the feature maps of each channel in the cascaded feature map, the sum of the feature maps of all channels in the cascaded feature map, i.e., the spatial feature map, is calculated using the following formula:

[0049] S=∑ I X k ;

[0050] Where S is the spatial feature map, X k Let X be the feature map of the k-th channel in the cascaded feature map, and let X be the cascaded feature map, which can also be represented as X H×W×I H is the height of the cascaded feature map, W is the width of the cascaded feature map, and I is the number of channels in the cascaded feature map. S can be understood as the concatenation of all channel feature maps along the third dimension I.

[0051] Step A3: Determine the spatial weight of the pixel at the preset position based on the preset pixel value, the total pixel value, and the normalization parameter.

[0052] Among them, the preset pixel value is the value of the pixel at the preset position, and the total pixel value is the sum of the pixel values ​​of all pixels in the spatial feature map.

[0053] Specifically, based on the preset pixel value, the sum of pixel values, and the normalization parameters, the spatial weight of the pixel at the preset position is determined, including:

[0054] Based on the preset pixel value, the sum of pixel values, and the normalization parameter, the spatial weight of the pixel at the preset position is determined using the following first formula:

[0055]

[0056] Among them, S′xy S is the spatial weight of the pixel at a preset position (x, y) in the spatial feature map. xy The preset pixel value is the value of the pixel at the preset position (x, y) in the spatial feature map, ∑ m，n S mn Let be the sum of pixel values ​​of all pixels in the spatial feature map, a and b be normalization parameters, x be the horizontal coordinate of the pixel at the preset position, y be the vertical coordinate of the pixel at the preset position, m be the maximum value of the horizontal coordinate of the pixel in the spatial feature map, and n be the maximum value of the vertical coordinate of the pixel in the spatial feature map. The value range of x is [1, m], and the value range of y is [1, n].

[0057] The normalization parameters a and b are set to 0.5 and 2, respectively.

[0058] S′ xy The spatial weight is determined by the proportion of the pixel value at a preset position (x, y) in the spatial feature map to the total pixel value of all pixels in the entire spatial feature map. The resulting spatial weight of the pixel at the preset position represents the relative importance of each pixel.

[0059] Step A4: Adjust the spatial feature map according to the spatial weights to obtain the weighted feature map.

[0060] The weighted feature map includes multiple channels.

[0061] Specifically, the spatial feature map is adjusted according to the spatial weights to obtain a weighted feature map, including:

[0062] Using the second formula below, the spatial feature map is adjusted according to the spatial weights to obtain the weighted feature map:

[0063]

[0064] Where Φ is the weighted feature, S xy H1 is the preset pixel value, which is the value of the pixel at the preset position (x, y) in the spatial feature map. H1 is the height of the spatial feature map, and W1 is the width of the spatial feature map.

[0065] Adjusting the spatial feature map according to spatial weights can emphasize salient objects in the spatial feature map.

[0066] Step A5: Calculate the final weight of each channel in the weighted feature map based on the attributes of the cascaded feature map and the total number of channels in the weighted feature map.

[0067] The attributes of the cascaded feature map include preset height, preset width, and pixel values ​​for each channel.

[0068] Specifically, based on the attributes of the cascaded feature map and the total number of channels in the weighted feature map, the final weight of each channel in the weighted feature map is calculated, including:

[0069] Step A51: Calculate the sparsity of each channel in the cascaded feature map based on the preset height, preset width, and pixel values ​​of each channel.

[0070] The sparsity of each channel in a cascaded feature map refers to the proportion of non-zero elements in each channel, which helps to identify sparse features containing important information.

[0071] Specifically, based on the preset height, preset width, and pixel values ​​of each channel, the sparsity of each channel in the cascaded feature map is calculated using the following formula:

[0072]

[0073] Among them, Ω k Let H represent the sparsity of each channel in the cascaded feature map, i.e., the proportion of non-zero elements; H be the height of the cascaded feature map; W be the width of the cascaded feature map; i be the channel index used to traverse all channels in the cascaded feature map; j be the row index used to traverse each row of the cascaded feature map; and λ be the line index. ij The pixel value corresponding to the j-th row of the i-th channel index. This is an indicator function that returns 1 if the condition is true, otherwise returns 0, where k is the channel of the cascaded feature map.

[0074] In calculating Ω k Then, we can also use the following formula, based on Ω k Calculate the channel density of the cascaded feature map:

[0075] Θ k =1-Ω k ;

[0076] Where, Θ k The non-sparseness of the channels in the cascaded feature map, i.e. the proportion of zero elements, is used to adjust the channel importance in the feature representation.

[0077] Step A52: Calculate the final weights based on the sparsity of each channel and the total number of channels in the weighted feature map.

[0078] Specifically, based on the sparsity of each channel and the total number of channels in the weighted feature map, the final weights are calculated, including:

[0079] Based on the total number of channels in each sparse and weighted feature map, the final weights are calculated using the following third formula:

[0080]

[0081] Where, ω KFor the final weight, Ω k To represent the sparsity of each channel, K is the total number of channels in the weighted feature map, k is the number of channels in the cascaded feature map, h is the total number of channels in the cascaded feature map, the value of k ranges from [1, h], and δ is a constant used to prevent division by zero or to handle cases with extremely low sparsity.

[0082] Calculate the final weight ω for each channel K By considering the sparsity and weights of the channels and a small constant δ, different weights are assigned to different channels, and these weights will be used for weighted feature representation.

[0083] Step A6: Adjust the weighted feature map according to the final weights to obtain the adjusted weighted feature map.

[0084] The extracted cascaded feature map is input into the spatial channel joint representation module, which determines the adjusted weighted feature map through steps A2 to A6. The spatial channel joint representation module emphasizes important features in the image and reduces the influence of unimportant features through spatial weighting and channel weighting. The output is a weighted feature map with higher discriminative feature representation after weighting and filtering.

[0085] Step A7: Input the adjusted weighted feature map into the deep convolutional neural network model to extract multiple feature vectors.

[0086] The adjusted weighted feature map can be input into the VGG-VD16 model to extract multiple feature vectors.

[0087] S3. Set filtering conditions based on the number of multiple elements and feature vectors, and filter out multiple preset prototype features from multiple feature vectors according to the filtering conditions.

[0088] Each feature vector contains multiple elements.

[0089] Specifically, filtering conditions are set based on the number of multiple elements and feature vectors, and multiple preset prototype features are selected from the multiple feature vectors according to the filtering conditions, including:

[0090] Step B1: Determine the energy of each eigenvector based on multiple elements.

[0091] The energy of each feature vector represents its importance in the predefined remote sensing image. Feature vectors with high energy may contain more discriminative information, while feature vectors with low energy may contain less key information.

[0092] Specifically, the energy of each extracted feature vector is calculated, that is, based on multiple elements, the energy of each feature vector is determined using the following formula:

[0093]

[0094] Among them, F p ={f1, f2, ..., f p} is the feature vector extracted from the weighted feature map Φ, E(F p ) represents the energy of each eigenvector, and F represents the energy of each eigenvector. p The richness of information in f p For the eigenvector F p The p-th element, where P is the eigenvector F p The number of elements in the vector F, where p is the feature vector. p The label of the element.

[0095] Step B2: Determine the threshold parameter based on the energy and number of each eigenvector.

[0096] Specifically, the threshold parameter is determined based on the energy and number of each eigenvector, including:

[0097] Based on the energy and number of each eigenvector, the threshold parameter is determined using the following fourth formula:

[0098]

[0099] Where M is the threshold parameter, N is the number of feature vectors, and E(F) p F represents the energy of each eigenvector. p Let p be the eigenvector, and F be the eigenvector. p The label of the element.

[0100] The threshold parameter is obtained by averaging the energy E(F) of all eigenvectors. p The threshold parameter is determined by the threshold parameter and is used to select a representative feature prototype, i.e., a preset prototype feature.

[0101] Step B3: Set the filtering conditions based on the threshold parameter.

[0102] The filtering criteria set based on the threshold parameter are as follows:

[0103] M*0.7≤E(F p )≤M*1.3or E(F p )>M*1.7or E(F p ) <M*0.3;

[0104] Step B4: Based on the filtering criteria, select the preset prototype features from multiple feature vectors.

[0105] Specifically, based on the above screening criteria, preset prototype features whose energy falls within the range of the above screening criteria are selected from multiple feature vectors by using the energy of each feature vector.

[0106] These feature prototypes will only be selected and stored in memory when the energy value of a feature falls within a specific threshold range in the filtering criteria, or when the energy value is abnormally high or low.

[0107] S4. Store multiple preset prototype features into the memory bank.

[0108] Based on the energy calculation results, representative pre-defined prototype features are selected and stored in a memory bank. This memory bank acts as a knowledge base, preserving key information learned from previous tasks for use in subsequent tasks.

[0109] Steps S1 to S4 described above can be collectively referred to as the operation steps of polymer energy distillation.

[0110] S5. Repeat steps S1 to S3 to obtain multiple new prototype features.

[0111] Repeat steps S1 to S3, using a different remote sensing image than the one used in the first execution of step S1. S5 is for starting from a new task. n-1 Extract new prototype features. The remote sensing images in the new task are different from those in the old task, i.e., the remote sensing images in the first execution of step S1. The convolutional neural network model used is the same as that in step S1, which can be the VGG-VD16 model.

[0112] S6. Based on the adaptive feature distillation loss function, process multiple preset prototype features and multiple new prototype features in the memory to obtain multiple target prototype features.

[0113] The adaptive feature distillation loss function includes memory loss, correction loss, distillation loss, and cross-entropy loss.

[0114] Specifically, based on the adaptive feature distillation loss function, multiple preset prototype features and multiple new prototype features in the memory are processed to obtain multiple target prototype features, including:

[0115] Step C1: Based on memory loss, correction loss, and hyperparameters, determine the feature fitness loss using the following fifth formula:

[0116]

[0117] in, For characteristic adaptive loss, For memory loss, To correct the loss, α1 and α2 are hyperparameters, which are constants. The weights of memory loss and correction loss can be balanced by α1 and α2.

[0118] Step C2: Using the sixth formula below, determine the adaptive feature distillation loss function based on the feature fitness loss, distillation loss, and cross-entropy loss:

[0119]

[0120] in, For adaptive feature distillation loss function, For distillation losses, Cross-entropy loss serves as a constraint for learning new types of data, ensuring accuracy for the current task. cosine is the cosine similarity function used to calculate the similarity between two prototype features. q-1 For multiple pre-defined prototype features in the memory bank, F q This includes several new prototype features.

[0121] Step C3: Input multiple preset prototype features and multiple new prototype features from the memory bank into the adaptive feature distillation loss function for processing to obtain multiple target prototype features.

[0122] The adaptive feature distillation loss function uses multiple pre-set prototype features from the memory to help the model adapt to new tasks while retaining knowledge of old tasks.

[0123] S7. Store multiple target prototype features in the memory bank.

[0124] S8. Repeat steps S1 to S6 to extract multiple final prototype features.

[0125] Repeat steps S1 to S6, using different remote sensing images than those used in each execution of step S1 before S8. S8 is for starting from a new task. n Extract multiple final prototype features. The remote sensing images in the new task are different from those in the old task, i.e., the remote sensing images in each execution step S1 before executing S8.

[0126] S9. Input multiple final prototype features, multiple target prototype features from the memory bank, and multiple preset prototype features into the classifier, and output the classification of the features.

[0127] The classifier, located at the end of the entire operation, receives multiple final prototype features from the output of step S8 and combines them with knowledge stored in the memory bank—namely, multiple target prototype features and multiple preset prototype features—to make the final classification decision. The classifier can determine the predicted probability or confidence level of the category to which each prototype feature belongs based on all the input prototype features, and determine the most likely category for each prototype feature based on the predicted probability or confidence level, thus outputting the classification for each prototype feature.

[0128] The user can stop inputting preset remote sensing images according to their requirements.

[0129] The steps S5 to S9 described above can be collectively referred to as the operation steps of the incremental task.

[0130] Based on the above Figure 1 As can be seen from the implementation method, the embodiments of the present invention acquire a preset remote sensing image, input the preset remote sensing image sequentially into a convolutional neural network model, a spatial channel joint representation module, and a convolutional neural network model, extract multiple feature vectors, set filtering conditions based on the number of multiple elements and feature vectors, and select multiple preset prototype features from the multiple feature vectors according to the filtering conditions, and store the multiple preset prototype features in a memory bank. That is, by using a polymer energy distillation operation, discriminative prototype features are selected and retained. These prototype features help reduce the forgetting of knowledge of old tasks when learning new tasks. By repeatedly filtering feature vectors, multiple new prototype features are obtained. Using an adaptive feature distillation loss function, multiple preset prototype features and new prototype features in the memory are processed to obtain multiple target prototype features, which are stored in the memory. This process of obtaining multiple target prototype features using the adaptive feature distillation loss function is repeated to extract multiple final prototype features. These final prototype features, along with the target and preset prototype features in the memory, are input into a classifier, which outputs the feature classification. Throughout the incremental learning process, the old target and preset prototype features are effectively transferred to the new final prototype features, thus transferring knowledge from the old task to the new task. This achieves effective knowledge transfer between the old and new tasks, resulting in higher learning efficiency for the model on new tasks. Using an adaptive feature distillation loss function to process the multiple preset and new prototype features in the memory ensures that the model can stably utilize and transfer knowledge from the old task while adapting to the new task. This fully considers the new task, leading to better model performance on new tasks and improving the model's generalization ability. It is not necessary to store all extracted feature vectors. Instead, it stores only a number of preset prototype features selected from all feature vectors and a number of target prototype features obtained by processing with an adaptive feature distillation loss function. This reduces storage requirements and thus storage costs.

[0131] Based on the same inventive concept, as an implementation of the above-mentioned method for incremental learning of remote sensing images based on polymeric energy distillation, this embodiment of the invention also provides a device for incremental learning of remote sensing images based on polymeric energy distillation. Figure 2 This is a structural diagram of the device in an embodiment of the present invention. See also: Figure 2 As shown, the device may include:

[0132] The acquisition module 201 is used to acquire preset remote sensing images;

[0133] The first extraction module 202 is used to sequentially input the preset remote sensing image into the convolutional neural network model, the spatial channel joint representation module and the convolutional neural network model to extract multiple feature vectors. The spatial channel joint representation module is a module that determines the corresponding weighted feature map and adjusts the weighted feature map of the cascaded feature map output by the convolutional neural network model. Each feature vector includes multiple elements.

[0134] The filtering module 203 is used to set filtering conditions based on the number of multiple elements and feature vectors, and to filter out multiple preset prototype features from multiple feature vectors based on the filtering conditions.

[0135] The first storage module 204 is used to store multiple preset prototype features into a memory bank;

[0136] The repeat module 205 is used to repeat the operations of the acquisition module, the first extraction module, and the filtering module to obtain multiple new prototype features;

[0137] Processing module 206 is used to process multiple preset prototype features and multiple new prototype features in the memory bank according to the adaptive feature distillation loss function to obtain multiple target prototype features. The adaptive feature distillation loss function includes memory loss, correction loss, distillation loss and cross-entropy loss.

[0138] The second storage module 207 is used to store multiple target prototype features into the memory bank;

[0139] The second extraction module 208 is used to repeat the operations of the acquisition module, the first extraction module, the filtering module, the first storage module, the repeating module, and the processing module to extract multiple final prototype features;

[0140] The classification module 209 is used to input multiple final prototype features, multiple target prototype features in the memory, and multiple preset prototype features into the classifier and output the classification of features.

[0141] The first extraction module 202 is specifically used to input a preset remote sensing image into a deep convolutional neural network model to extract a cascaded feature map of the preset remote sensing image. The cascaded feature map includes feature maps of multiple channels. Based on the feature maps of each channel, a spatial feature map is calculated. The spatial feature map is the sum of the feature maps of all channels. The spatial feature map contains multiple pixels and their corresponding pixel values. Based on the preset pixel values, the sum of pixel values, and normalization parameters, the spatial weight of the pixel at a preset position is determined. The preset pixel value is the value of the pixel at the preset position, and the sum of pixel values ​​is the sum of the pixel values ​​of all pixels in the spatial feature map. The spatial feature map is adjusted according to the spatial weight to obtain a weighted feature map, which includes multiple channels. Based on the attributes of the cascaded feature map and the total number of channels in the weighted feature map, the final weight of each channel in the weighted feature map is calculated. The weighted feature map is adjusted according to the final weight to obtain an adjusted weighted feature map. The adjusted weighted feature map is input into the deep convolutional neural network model to extract multiple feature vectors.

[0142] The first extraction module 202 determines the spatial weight of the pixel at a preset position based on the preset pixel value, the sum of pixel values, and the normalization parameters, including:

[0143] Based on the preset pixel value, the sum of pixel values, and the normalization parameter, the spatial weight of the pixel at the preset position is determined using the following first formula:

[0144]

[0145] Among them, S′ xy S is the spatial weight of the pixel at a preset position (x, y) in the spatial feature map. xy For preset pixel values, ∑ m，n S mn Let be the sum of pixel values ​​of all pixels in the spatial feature map, a and b be normalization parameters, x be the horizontal coordinate of the pixel at the preset position, y be the vertical coordinate of the pixel at the preset position, m be the maximum value of the horizontal coordinate of the pixel in the spatial feature map, and n be the maximum value of the vertical coordinate of the pixel in the spatial feature map. The value range of x is [1, m], and the value range of y is [1, n].

[0146] The first extraction module 202 adjusts the spatial feature map according to the spatial weights to obtain a weighted feature map, including:

[0147] Using the second formula below, the spatial feature map is adjusted according to the spatial weights to obtain the weighted feature map:

[0148]

[0149] Where Φ is the weighted feature, S xyH1 is the preset pixel value, H1 is the height of the spatial feature map, and W1 is the width of the spatial feature map.

[0150] The first extraction module 202 calculates the final weight of each channel in the weighted feature map based on the attributes of the cascaded feature map and the total number of channels in the weighted feature map. This includes: calculating the sparsity of each channel in the cascaded feature map based on the preset height, preset width, and pixel values ​​of each channel; and calculating the final weight based on the sparsity of each channel and the total number of channels in the weighted feature map. The attributes of the cascaded feature map include the preset height, preset width, and pixel values ​​of each channel.

[0151] The first extraction module 202 calculates the final weights based on the sparsity of each channel and the total number of channels in the weighted feature map, including:

[0152] Based on the total number of channels in each sparse and weighted feature map, the final weights are calculated using the following third formula:

[0153]

[0154] Where, ω K For the final weight, Ω k To represent the sparsity of each channel, K is the total number of channels in the weighted feature map, δ is a constant, k is the number of channels in the cascaded feature map, h is the total number of channels in the cascaded feature map, and the value of k ranges from [1, h].

[0155] The filtering module 203 is specifically used to determine the energy of each feature vector based on multiple elements; determine the threshold parameter based on the energy and number of each feature vector; set the filtering conditions based on the threshold parameter; and filter out the preset prototype features from multiple feature vectors based on the filtering conditions, where each feature vector includes multiple elements.

[0156] The filtering module 203 determines the threshold parameters based on the energy and number of each feature vector, including:

[0157] Based on the energy and number of each eigenvector, the threshold parameter is determined using the following fourth formula:

[0158]

[0159] Where M is the threshold parameter, N is the number of feature vectors, and E(F) p F represents the energy of each eigenvector. p Let p be the eigenvector, and F be the eigenvector. p The label of the element.

[0160] Processing module 206 is specifically used to determine the feature fitness loss based on memory loss, correction loss, and hyperparameters using the following fifth formula:

[0161]

[0162] in, For characteristic adaptive loss, For memory loss, To correct for the loss, α1 and α2 are hyperparameters, which are constants;

[0163] Using the sixth formula below, the adaptive feature distillation loss function is determined based on the feature fitness loss, distillation loss, and cross-entropy loss:

[0164]

[0165] in, For adaptive feature distillation loss function, For distillation loss, For cross-entropy loss, cosine is the cosine similarity function, F q-1 For multiple pre-defined prototype features in the memory bank, F q For multiple new prototype features;

[0166] Multiple preset prototype features and multiple new prototype features from the memory are input into the adaptive feature distillation loss function for processing to obtain multiple target prototype features.

[0167] It should be noted that the above description of the embodiment of the remote sensing image incremental learning device based on polymeric energy distillation is similar to the description of the method embodiment above, and has similar beneficial effects. For technical details not disclosed in the embodiments of the remote sensing image incremental learning device based on polymeric energy distillation of this invention, please refer to the description of the method embodiment of this invention for understanding.

[0168] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A remote sensing image incremental learning method based on condensing energy distillation, characterized in that, The method includes: Step S1: Acquire a preset remote sensing image; Step S2: Input the preset remote sensing image into the convolutional neural network model, the spatial channel joint representation module and the convolutional neural network model in sequence to extract multiple feature vectors. The spatial channel joint representation module is a module that determines the corresponding weighted feature map and adjusts the weighted feature map of the cascaded feature map output by the convolutional neural network model. Each feature vector includes multiple elements. Step S3: Set filtering conditions according to the number of elements and feature vectors, and filter out multiple preset prototype features from the multiple feature vectors according to the filtering conditions; Step S4: Store the multiple preset prototype features in the memory bank; Step S5: Repeat steps S1 to S3 to obtain multiple new prototype features; Step S6: Process the multiple preset prototype features and the multiple new prototype features in the memory bank according to the adaptive feature distillation loss function to obtain multiple target prototype features. The adaptive feature distillation loss function includes memory loss, correction loss, distillation loss and cross-entropy loss. Step S7: Store the multiple target prototype features into the memory bank; Step S8: Repeat steps S1 to S6 to extract multiple final prototype features; Step S9: Input the multiple final prototype features, the multiple target prototype features in the memory bank, and the multiple preset prototype features into the classifier, and output the classification of the features.

2. The method according to claim 1, characterized in that, The process involves sequentially inputting the preset remote sensing image into a convolutional neural network model, a spatial channel joint representation module, and the convolutional neural network model to extract multiple feature vectors, including: The preset remote sensing image is input into a deep convolutional neural network model to extract the cascaded feature map of the preset remote sensing image, which includes feature maps of multiple channels. Based on the feature maps of each channel, a spatial feature map is calculated. The spatial feature map is the sum of the feature maps of all channels. The spatial feature map contains multiple pixels and the pixel value corresponding to each pixel. The spatial weight of a pixel at a preset position is determined based on a preset pixel value, a sum of pixel values, and a normalization parameter. The preset pixel value is the value of the pixel at the preset position, and the sum of pixel values ​​is the sum of the pixel values ​​of all pixels in the spatial feature map. The spatial feature map is adjusted according to the spatial weights to obtain a weighted feature map, which includes multiple channels. Based on the attributes of the cascaded feature map and the total number of channels in the weighted feature map, calculate the final weight of each channel in the weighted feature map; The weighted feature map is adjusted according to the final weight to obtain the adjusted weighted feature map; The adjusted weighted feature map is input into the deep convolutional neural network model to extract multiple feature vectors.

3. The method according to claim 2, characterized in that, The step of determining the spatial weight of a pixel at a preset position based on a preset pixel value, a sum of pixel values, and a normalization parameter includes: Based on the preset pixel value, the summed pixel value, and the normalization parameter, the spatial weight of the pixel at the preset position is determined using the following first formula: Among them, S′ xy S represents the spatial weight of the pixel at the preset position (x, y) in the spatial feature map. xy For the preset pixel value, ∑ m，n S mn The sum of pixel values ​​of all pixels in the spatial feature map is given by , where a and b are normalization parameters, x is the horizontal coordinate of the pixel at the preset position, y is the vertical coordinate of the pixel at the preset position, m is the maximum value of the horizontal coordinate of the pixel in the spatial feature map, and n is the maximum value of the vertical coordinate of the pixel in the spatial feature map. The range of x is [1, m], and the range of y is [1, n].

4. The method according to claim 3, characterized in that, The step of adjusting the spatial feature map according to the spatial weights to obtain a weighted feature map includes: The weighted feature map is obtained by adjusting the spatial feature map according to the spatial weights using the following second formula: Where Φ is the weighted feature, S xy H1 is the preset pixel value, H1 is the height of the spatial feature map, and W1 is the width of the spatial feature map.

5. The method according to claim 2, characterized in that, The attributes of the cascaded feature map include a preset height, a preset width, and pixel values ​​for each channel. The step of calculating the final weight of each channel in the weighted feature map based on the attributes of the cascaded feature map and the total number of channels in the weighted feature map includes: Based on the preset height, the preset width, and the pixel values ​​of each channel, the sparsity of each channel in the cascaded feature map is calculated. The final weight is calculated based on the sparsity of each channel and the total number of channels in the weighted feature map.

6. The method according to claim 5, characterized in that, The step of calculating the final weight based on the sparsity of each channel and the total number of channels in the weighted feature map includes: Based on the sparsity and the total number of channels in the weighted feature map, the final weights are calculated using the following third formula: Where, ω K For the final weight, Ω k The sparsity of each channel is given by K, the total number of channels in the weighted feature map is given by δ, a constant is given by k, the number of channels in the cascaded feature map is given by h, and the value of k ranges from [1, h].

7. The method according to claim 1, characterized in that, Each feature vector includes multiple elements. The step involves setting filtering conditions based on the number of elements and feature vectors, and then filtering out multiple preset prototype features from the multiple feature vectors according to the filtering conditions, including: The energy of each feature vector is determined based on the multiple elements; The threshold parameter is determined based on the energy of each feature vector and the number of feature vectors. Set the filtering conditions according to the threshold parameter; Based on the filtering criteria, preset prototype features are selected from the plurality of feature vectors.

8. The method according to claim 7, characterized in that, The step of determining the threshold parameter based on the energy of each feature vector and the number of feature vectors includes: Based on the energy of each feature vector and the number of feature vectors, the threshold parameter is determined using the following fourth formula: Where M is the threshold parameter, N is the number of feature vectors, and E(F) p F represents the energy of each eigenvector. p Let p be the feature vector F. p The label of the element.

9. The method according to claim 1, characterized in that, The process involves processing the multiple preset prototype features and the multiple new prototype features in the memory bank according to the adaptive feature distillation loss function to obtain multiple target prototype features, including: Based on the memory loss, the correction loss, and the hyperparameters, the feature adaptation loss is determined using the following fifth formula: in, For the aforementioned feature adaptive loss, For the aforementioned memory loss, The correction loss is α1 and α2, which are hyperparameters and are constants. Using the sixth formula below, the adaptive feature distillation loss function is determined based on the feature fitness loss, the distillation loss, and the cross-entropy loss: in, The adaptive feature distillation loss function is... For the distillation loss, For the cross-entropy loss, cosine is the cosine similarity function, F q-1 For the multiple preset prototype features in the memory bank, F q These are the multiple new prototype features; The multiple preset prototype features and the multiple new prototype features in the memory are input into the adaptive feature distillation loss function for processing to obtain the multiple target prototype features.

10. A remote sensing image incremental learning device based on polymeric energy distillation, characterized in that, The device includes: The acquisition module is used to acquire preset remote sensing images; The first extraction module is used to sequentially input the preset remote sensing image into the convolutional neural network model, the spatial channel joint representation module, and the convolutional neural network model to extract multiple feature vectors. The spatial channel joint representation module is a module that determines the corresponding weighted feature map and adjusts the weighted feature map of the cascaded feature map output by the convolutional neural network model. Each feature vector includes multiple elements. The filtering module is used to set filtering conditions based on the number of the multiple elements and feature vectors, and to filter out multiple preset prototype features from the multiple feature vectors according to the filtering conditions. The first storage module is used to store the plurality of preset prototype features into a memory bank; The repeating module is used to repeat the operations of the acquisition module, the first extraction module, and the filtering module to obtain multiple new prototype features; The processing module is used to process the multiple preset prototype features and the multiple new prototype features in the memory bank according to the adaptive feature distillation loss function to obtain multiple target prototype features. The adaptive feature distillation loss function includes memory loss, correction loss, distillation loss and cross-entropy loss. The second storage module is used to store the multiple target prototype features into the memory bank; The second extraction module is used to repeat the operations of the acquisition module, the first extraction module, the filtering module, the first storage module, the repeating module, and the processing module to extract multiple final prototype features; The classification module is used to input the multiple final prototype features, the multiple target prototype features in the memory, and the multiple preset prototype features into the classifier, and output the classification of the features.

Images