An Open Set Incremental Remote Sensing Image Scene Classification Method and Device

Abstract

The present invention provides an open-set incremental remote sensing image scene classification method and device. The method comprises: obtaining a remote sensing image dataset to be scene classified, inputting the dataset into a feature extraction model, and obtaining a feature mapping result for each remote sensing image in the dataset; inputting the feature mapping result of the remote sensing image into an open-set recognition model, and obtaining an open-set scene classification result for each remote sensing image; the open-set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown; wherein both the feature extraction model and the open-set recognition model are trained based on an original sample set; when the open-set scene classification result indicates that the scene category of the remote sensing image is unknown, the remote sensing image with the unknown scene category is further merged with the original sample set to obtain a current sample set, and the feature extraction model and the open-set recognition model are respectively updated based on the current sample set. The present invention can realize the recognition and learning of remote sensing images of unknown scene categories.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an open-set incremental remote sensing image scene classification method and device. Background Art

[0002] Remote sensing image scene classification involves interpreting and analyzing the content of remote sensing scene images to classify them into specific scene categories, such as tennis courts, airports, and overpasses. With the continuous development of deep learning, particularly convolutional neural networks, image recognition tasks have achieved remarkable results. Convolutional neural networks have also been widely used in remote sensing image scene classification, but the vast majority of existing remote sensing image scene classification methods are closed-set classifications, meaning that the scene categories of the remote sensing images to be classified must be those that were trained during the training phase. However, in real-world applications, the environment is often open and new scene categories are constantly emerging. When a trained recognition system encounters an unlearned category (an unknown category), it will classify it as a known category. For example, a classifier trained on tennis courts and airports may classify an overpass as either a tennis court or an airport, significantly impacting recognition accuracy. Therefore, recognition systems need to automatically detect unknown categories and continuously learn new ones to achieve better recognition results. However, most existing remote sensing image scene classification methods lack the ability to recognize and learn unknown categories. Furthermore, the samples used to learn new categories are manually collected, which cannot guarantee the accuracy and efficiency of remote sensing image scene classification. Summary of the Invention

[0003] The present invention provides an open-set incremental remote sensing image scene classification method and device, which are used to solve the defect of the existing technology that it is unable to recognize and learn unknown scene categories, and realize the recognition and learning of remote sensing images of unknown scene categories.

[0004] The present invention provides an open set incremental remote sensing image scene classification method, comprising: obtaining a remote sensing image data set to be subjected to scene classification;

[0005] Inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset;

[0006] Inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0007] In which, the feature extraction model and the open set recognition model are both trained based on the original sample set; when the open set scene classification result is that the scene category of the remote sensing image is unknown, the remote sensing image with unknown scene category is also merged with the original sample set to obtain the current sample set, and the feature extraction model and the open set recognition model are updated respectively based on the current sample set.

[0008] According to an open set incremental remote sensing image scene classification method provided by the present invention, the training method of the feature extraction model includes:

[0009] Construct a loss function for the feature extraction model, and based on the loss function, use a prototype learning method to train the feature extraction model; the loss function includes a cross entropy loss term and a regularization term, the cross entropy loss term is used to constrain the distance between each sample data in the original sample set; the regularization term is used to constrain the distribution space of each scene category in the original sample set; wherein, the loss function As shown in formula 1:

[0010]

[0011] Where, is the cross entropy loss term, is the regularization term, λ R is a hyperparameter; among them, The expressions of are shown in Equation 2 and Equation 3 respectively:

[0012]

[0013]

[0014] Where, ψ() is the feature mapping function of the feature extraction model; x is the sample data in the original sample set; c k 、c i are the category centers of scene category k and scene category i respectively; d() is the Euclidean distance; γ is a hyperparameter; K is the number of scene categories in the original sample set; p() is the probability function; y is the output of the feature extraction model; n k is the number of samples corresponding to scene category k; R k is the radius parameter; x j is the j-th sample data corresponding to scene category k.

[0015] According to an open set incremental remote sensing image scene classification method provided by the present invention, before inputting the feature mapping result of the remote sensing image into the open set recognition model, the method further includes:

[0016] The feature mapping results of the remote sensing images are input into a closed set classifier to obtain a closed set scene classification result of each remote sensing image; wherein the closed set classifier is trained based on the original sample set.

[0017] According to an open set incremental remote sensing image scene classification method provided by the present invention, the feature mapping results of the remote sensing images are input into an open set recognition model to obtain the open set scene classification results of each remote sensing image, including:

[0018] According to the closed-set scene classification result, the feature mapping result of the remote sensing image is input into the open-set recognition sub-model of the corresponding scene category to obtain the open-set scene classification result of each remote sensing image; wherein, the open-set recognition model includes several open-set recognition sub-models, and several open-set recognition sub-models are respectively used to identify the scene category of the remote sensing images of different scene categories.

[0019] According to an open set incremental remote sensing image scene classification method provided by the present invention, the feature mapping results of the remote sensing images are input into the open set recognition sub-model of the corresponding scene category to obtain the open set scene classification results of each remote sensing image, including:

[0020] The open set recognition sub-model is used to identify whether the remote sensing image belongs to the scene category corresponding to the open set recognition sub-model. If yes, the open set scene classification result is the scene category corresponding to the open set recognition sub-model; if not, the open set scene classification result is that the scene category of the remote sensing image is unknown.

[0021] According to an open set incremental remote sensing image scene classification method provided by the present invention, before merging the remote sensing image with unknown scene category with the original sample set, the method further includes:

[0022] The method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set; wherein the method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set; wherein, the method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set.

[0023] Convex hull construction step: constructing a d-dimensional convex hull in the sample data of the same scene category based on the feature mapping results of each sample data in the original sample set;

[0024] Convex hull decomposition step: performing convex hull decomposition on the d-dimensional convex hull to obtain a plurality of sub-convex hulls;

[0025] Sub-convex hull updating step: updating the sub-convex hull based on the number of sample data in the feature space corresponding to the sub-convex hull; repeating the convex hull decomposition step and the sub-convex hull updating step based on the updated sub-convex hull until a preset condition is met;

[0026] Data selection step: based on the importance of the sample data, selecting the sample data to be retained in the scene category from the sample data contained in each of the sub-convex hulls.

[0027] The present invention also provides an open-set incremental remote sensing image scene classification device, comprising:

[0028] A data acquisition module is used to obtain a remote sensing image dataset for scene classification;

[0029] a feature extraction module, inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset;

[0030] an open set recognition module, configured to input the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0031] The model training module is used to train the feature extraction model and the open set recognition model based on the original sample set respectively; and is also used to merge the remote sensing image with unknown scene category with the original sample set to obtain a current sample set when the open set scene classification result is that the scene category of the remote sensing image is unknown, and to update the feature extraction model and the open set recognition model based on the current sample set respectively.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any of the above-described open-set incremental remote sensing image scene classification methods are implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described open-set incremental remote sensing image scene classification methods.

[0034] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned open-set incremental remote sensing image scene classification methods.

[0035] The open-set incremental remote sensing image scene classification method and device provided by the present invention perform feature extraction on the remote sensing image to be scene classified through a feature extraction model to obtain a feature mapping result, and input the feature mapping result into an open-set recognition model for scene classification. The scene classification result of the remote sensing image can be obtained or the scene category of the remote sensing image is an unknown scene category. The remote sensing image with an unknown scene category is merged with the original sample set and then the feature extraction model and the open-set recognition model are updated, so that the feature extraction model and the open-set recognition model can learn new scene categories, thereby being able to recognize new scene categories without forgetting the old scene categories, and samples of the new scene category can be automatically obtained according to the open-set recognition results without manual collection, effectively ensuring the accuracy and efficiency of remote sensing image scene classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 Schematic diagram of the process of the open set incremental remote sensing image scene classification method provided by the present invention;

[0038] Figure 2 It is a structural diagram of the open set incremental remote sensing image scene classification device provided by the present invention;

[0039] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0041] The following combination Figure 1 The open set incremental remote sensing image scene classification method of the present invention is described as follows: Figure 1 As shown, the open set incremental remote sensing image scene classification method of the present invention includes:

[0042] S100: Obtain a remote sensing image dataset for scene classification.

[0043] Specifically, remote sensing images typically include multiple scene categories during the capture process, such as tennis courts, airports, and overpasses. Therefore, it is necessary to classify the captured remote sensing images so that remote sensing images of corresponding scenes can be selected according to specific tasks in practical applications. Due to the limited number of samples in the sample set, it is impossible to cover all scene categories. Therefore, the remote sensing image dataset to be scene classified not only includes remote sensing images of old scene categories (i.e., scene categories included in the sample set during model training), but may also include remote sensing images of new scene categories (i.e., scene categories not included in the sample set during model training).

[0044] S200: Input the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset.

[0045] Specifically, the feature extraction model is used to extract features from the input remote sensing images. By inputting the remote sensing image dataset into the feature extraction model, the d-dimensional features of each remote sensing image can be output. The feature mapping function of the feature extraction model is: x is the sample data; wherein, the dimension of the feature can be determined according to the scene classification accuracy of the remote sensing image. The construction of the feature extraction network can be constructed through a machine learning network according to performance requirements. For example, the machine learning network can be a neural network. The neural network can be designed according to the requirements of remote sensing image feature extraction, or an existing designed neural network can be used. In the embodiment of the present invention, an 18-layer ResNet network is used, and the output dimension of the last fully connected layer is changed to 200, that is, d = 200 (which can be adjusted according to the scene classification accuracy). There is no requirement for the specific structure of the neural network here, as long as it can extract d-dimensional features of the remote sensing image. After obtaining the neural network, the output layer of the neural network is removed to complete the construction of the feature extraction model.

[0046] S300, inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0047] In which, the feature extraction model and the open set recognition model are both trained based on the original sample set; when the open set scene classification result is that the scene category of the remote sensing image is unknown, the remote sensing image with unknown scene category is also merged with the original sample set to obtain the current sample set, and the feature extraction model and the open set recognition model are updated respectively based on the current sample set.

[0048] Specifically, open set recognition (OSR) models can identify unknown categories and have been a research hotspot in machine learning and deep learning in recent years. Open set recognition algorithms can identify unknown categories without training samples of the unknown category, making them an effective recognition method in open environments.

[0049] In addition, the feature extraction model and the open set recognition model in the embodiment of the present invention are trained based on the original sample set, and the original sample set is {X 1 ,…,X K}, K is the number of scene categories in the original sample set, where the original sample set includes sample data of old scene categories. Therefore, after inputting the feature mapping results of the remote sensing image into the open set recognition model, for the old scene categories, it is possible to determine which scene category the remote sensing image belongs to. For the new scene categories, it is only known that the remote sensing image belongs to an unknown scene category. The original sample set can be selected according to actual needs. For example, the real remote sensing dataset NWPU-RESISC45 can be used. This dataset was collected by Northwestern Polytechnical University and contains 45 categories, 700 images per category, and a total of 31,500 images. The image size is 256*256. The samples are divided into training and test sets at a ratio of 4:1, and the center of the image is cropped to a size of 224*224.

[0050] At the same time, when it is recognized that the scene category of the remote sensing image is unknown, the remote sensing image with the unknown scene category is merged with the original sample set to form a new sample set, namely the current sample set. The feature extraction model and the open set recognition model are updated with the new sample set, so that the feature extraction model and the open set recognition model can learn the new scene category. The updated feature extraction model and the open set recognition model can recognize the new scene category without forgetting the old scene category. Before updating the feature extraction model and the open set recognition model with the new sample set, the remote sensing image with the unknown scene category needs to be labeled with the scene category. The specific scene category labeling method can be selected according to actual needs. For example, manual labeling can be used, or automatic labeling can be performed through machine learning methods. In existing remote sensing image scene classification methods, the sample data used to learn new categories are all collected manually, resulting in increased labor costs, low efficiency, and being affected by human factors, and the accuracy of model learning cannot be guaranteed. The embodiment of the present invention automatically identifies remote sensing images with unknown scene categories through an open set recognition model and adds them to the original sample set, thereby realizing the automatic collection of samples for new category learning and ensuring the efficiency and accuracy of model learning.

[0051] Thus, the embodiment of the present invention uses a feature extraction model to extract features from a remote sensing image to be scene classified, obtains a feature mapping result, and inputs the feature mapping result into an open set recognition model for scene classification, which can obtain a scene classification result of the remote sensing image or a scene category of the remote sensing image as an unknown scene category, and merges the remote sensing image with an unknown scene category with the original sample set to update the feature extraction model and the open set recognition model, so that the feature extraction model and the open set recognition model can learn new scene categories, thereby being able to recognize new scene categories without forgetting the old scene categories, and samples of the new scene category can be automatically obtained based on the open set recognition results without manual collection, effectively ensuring the accuracy and efficiency of remote sensing image scene classification. In addition, the remote sensing image scene classification method proposed by the present invention can be better applied to classification problems in open environments by combining open set recognition and incremental learning, has strong scalability, can be applied to various image recognition tasks, and is conducive to improving the efficiency of incremental learning.

[0052] Based on the above embodiment, the training method of the feature extraction model includes:

[0053] Construct a loss function for the feature extraction model, and based on the loss function, use a prototype learning method to train the feature extraction model; the loss function includes a cross entropy loss term and a regularization term, the cross entropy loss term is used to constrain the distance between each sample data in the original sample set; the regularization term is used to constrain the distribution space of each scene category in the original sample set; wherein, the loss function As shown in formula (1):

[0054]

[0055] Where, is the cross entropy loss term, is the regularization term, λ R (0<λ R ≤1) is a hyperparameter used to control The proportion of The expressions of are shown in formula (2) and formula (3):

[0056]

[0057]

[0058] Where, ψ() is the feature mapping function of the feature extraction model; x is the sample data in the original sample set; c k 、c iare the category centers of scene category k and scene category i respectively; d() is the Euclidean distance; γ (0<γ≤1) is a hyperparameter used to control the influence of distance; K is the number of scene categories in the original sample set; p() is the probability function; y is the output of the feature extraction model; p(y=k|x) represents the probability of outputting category y=k for input sample x; n k is the number of samples corresponding to scene category k; R k is the learnable radius parameter; x j is the j-th sample data corresponding to scene category k.

[0059] Specifically, after the feature extraction model is constructed, it is necessary to design a corresponding loss function to train the feature extraction model. The embodiment of the present invention adopts the prototype learning method to train the feature extraction model, that is, for each scene category k, k∈[1,K] in the feature space, a learnable category center c is set. k , with the sample data distance from the category center c k The Euclidean distance d(ψ(x),c k ) is used to measure the probability that the sample data belongs to scene category k, where Based on the Euclidean distance, the cross entropy loss term is designed In addition, in order to limit the distribution of sample data of each scene category in the feature space, a regularization term is introduced In this way, the sample data of the same scene category are constrained to be distributed within a certain range of space. The final loss function is shown in formula (1). Through this loss function, the distance between the sample data of the same scene category in the feature space is made smaller, and the distance between the sample data of different scene categories is made larger. At the same time, the sample data of each scene category are distributed in a limited area in the feature space, which is beneficial to the subsequent open set recognition. Through the loss function After training the feature extraction model, the final feature mapping function ψ can be obtained. The parameters in the feature extraction model training process can be set according to the actual situation. For example, if the feature extraction model is constructed by an 18-layer ResNet network, the network parameters of the feature extraction model can be initialized using the resnet18 network pre-training parameters in pytorch during training. The category center c k and radius parameter R k All are initialized to 1, and the Adam optimizer is used to optimize the network parameters of the feature extraction model, setting epochs = 30, batch size = 32, and radius parameter R k The learning rate of γ and λ is 0.01, and the learning rates of the other parameters are 0.0001. Rare 1 and 0.05 respectively. Based on the above parameters and the sample data in the original sample set, the feature extraction model is trained to obtain the final feature mapping function ψ.

[0060] It can be seen that the embodiment of the present invention constructs the loss function of the feature extraction model through the cross-entropy loss term and the regularization term, and adopts the prototype learning method to train the feature extraction model, which can make the distance between sample data of the same scene category in the feature space smaller, and the distance between sample data of different scene categories larger. At the same time, the sample data of each scene category is distributed in a limited area in the feature space, thereby improving the separability between the sample data, thereby effectively improving the accuracy of the classification results in the process of scene classification of remote sensing images based on feature mapping results.

[0061] Based on any of the above embodiments, before inputting the feature mapping result of the remote sensing image into the open set recognition model, the method further includes:

[0062] The feature mapping results of the remote sensing images are input into a closed set classifier to obtain closed set scene classification results of each remote sensing image; wherein, the closed set classifier is trained based on the original sample set; and the closed scene classification results are the known scene categories.

[0063] Specifically, the closed set classifier assumes that the input remote sensing image belongs to the old scene category and performs scene classification on each remote sensing image, that is, the classification results are all the old scene categories. The feature mapping results of the remote sensing image are input into the closed set classifier. For remote sensing images of the old scene category and remote sensing images of the new scene category, the classification results obtained are all the old scene categories. Among them, the specific type of the closed set classifier can be selected according to actual needs, for example, SVM (Support Vector Machine) classifier, K-nearest neighbor classifier. Among them, the closed set classifier is trained based on the feature mapping results of the original sample set. The specific training method can be set according to the actual situation. For example, for the SVM classifier, sklearn can be used for training, the classification strategy is "ovo", the penalty coefficient is 0.008, and the kernel is a linear kernel.

[0064] It can be seen that the embodiment of the present invention can perform preliminary identification of the scene category of the remote sensing image through the closed set classifier, providing a priori prediction categories for the subsequent open set recognition model. At the same time, the double prediction of the scene category of the remote sensing image through the closed set classifier and the open set recognition model can effectively improve the accuracy of the scene classification results of the remote sensing image with known scene categories.

[0065] Based on any of the above embodiments, inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image includes:

[0066] According to the closed-set scene classification result, the feature mapping result of the remote sensing image is input into the open-set recognition sub-model of the corresponding scene category to obtain the open-set scene classification result of each remote sensing image; wherein, the open-set recognition model includes several open-set recognition sub-models, and several open-set recognition sub-models are respectively used to identify the scene category of the remote sensing images of different scene categories.

[0067] Specifically, the open set recognition model includes several open set recognition sub-models, each open set recognition sub-model corresponds to a scene category, that is, the number of open set recognition sub-models is equal to the number of scene categories in the original sample set, and each open set recognition sub-model is used to recognize the remote sensing image of the scene category; according to the closed set scene classification result, the known scene category to which the remote sensing image belongs is obtained, and the feature mapping result of the remote sensing image is input into the open set recognition sub-model corresponding to the scene category to obtain the open set scene classification result of the remote sensing image.

[0068] It can be seen that in the embodiment of the present invention, a priori scene category prediction results are obtained based on the closed-set scene classification results, and the feature mapping results of the remote sensing image are input into the open-set recognition sub-model of the corresponding category based on the priori scene category prediction results, so that there is no need to compare each open-set recognition sub-model during the open-set recognition process, which effectively improves the efficiency of remote sensing image scene classification; at the same time, the closed-set scene classification results are further tested by the open-set recognition model, which effectively improves the accuracy of the scene classification results of remote sensing images with known scene categories.

[0069] Based on any of the above embodiments, inputting the feature mapping results of the remote sensing images into the open set recognition sub-model of the corresponding scene category to obtain the open set scene classification results of each remote sensing image includes:

[0070] The open set recognition sub-model is used to identify whether the remote sensing image belongs to the scene category corresponding to the open set recognition sub-model. If yes, the open set scene classification result is the scene category corresponding to the open set recognition sub-model; if not, the open set scene classification result is that the scene category of the remote sensing image is unknown.

[0071] Specifically, the construction of the open set recognition sub-model can be selected according to actual needs. For example, in the training process of the feature extraction model, the regularization term is introduced. RTo constrain the sample data of the same scene category to be distributed within a certain range of space, the distribution of each scene category in the feature space is bounded. One-class SVM (one-class support vector machine) can be used to establish a closed decision boundary for each scene category in the feature space, and obtain several open set recognition sub-models {M1,…M K For the sample data x, if its closed-set scene classification result is k, then its feature mapping result is input into the open-set recognition sub-model M k , identify the submodel M through the open set k Identify whether the remote sensing image belongs to scene category k; if the model outputs 1, that is, M k (ψ(x))=1, indicating that the sample data is within the boundary of scene category k, that is, the scene category of the sample data is k; if the model output is -1, that is, M k (ψ(x)) = -1, indicating that the sample data is not within the boundary of category k, and the scene category of the sample data is unknown. Among them, the parameters of the one-class SVM can be set according to the actual situation, for example, nu = 0.005, kernel = ′ rbf',gamma='scale'.

[0072] It can be seen that the open set recognition sub-model in the embodiment of the present invention can quickly and accurately determine whether the remote sensing image belongs to the scene category corresponding to the open set recognition sub-model by comparing the remote sensing image with its own decision boundary, thereby effectively improving the accuracy and efficiency of remote sensing image scene classification.

[0073] Based on any of the above embodiments, before merging the remote sensing image with unknown scene category with the original sample set, the method further includes:

[0074] The original sample set is screened based on a convex hull point selection method to obtain samples to be retained; the samples to be retained are used to be merged with the remote sensing image with unknown scene category to obtain the current sample set.

[0075] Specifically, the original sample set is screened by the convex hull point selection method to obtain the samples to be retained, and the samples to be retained are merged with the remote sensing images of unknown scene categories to obtain the current sample set. The feature extraction model, closed set classifier and open set recognition model are updated through the current sample set, and the incremental learning of the feature extraction model, closed set classifier and open set recognition model is realized. Through the incremental learning method, new categories can be continuously learned without retaining all the training samples of known categories, while retaining the recognition accuracy of the original categories, alleviating the catastrophic forgetting problem in the learning process of new categories. In addition, it can also effectively reduce the memory requirements and training time for storing samples.

[0076] The method of selecting points based on the convex hull to screen the original sample set includes:

[0077] S410, convex hull construction step: based on the feature mapping results of each sample data in the original sample set, construct a d-dimensional convex hull in the sample data of the same scene category; wherein the d-dimensional convex hull is a convex polyhedron including d+1 vertices, and the dimension of the convex hull is the same as the feature dimension extracted by the feature extraction model; specifically comprising:

[0078] S411. Randomly select a reference point in the feature space of the sample data of the scene category;

[0079] S412: In the feature space of the sample data of the scene category, select the point farthest from the reference point as the first vertex of the d-dimensional convex hull;

[0080] S413, constructing the selected vertices into a linear space;

[0081] S414. Selecting, in the feature space of the sample data of the scene category, the point farthest from the linear space;

[0082] Repeat steps S413-S414 until d+1 vertices of the d-dimensional convex hull are obtained, and the d+1 vertices are connected in pairs to construct a d-dimensional convex hull; for example, for a two-dimensional simplex, after obtaining the first vertex V1, select the point farthest from V1 as the second vertex V2; then select the point farthest from the line segment formed by V1 and V2 as the third vertex V3.

[0083] S420, convex hull decomposition step: performing convex hull decomposition on the d-dimensional convex hull to obtain a plurality of sub-convex hulls; specifically, connecting the center point of the d-dimensional convex hull with d+1 vertices respectively, and dividing the d-dimensional simplex into d+1 sub-convex hulls; for example, for a two-dimensional simplex, its structure is a triangle, and the lines connecting the center point of the triangle with the three vertices of the triangle divide the triangle into three small triangles, each small triangle being a sub-convex hull.

[0084] S430, sub-convex hull updating step: based on the number of sample data in the feature space corresponding to the sub-convex hull, the sub-convex hull is updated; based on the updated sub-convex hull, the convex hull decomposition step and the sub-convex hull updating step are repeated until a preset condition is met; wherein, the feature space of the sample data of the scene category is divided into d+1 non-overlapping sub-regions by d+1 rays from the center point of the d-dimensional convex hull to the d+1 vertices, and the feature space corresponding to the sub-convex hull is the sub-region to which the sub-convex hull belongs; specifically, the sub-region with the largest number of samples is selected, and The point farthest from the sub-convex hull in the sub-region is selected in the sub-region as a new vertex, and a new d-dimensional convex hull is formed based on the new vertex and the sub-convex hull (for example, the new vertex is V4, and the two vertices of the sub-convex hull are V1 and V2 in step S410, then vertices V1, V2 and V4 form a new convex hull); based on the new d-dimensional convex hull, steps S420-S430 are repeated until a preset condition is met; wherein the preset condition can be selected according to actual needs, for example, reaching a preset number of iterations, or the number of samples to be retained reaches a preset value.

[0085] S440, data selection step: based on the importance of the sample data, select the sample data to be retained in the scene category from the sample data contained in each of the sub-convex hulls; wherein, based on the sample data contained in all the sub-convex hulls generated in the iterative process of steps S420-S430, the sample data to be retained in the scene category is selected according to the importance of the sample data, that is, the order in which the samples are selected is related to the importance of the samples, the later the order, the lower the importance of the samples, so that the sample data with higher importance can be effectively retained; the samples selected by the convex hull point selection method constitute a representative sample set of the scene category.

[0086] S450, repeat steps S410-S440 to complete the sample screening of K scene categories. The original sample set after sample screening is {S 1 ,…,S K}.

[0087] After the filtered original sample set is merged with the remote sensing image of unknown scene category, the current sample set is {S 1 ,…,S K ,X K+1 ,…,X K+m}, where m is the number of new scene categories. In addition, after updating the feature extraction model, closed set classifier and open set recognition model through the current sample set, the new scene category sample set {X K+1 ,…,X K+m} to screen samples and obtain {S K+1 ,…,S K+m}; At the same time, {S 1 ,…,S K ,S K+1 ,…,S K+m}Further screening is performed, which means that some samples of the old scene categories will be replaced by samples of the new scene categories, further reducing the storage space of the sample set and ensuring the accuracy of the updated model classification.

[0088] It can be seen that the embodiment of the present invention filters the sample data in each scene category through the convex hull point selection method, which can retain the representative sample data in each scene category, thereby helping to control the number of samples, reduce the storage space of the sample set, and improve the classification accuracy of the model.

[0089] The effectiveness of the open-set incremental remote sensing image scene classification method of the present invention is further verified by comparative experiments below:

[0090] To date, there is no benchmark experiment for open-set incremental recognition methods. The embodiment of the present invention sets up a baseline experiment. Specifically, iGaRL is used as the incremental learning model, and a one-class SVM is used to establish a closed decision model in the feature space. The classifier in iGaRL provides the prior category, and then determines whether the sample is in the decision boundary of the category. If so, the sample belongs to the category; otherwise, the sample belongs to an unknown category.

[0091] Table 1 shows the average accuracy of the baseline experimental method and the proposed method in identifying unknown categories and known categories during the incremental learning process.

[0092] Table 1

[0093]

[0094] As shown in Table 1, the classification accuracy of the proposed method is significantly higher than that of the baseline method. Furthermore, in closed-set remote sensing image scene classification, existing methods, using a classifier trained on 15 known scene categories, are unable to classify remote sensing images with five unknown scene categories. However, the proposed method achieves an average accuracy of 86.5% in recognizing remote sensing images with 15 known scene categories and five unknown scene categories. Furthermore, these new categories can be learned through incremental learning.

[0095] The open set incremental remote sensing image scene classification device provided by the present invention is described below. The open set incremental remote sensing image scene classification device described below and the open set incremental remote sensing image scene classification method described above can be referenced to each other.

[0096] like Figure 2 As shown, the device includes:

[0097] The data acquisition module 210 is used to acquire a remote sensing image dataset to be used for scene classification;

[0098] A feature extraction module 220 is configured to input the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset;

[0099] An open set recognition module 230 is configured to input the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result for each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0100] The model training module 240 trains the feature extraction model and the open set recognition model based on the original sample set respectively; and is also used to merge the remote sensing image with unknown scene category with the original sample set to obtain a current sample set when the open set scene classification result is that the scene category of the remote sensing image is unknown, and to update the feature extraction model and the open set recognition model based on the current sample set respectively.

[0101] Based on the above embodiment, the training method of the feature extraction model in the model training module 240 includes:

[0102] Construct a loss function for the feature extraction model, and based on the loss function, use a prototype learning method to train the feature extraction model; the loss function includes a cross entropy loss term and a regularization term, the cross entropy loss term is used to constrain the distance between each sample data in the original sample set; the regularization term is used to constrain the distribution space of each scene category in the original sample set; wherein, the loss function As shown in formula (1):

[0103]

[0104] Where, is the cross entropy loss term, is the regularization term, λ R is a hyperparameter; among them, The expressions of are shown in formula (2) and formula (3):

[0105]

[0106]

[0107] Where, ψ() is the feature mapping function of the feature extraction model; x is the sample data in the original sample set; c k 、c iare the category centers of scene category k and scene category i respectively; d() is the Euclidean distance; γ is a hyperparameter; K is the number of scene categories in the original sample set; p() is the probability function; y is the output of the feature extraction model; n k is the number of samples corresponding to scene category k; R k is the radius parameter; x j is the j-th sample data corresponding to scene category k.

[0108] Based on any of the above embodiments, it also includes a closed set recognition module, which is used to input the feature mapping results of the remote sensing image into a closed set classifier to obtain a closed set scene classification result of each remote sensing image; wherein, the closed set classifier is trained based on the original sample set.

[0109] Based on any of the above embodiments, the open set recognition module 230 inputs the feature mapping results of the remote sensing images into the open set recognition model to obtain the open set scene classification results of each remote sensing image, including:

[0110] According to the closed-set scene classification result, the feature mapping result of the remote sensing image is input into the open-set recognition sub-model of the corresponding scene category to obtain the open-set scene classification result of each remote sensing image; wherein, the open-set recognition model includes several open-set recognition sub-models, and several open-set recognition sub-models are respectively used to identify the scene category of the remote sensing images of different scene categories.

[0111] Based on any of the above embodiments, the open set recognition module 230 inputs the feature mapping results of the remote sensing images into the open set recognition sub-model of the corresponding scene category to obtain the open set scene classification results of each remote sensing image, including:

[0112] The open set recognition sub-model is used to identify whether the remote sensing image belongs to the scene category corresponding to the open set recognition sub-model. If yes, the open set scene classification result is the scene category corresponding to the open set recognition sub-model; if not, the open set scene classification result is that the scene category of the remote sensing image is unknown.

[0113] Based on any of the above embodiments, the method further includes a data processing module for filtering samples of each scene category in the original sample set based on a convex hull point selection method before merging the remote sensing image with an unknown scene category with the original sample set, thereby obtaining sample data to be retained in each scene category, and merging the sample data to be retained with the remote sensing image with an unknown scene category to obtain the current sample set; wherein the method for filtering samples of each scene category in the original sample set based on a convex hull point selection method includes:

[0114] Convex hull construction step: constructing a d-dimensional convex hull in the sample data of the same scene category based on the feature mapping results of each sample data in the original sample set;

[0115] Convex hull decomposition step: performing convex hull decomposition on the d-dimensional convex hull to obtain a plurality of sub-convex hulls;

[0116] Sub-convex hull updating step: updating the sub-convex hull based on the number of sample data in the feature space corresponding to the sub-convex hull; repeating the convex hull decomposition step and the sub-convex hull updating step based on the updated sub-convex hull until a preset condition is met;

[0117] Data selection step: based on the importance of the sample data, selecting the sample data to be retained in the scene category from the sample data contained in each of the sub-convex hulls.

[0118] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call logic instructions in the memory 330 to execute an open-set incremental remote sensing image scene classification method, which includes: obtaining a remote sensing image dataset to be subjected to scene classification;

[0119] Inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset;

[0120] Inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0121] In which, the feature extraction model and the open set recognition model are both trained based on the original sample set; when the open set scene classification result is that the scene category of the remote sensing image is unknown, the remote sensing image with unknown scene category is also merged with the original sample set to obtain the current sample set, and the feature extraction model and the open set recognition model are updated respectively based on the current sample set.

[0122] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0123] On the other hand, the present invention further provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is capable of performing the open set incremental remote sensing image scene classification method provided by the above methods, the method including: obtaining a remote sensing image dataset to be subjected to scene classification;

[0124] Inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset;

[0125] Inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0126] In which, the feature extraction model and the open set recognition model are both trained based on the original sample set; when the open set scene classification result is that the scene category of the remote sensing image is unknown, the remote sensing image with unknown scene category is also merged with the original sample set to obtain the current sample set, and the feature extraction model and the open set recognition model are updated respectively based on the current sample set.

[0127] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for performing the open set incremental remote sensing image scene classification method provided by the above methods is implemented, the method comprising: obtaining a remote sensing image dataset to be subjected to scene classification;

[0128] Inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset;

[0129] Inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; the open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown;

[0130] In which, the feature extraction model and the open set recognition model are both trained based on the original sample set; when the open set scene classification result is that the scene category of the remote sensing image is unknown, the remote sensing image with unknown scene category is also merged with the original sample set to obtain the current sample set, and the feature extraction model and the open set recognition model are updated respectively based on the current sample set.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0132] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An open set incremental remote sensing image scene classification method, characterized by: include: Obtain a remote sensing image dataset for scene classification; Inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset; Inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; The open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown; wherein the feature extraction model and the open set recognition model are both trained based on the original sample set; when the open set scene classification result is that the scene category of the remote sensing image is unknown, the remote sensing image with the unknown scene category is further merged with the original sample set to obtain a current sample set, and the feature extraction model and the open set recognition model are respectively updated based on the current sample set; Before merging the remote sensing image with unknown scene category with the original sample set, the method further includes: The method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set; wherein the method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set; wherein, the method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set. Convex hull construction step: constructing a d-dimensional convex hull in the sample data of the same scene category based on the feature mapping results of each sample data in the original sample set; Convex hull decomposition step: performing convex hull decomposition on the d-dimensional convex hull to obtain a plurality of sub-convex hulls; Sub-convex hull updating step: updating the sub-convex hull based on the number of sample data in the feature space corresponding to the sub-convex hull; repeating the convex hull decomposition step and the sub-convex hull updating step based on the updated sub-convex hull until a preset condition is met; Data selection step: based on the importance of the sample data, selecting the sample data to be retained in the scene category from the sample data contained in each of the sub-convex hulls.

2. The open set incremental remote sensing image scene classification method according to claim 1, characterized in that: The training method of the feature extraction model includes: Construct a loss function for the feature extraction model, and based on the loss function, use a prototype learning method to train the feature extraction model; the loss function includes a cross entropy loss term and a regularization term, the cross entropy loss term is used to constrain the distance between each sample data in the original sample set; the regularization term is used to constrain the distribution space of each scene category in the original sample set; wherein, the loss function As shown in formula 1: Where, is the cross entropy loss term, is the regularization term, λ R is a hyperparameter; among them, The expressions of are shown in Equation 2 and Equation 3 respectively: Where, ψ() is the feature mapping function of the feature extraction model; x is the sample data in the original sample set; c k 、c i are the category centers of scene category k and scene category i respectively; d() is the Euclidean distance; γ is a hyperparameter; K is the number of scene categories in the original sample set; p() is the probability function; y is the output of the feature extraction model; n k is the number of samples corresponding to scene category k; R k is the radius parameter; x j is the j-th sample data corresponding to scene category k.

3. The open set incremental remote sensing image scene classification method according to claim 1, characterized in that: Before inputting the feature mapping result of the remote sensing image into the open set recognition model, the method further includes: The feature mapping results of the remote sensing images are input into a closed set classifier to obtain a closed set scene classification result of each remote sensing image; wherein the closed set classifier is trained based on the original sample set.

4. The open set incremental remote sensing image scene classification method according to claim 3, characterized in that: Inputting the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image includes: According to the closed-set scene classification result, a priori scene category prediction result is obtained, and according to the priori scene category prediction result, the feature mapping result of the remote sensing image is input into the open-set recognition sub-model of the corresponding scene category to obtain the open-set scene classification result of each remote sensing image; wherein, the open-set recognition model includes several open-set recognition sub-models, and the several open-set recognition sub-models are respectively used to identify the scene category of the remote sensing images of different scene categories.

5. The open set incremental remote sensing image scene classification method according to claim 4, characterized in that: Inputting the feature mapping results of the remote sensing images into the open set recognition sub-model of the corresponding scene category to obtain the open set scene classification results of each remote sensing image includes: The open set recognition sub-model is used to identify whether the remote sensing image belongs to the scene category corresponding to the open set recognition sub-model. If yes, the open set scene classification result is the scene category corresponding to the open set recognition sub-model; if not, the open set scene classification result is that the scene category of the remote sensing image is unknown.

6. An open-set incremental remote sensing image scene classification device, characterized in that: include: A data acquisition module is used to obtain a remote sensing image dataset for scene classification; a feature extraction module, inputting the remote sensing image dataset into a feature extraction model to obtain a feature mapping result of each remote sensing image in the remote sensing image dataset; an open set recognition module, configured to input the feature mapping results of the remote sensing images into an open set recognition model to obtain an open set scene classification result of each remote sensing image; The open set scene classification result is used to indicate the scene category of the remote sensing image or that the scene category of the remote sensing image is unknown; A model training module, which trains the feature extraction model and the open set recognition model based on the original sample set; further configured to, when the open set scene classification result indicates that the scene category of the remote sensing image is unknown, merge the remote sensing image with the unknown scene category with the original sample set to obtain a current sample set, and update the feature extraction model and the open set recognition model based on the current sample set; Before merging the remote sensing image with unknown scene category with the original sample set, the model training module is further used to: The method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set; wherein the method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set; wherein, the method based on the convex hull point selection performs sample screening on each scene category in the original sample set to obtain sample data to be retained in each scene category, and the sample data to be retained is used to merge with the remote sensing image of unknown scene category to obtain the current sample set. Convex hull construction step: constructing a d-dimensional convex hull in the sample data of the same scene category based on the feature mapping results of each sample data in the original sample set; Convex hull decomposition step: performing convex hull decomposition on the d-dimensional convex hull to obtain a plurality of sub-convex hulls; Sub-convex hull updating step: updating the sub-convex hull based on the number of sample data in the feature space corresponding to the sub-convex hull; repeating the convex hull decomposition step and the sub-convex hull updating step based on the updated sub-convex hull until a preset condition is met; Data selection step: based on the importance of the sample data, selecting the sample data to be retained in the scene category from the sample data contained in each of the sub-convex hulls.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the open set incremental remote sensing image scene classification method as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the open set incremental remote sensing image scene classification method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the open set incremental remote sensing image scene classification method according to any one of claims 1 to 5 are implemented.

Images