An open-set image classification method based on cluster analysis

Abstract

The application discloses an open set image classification method based on clustering analysis, comprising the following steps: inputting a to-be-classified remote sensing image into an embedding network to obtain embedding features, determining corresponding clusters of the embedding features based on a clustering module, and determining a coding and decoding network corresponding to the clusters according to the clusters; or determining corresponding clusters based on the clustering module, determining an embedding network corresponding to the clusters and a coding and decoding network corresponding to the clusters according to the clusters, inputting the to-be-classified remote sensing image into the embedding network to obtain embedding features; inputting the embedding features into the coding and decoding network corresponding to the clusters to obtain a classification result; the classification result comprises a classification category and an unknown category. Since the coding and decoding network is divided into multiple parts according to the clusters, the embedding features of each cluster are relatively similar, and the embedding features of the to-be-classified remote sensing image of the unknown category are quite different from the embedding features, so that the to-be-classified remote sensing image of the unknown category can be classified as the unknown category by the coding and decoding network corresponding to the cluster, and the distinguishing ability of a representative-discriminative open set recognition framework based on clustering analysis to the unknown category is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, and particularly relates to an open set image classification method based on clustering analysis. BACKGROUND

[0002] Remote sensing images are an important data source for earth observation, which can help us measure and observe the detailed structure of the earth's surface. Due to the advancement of earth observation technology, the number of aerial or satellite images is growing rapidly. This makes it particularly urgent to make full use of the growing remote sensing images for intelligent earth observation. Therefore, it is extremely important to understand huge and complex satellite images. Remote sensing image scene classification is to correctly label the given scene image with predefined semantic categories.

[0003] With the latest advances in computer vision, especially the emergence of convolutional neural networks (CNN), the performance of image classification, detection and segmentation tasks has been greatly improved, making it applicable to many different fields. Land cover classification is an important direction in remote sensing image processing. According to the characteristics of remote sensing images, computers can automatically classify remote sensing images according to certain meanings, making it more convenient for people to filter and analyze applications when faced with massive remote sensing data. Therefore, researchers in the field of remote sensing science are constantly exploring in the field of image classification algorithm, improvement and innovation, one of the application fields is satellite image analysis, including resource management, urban development planning and climate control.

[0004] As shown in Figure 1 The representative-discriminative open set recognition (RDOSR) framework is a representative network architecture for open set recognition. The network architecture mainly consists of two parts: one is a closed set embedding component, which is used to project data from the original image domain to the embedding domain, so that different known classes are more easily distinguished; the other is a representative description learning component, which is used to obtain a more fine-grained representation scheme in the abundance space, so that the unknown class is better distinguished from the known class.

[0005] The closed set embedding component greatly improves the recognition ability of the network, so that even if the spectrum of the unknown class is similar to that of the known class, the unknown class can be well recognized. The weight of the closed set embedding component is trained with known class data. The representative description learning component improves the representativeness and discriminative ability of the extracted feature vector, so that the network is more informative and effective in recognizing unknown classes. The component consists of an encoder-decoder architecture, the representative feature S is extracted by a sparse Dirichlet encoder E, and the decoder is formed by the basis trained by the known class. In addition, the component also includes a classifier C applied to S to further improve the discriminative ability. In this way, when the unknown class is input into the network, a higher reconstruction error will be generated, which can be used to detect the unknown class.

[0006] In representative-discriminative open set recognition (RDOSR), in order to distinguish unknown classes and known classes, an encoder-decoder architecture is used to reconstruct the embedding space feature z, and the size of the reconstruction loss is used to detect unknown classes. That is, the encoder-decoder architecture implicitly models the known class space.

[0007] The encoder-decoder architecture does not require a specific relationship between the known classes it models, and there can be a large difference. The method uses an encoder-decoder architecture to establish a space representing all known classes. Due to the existence of differences between known classes, the reconstruction loss of known classes may be large when reconstructed by an encoder-decoder. It may even be similar to or larger than the reconstruction loss of unknown classes, making it difficult to identify unknown classes by the size of the reconstruction error.

[0008] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0009] The technical problem to be solved by the present application is to provide a cluster analysis-based open set image classification method to solve the problem of poor identification performance of unknown classes in representative-discriminative open set recognition in the prior art.

[0010] The technical solution adopted by the present application to solve the technical problem is as follows:

[0011] A cluster analysis-based open set image classification method, which is applied to a cluster analysis-based representative-discriminative open set recognition framework, the cluster analysis-based representative-discriminative open set recognition framework comprising: embedding networks, a clustering module, and a plurality of encoding-decoding networks arranged in sequence, or the clustering module, a plurality of embedding networks, and a plurality of encoding-decoding networks arranged in sequence.

[0012] The classification method comprises the following steps:

[0013] Obtaining a remote sensing image to be classified;

[0014] Inputting the remote sensing image to be classified into the embedding network to obtain the embedding feature of the remote sensing image to be classified, and determining the cluster corresponding to the embedding feature based on the clustering module, and determining the encoding-decoding network corresponding to the cluster according to the cluster; or, determining the cluster corresponding to the remote sensing image to be classified based on the clustering module, determining the embedding network corresponding to the cluster and the encoding-decoding network corresponding to the cluster according to the cluster, and inputting the remote sensing image to be classified into the embedding network to obtain the embedding feature of the remote sensing image to be classified;

[0015] input the embedding feature into a coding-decoding network corresponding to the cluster to obtain a classification result of the remote sensing image to be classified, wherein the classification result comprises a classification category and an unknown category.

[0016] The open set image classification method based on cluster analysis, wherein the cluster corresponding to the embedding feature has one or more, or the cluster corresponding to the remote sensing image to be classified has one or more; the inputting of the embedding feature into the coding-decoding network corresponding to the remote sensing image to be classified to obtain the classification result of the remote sensing image to be classified comprises:

[0017] input the embedding feature into the coding-decoding network corresponding to each cluster respectively to obtain the classification category and the reconstruction feature corresponding to each cluster respectively;

[0018] determine a reconstruction loss according to the embedding feature and the reconstruction feature;

[0019] when the reconstruction loss corresponding to each cluster respectively is greater than or equal to a preset threshold, the classification result of the remote sensing image to be classified is determined as the unknown category.

[0020] The open set image classification method based on cluster analysis, wherein the inputting of the embedding feature into the coding-decoding network corresponding to the remote sensing image to be classified to obtain the classification result of the remote sensing image to be classified further comprises:

[0021] when the minimum value of the reconstruction loss in the reconstruction loss corresponding to each cluster respectively is less than the preset threshold, the classification category corresponding to the minimum value of the reconstruction loss is taken as the classification result of the remote sensing image to be classified.

[0022] The open set image classification method based on cluster analysis, wherein the coding-decoding network comprises an encoder, a decoder and a first classifier.

[0023] The inputting of the embedding feature into the coding-decoding network corresponding to each cluster respectively to obtain the classification category and the reconstruction feature corresponding to each cluster respectively comprises:

[0024] input the embedding feature into the encoder corresponding to the cluster to obtain an encoding feature corresponding to the cluster;

[0025] input the encoding feature into the decoder corresponding to the cluster to obtain a reconstruction feature corresponding to the cluster;

[0026] input the encoding feature into the first classifier corresponding to the cluster to obtain a classification category corresponding to the cluster.

[0027] The open set image classification method based on cluster analysis, wherein the representative-discriminative open set identification framework based on cluster analysis is obtained by training using the following steps:

[0028] obtaining a training image set in an original spatial domain; wherein the training image set comprises training remote sensing images and corresponding labeled categories;

[0029] inputting the training remote sensing images in the original spatial domain into an embedding network to obtain embedding features of the training remote sensing images, the embedding features being in an embedding space;

[0030] clustering the embedding features in the embedding space to obtain a clustering module, and forming a clustering space corresponding to each cluster according to the embedding features of the cluster;

[0031] inputting the embedding features in the clustering space of each cluster into a coding-decoding network corresponding to the cluster to obtain a classification category and a reconstruction feature corresponding to the training remote sensing image;

[0032] determining a loss function value according to the labeled category, the classification category, the embedding feature, and the reconstruction feature;

[0033] updating network parameters of the embedding network and the coding-decoding network based on the loss function value;

[0034] After the step of inputting the training remote sensing images in the original spatial domain into the embedding network to obtain the embedding features of the training remote sensing images is continuously performed, the step of inputting the embedding features in the clustering space of each cluster into the coding-decoding network corresponding to the cluster to obtain the classification category and the reconstruction feature corresponding to the training remote sensing image is performed until a preset training condition is met, and a representative-discriminative open set recognition framework based on clustering analysis is obtained;

[0035] Alternatively, the representative-discriminative open set recognition framework based on clustering analysis is obtained by training using the following steps:

[0036] obtaining a training image set in an original spatial domain; wherein the training image set comprises training remote sensing images and corresponding labeled categories;

[0037] clustering the training remote sensing images in the original spatial domain to obtain a clustering module, and forming a clustering space corresponding to each cluster according to the training remote sensing images of the cluster;

[0038] inputting the training remote sensing images in the clustering space of each cluster into an embedding network to obtain embedding features of the training remote sensing images, the embedding features being in an embedding space;

[0039] inputting the embedding features in the clustering space of each cluster into a coding-decoding network corresponding to the cluster to obtain a classification category and a reconstruction feature corresponding to the training remote sensing image;

[0040] determining a loss function value according to the labeled category, the classification category, the embedding feature, and the reconstruction feature;

[0041] updating network parameters of the embedding network and the encoder-decoder network based on the loss function value;

[0042] continuing to perform the step of inputting the training remote sensing image into the embedding network in the clustering space of each cluster to obtain the embedding feature of the training remote sensing image until a preset training condition is met, to obtain a representative-discriminative open set recognition framework based on clustering analysis.

[0043] The open set image classification method based on clustering analysis, wherein when the embedding features in the embedding space are clustered, an embedding feature center of each annotation category is determined according to all embedding features of the annotation category, and when the distance between the embedding feature center and a cluster center is less than a preset distance, all embedding features of the annotation category are classified into a cluster to which the cluster center belongs; or for each embedding feature, when the distance between the embedding feature and a cluster center is less than a preset distance, the embedding feature is classified into a cluster to which the cluster center belongs.

[0044] The open set image classification method based on clustering analysis, wherein when the training remote sensing images in the original space domain are clustered, a training remote sensing image center of each annotation category is determined according to all training remote sensing images of the annotation category, and when the distance between the training remote sensing image center and a cluster center is less than a preset distance, all training remote sensing images of the annotation category are classified into a cluster to which the cluster center belongs; or for each training remote sensing image, when the distance between the training remote sensing image and a cluster center is less than a preset distance, the training remote sensing image is classified into a cluster to which the cluster center belongs.

[0045] The open set image classification method based on clustering analysis, wherein the loss function value comprises a first loss function value and a second loss function value.

[0046] The first loss function value is:

[0047] L1=λ f L f +λ z L z

[0048] wherein L1 represents the first loss function value, L f represents a cross-entropy loss, L z represents a sparsity loss, λ f and λ z represent parameters for balancing the cross-entropy loss and the sparsity loss;

[0049] The second loss function value is:

[0050] L2=λf L r +λ s H s +λ c L c

[0051] wherein, L2 represents a second loss function value, L r represents a reconstruction loss, H s represents a sparsity loss, L c represents a classification loss, λ f , λ s and λ c represent parameters used to measure the reconstruction loss, the sparsity loss and the classification loss;

[0052] The network parameters of the embedding network and the coding-decoding network are updated based on the loss function values, comprising:

[0053] The network parameters of the embedding network are updated based on the first loss function value;

[0054] The network parameters of the coding-decoding network are updated based on the second loss function value.

[0055] A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of the above when executing the computer program.

[0056] A computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of any one of the above.

[0057] Beneficial effects: since the coding-decoding network is divided into multiple clusters, the embedding features of each cluster are relatively similar, and the embedding features of the unknown category of remote sensing images to be classified are quite different, so the unknown category of remote sensing images to be classified can be classified as the unknown category by the coding-decoding network corresponding to the cluster, thereby improving the distinguishing ability of the representative-discriminative open set recognition framework based on clustering analysis for unknown categories. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is the functional principle diagram of the representative-discriminative open set recognition framework in the prior art.

[0059] Figure 2 is the functional principle diagram of the first RDOSR framework in the present application.

[0060] Figure 3 is the specific structure diagram of the first RDOSR framework in the present application.

[0061] Figure 4 is the functional principle diagram of the second RDOSR framework in the present application.

[0062] Figure 5 is the specific structure diagram of the second RDOSR framework in the present application. DETAILED DESCRIPTION

[0063] In order to make the objects, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0064] Please refer to Figures 1-5 Some embodiments of the present application provide an open set image classification method based on cluster analysis.

[0065] It is considered that the main reason for the large reconstruction error of known class samples at the output of the encoder-decoder architecture is that there is a large difference in the known classes in the training data. When the difference in the known classes in the training data is small, the reconstruction error of the known class samples at the output of the encoder-decoder architecture is small, while the reconstruction error of the unknown classes at the output of the encoder-decoder architecture is large, so that the unknown classes can be reliably detected. Therefore, it is proposed to process the training data of the known classes to make them as similar as possible to train the encoder-decoder architecture. Thus, the reconstruction error of the known class samples on this encoder-decoder architecture is smaller, and the reconstruction error of the unknown samples on this encoder-decoder architecture is larger, so that the unknown samples can be reliably identified according to the reconstruction error.

[0066] The processing idea for known class training samples is to make the known classes more similar from the perspective of (image) data or features, so that a suitable encoder-decoder architecture can be constructed to have smaller reconstruction error for known class training data. In contrast, the reconstruction error of unknown class training data is relatively larger.

[0067] The specific method is: for known class training data or features, clustering is performed to generate multiple clusters, so that the known class training data or features in each cluster are more similar in some measure. Then, for each cluster, network training is performed based on the representative-discriminative open set identification framework based on cluster analysis. For the data to be identified, first determine which cluster it belongs to according to the clustering parameters, and then apply the representative-discriminative open set identification framework based on cluster analysis corresponding to the cluster parameters for classification. If the final reconstruction error is greater than a set threshold, it is determined to be an unknown class.

[0068] Specifically, the application is based on the representative-discriminative open set recognition (RDOSR) framework, and known category training data is clustered into clusters by a clustering analysis technique, so that the model is more representative, and the reliability of recognizing unknown category samples is improved.

[0069] The RDOSR framework projects known category sample data in the original spatial domain to an embedding space domain to enhance the discriminativeness of spatial features and improve the classification ability of known category samples; further, the embedding space is mapped to an abundance space to enhance the representativeness of spatial features and improve the distinguishing ability of unknown category samples. In the framework, the application introduces a clustering analysis technique to further enhance the representativeness of spatial features and improve the ability to distinguish unknown categories of remote sensing images.

[0070] There are two specific implementation methods, which are related to the RDOSR framework. The first RDOSR framework includes an embedding network, a clustering module and a plurality of encoding-decoding networks arranged in sequence. The second RDOSR framework includes a clustering module, a plurality of embedding networks and a plurality of encoding-decoding networks arranged in sequence. The difference between the two is the position of the clustering module and the number of embedding networks. The following will be introduced in detail respectively.

[0071] The first implementation method is as shown in Figures 2-3 The clustering analysis-based open set image classification method of the application includes the following steps:

[0072] Step A100, obtaining a remote sensing image to be classified.

[0073] Specifically, the remote sensing image to be classified is usually obtained by using a remote sensing platform. The remote sensing platform includes an aerial remote sensing platform (such as an airplane, a model airplane, etc.), a space remote sensing platform (such as a satellite, a space laboratory, etc.), a space-vehicle remote sensing platform (such as an interstellar spacecraft, etc.), and a ground surface remote sensing platform (such as a vehicle-mounted platform, a ship-mounted platform, etc.). The remote sensing image to be classified contains a target, and the classification of the remote sensing image is to determine the classification category of the target in the remote sensing image.

[0074] Step A200, inputting the remote sensing image to be classified into the embedding network to obtain an embedding feature of the remote sensing image to be classified, determining a cluster corresponding to the embedding feature based on the clustering module, and determining an encoding-decoding network corresponding to the cluster according to the cluster.

[0075] Specifically, the remote sensing image to be classified is first input into the embedding network, and the embedding feature of the remote sensing image to be classified is output by the embedding network. Then the embedding feature is input into the clustering module to determine the cluster to which the embedding feature belongs. Different embedding features may belong to different clusters, and an embedding feature may belong to one cluster or multiple clusters. After determining the cluster to which the embedding feature belongs, the encoding-decoding network corresponding to the cluster can be determined. It should be noted that each cluster has a corresponding encoding-decoding network.

[0076]

[0076] Step A300, input the embedding feature into the coding-decoding network corresponding to the cluster to obtain a classification result of the remote sensing image to be classified; wherein the classification result comprises a classification category and an unknown category.

[0077] Specifically, the embedding feature is not input into all coding-decoding networks corresponding to the clusters, but only into the coding-decoding network corresponding to the cluster to which the embedding feature belongs. The embedding feature is classified by the coding-decoding network, and the classification result of the remote sensing image to be classified is output.

[0078] It should be noted that, in the present application, the remote sensing image to be classified is not directly classified, but the embedding feature of the remote sensing image to be classified is first extracted, the cluster to which the embedding feature belongs is determined, and the embedding feature is input into the coding-decoding network corresponding to the cluster to determine the classification result of the remote sensing image to be classified. When the remote sensing image to be classified of an unknown category is input into the embedding network and the cluster to which the remote sensing image to be classified belongs is determined, since the coding-decoding network is divided into multiple clusters according to the cluster, the embedding features of each cluster are relatively similar, of course, the coding-decoding network is trained according to the embedding features of a certain cluster, the embedding features of the cluster are relatively similar, and the embedding features of the remote sensing image to be classified of an unknown category are relatively different. Therefore, the remote sensing image to be classified of an unknown category can be classified as an unknown category by the coding-decoding network corresponding to the cluster, thereby improving the distinguishing ability of the representative-discriminative open set recognition framework based on clustering analysis for unknown categories.

[0079] The cluster corresponding to the embedding feature has one or more clusters. When the embedding feature is input into the clustering module, the embedding feature can be attributed to a certain cluster or multiple clusters. For example, the distance between the embedding feature and the cluster center of each cluster is calculated, and the embedding feature can be attributed to the cluster corresponding to the minimum distance. If the distance between the embedding feature and the cluster center of multiple clusters is less than a first preset distance, the embedding feature can be attributed to these clusters.

[0080] As shown in FIG. 1, the clustering module is clustered into M clusters, and M coding-decoding networks correspond to the M clusters. The clustering module can attribute the embedding feature to a certain cluster or multiple clusters. Figure 3

[0081] Step A300 comprises:

[0082] A310, input the embedding feature into the coding-decoding network corresponding to each cluster respectively to obtain the classification category and the reconstruction feature corresponding to each cluster respectively.

[0083] A320, determine the reconstruction loss according to the embedding feature and the reconstruction feature.

[0084] A330, when the reconstruction loss corresponding to each cluster respectively is greater than or equal to a preset threshold, the classification result of the remote sensing image to be classified is determined as an unknown category.​

[0085] A340, when the minimum reconstruction loss in the respective reconstruction loss of each cluster is less than the preset threshold, the classification category corresponding to the minimum reconstruction loss is taken as the classification result of the remote sensing image to be classified.

[0086] Specifically, since there can be multiple clusters to which the embedding feature belongs, the embedding feature needs to be input into each cluster corresponding coding-decoding network to obtain the classification category and the reconstructed feature output by each coding-decoding network. According to the embedding feature and the reconstructed feature, the reconstruction loss is calculated. If the reconstruction loss of all coding-decoding networks is large (the reconstruction loss is greater than or equal to the preset threshold), it indicates that all coding-decoding networks cannot be well reconstructed, and the classification category of the remote sensing image to be classified is quite different from the remote sensing image used to train the coding-decoding network of the cluster. Therefore, the classification result of the remote sensing image to be classified can be determined as the unknown category.

[0087] Since the reconstruction loss of each coding-decoding network is large or small, if the minimum reconstruction loss is small (the reconstruction loss is less than the preset threshold), it indicates that the coding-decoding network can be well reconstructed, and the classification category of the remote sensing image to be classified is quite small. The difference between the remote sensing image used to train the coding-decoding network of the cluster, so the classification result of the remote sensing image to be classified can be determined as the classification category corresponding to the minimum reconstruction loss.

[0088] The coding-decoding network includes an encoder, a decoder, and a first classifier; and each cluster corresponding coding-decoding network includes an encoder, a decoder, and a first classifier. For example, as shown in Figure 3 There are M coding-decoding networks, the encoder E specifically has an encoder E1, an encoder E2, …, and an encoder EM; the decoder D specifically has a decoder D1, a decoder D2, …, and a decoder DM; and the classifier C (i.e., the first classifier) specifically has a classifier C1, a classifier C2, …, and a classifier CM.

[0089] Step A310 specifically includes:

[0090] A311, input the embedding feature into the encoder corresponding to the cluster to obtain the encoding feature corresponding to the cluster.

[0091] A312, input the encoding feature into the decoder corresponding to the cluster to obtain the reconstructed feature corresponding to the cluster.

[0092] A313, input the encoding feature into the first classifier corresponding to the cluster to obtain the classification category corresponding to the cluster.

[0093] Specifically, the embedding feature is down-sampled and the like by the encoder to obtain an encoded feature, and then the encoded feature is input into the decoder to be up-sampled and the like to obtain a reconstructed feature, and the reconstruction loss can be calculated through the reconstructed feature and the embedding feature. The encoded feature is input into the first classifier to obtain the classification category. For example, the embedding feature Z1Fi is input into the encoder E1 to obtain an encoded feature, and then the encoded feature is input into the decoder D1 to obtain the reconstructed feature Z1F, and the reconstruction loss L1r (i.e., the reconstruction loss) can be calculated through the embedding feature Z1Fi and the reconstructed feature Z1F. Then the encoded feature is input into the classifier C1 to obtain the classification category Y.

[0094] The second implementation manner is shown in the following table. Figures 4-5 The cluster analysis-based open set image classification method provided by the application comprises the following steps:

[0095] Step B100, obtaining a remote sensing image to be classified.

[0096] Step B200, determining a cluster corresponding to the remote sensing image to be classified based on the clustering module, determining an embedding network corresponding to the cluster and a coding-decoding network corresponding to the cluster according to the cluster, and inputting the remote sensing image to be classified into the embedding network to obtain an embedding feature of the remote sensing image to be classified.

[0097] Step B300, inputting the embedding feature into the coding-decoding network corresponding to the cluster to obtain a classification result of the remote sensing image to be classified; wherein the classification result comprises a classification category and an unknown category.

[0098] Specifically, in the first implementation manner, the embedding feature of the remote sensing image to be classified is obtained first, and then the cluster to which the embedding feature belongs is determined. Different from the first implementation manner, in the second implementation manner, the cluster to which the remote sensing image to be classified belongs is determined first, and then the remote sensing image to be classified is input into the embedding network corresponding to the cluster to obtain the embedding feature. That is to say, in the first implementation manner, the embedding feature is clustered, and in the second implementation manner, the remote sensing image is clustered.

[0099] Since the remote sensing image to be classified is clustered in the second implementation manner, different embedding networks are used for different clusters to obtain the embedding feature, which further reduces the difference between the same cluster and increases the difference between different clusters, so as to improve the distinguishing ability of the unknown category. That is to say, for the remote sensing image to be classified of different clusters, the embedding network corresponding to the cluster is used to determine the embedding feature of the remote sensing image to be classified, and the coding-decoding network corresponding to the cluster is used to determine the classification result of the remote sensing image to be classified.

[0100] The cluster corresponding to the to-be-classified remote sensing image has one or more, and the to-be-classified remote sensing image is attributed to a cluster or multiple clusters. For example, the distance between the to-be-classified remote sensing image and the cluster center of each cluster is calculated, and the to-be-classified remote sensing image is attributed to the cluster corresponding to the minimum distance. If the distance between the to-be-classified remote sensing image and the cluster center of multiple clusters is less than a second preset distance, the to-be-classified remote sensing image is attributed to the clusters.

[0101] For example, as shown in FIG. 6, the clustering module clusters into M clusters, and then there are M embedding networks corresponding to the M clusters. The clustering module can attribute the to-be-classified remote sensing image to a cluster or multiple clusters. Figure 5 If the embedding network uses a second classifier (i.e., classifier F), there are classifier F1, classifier F2,..., and classifier FM. The to-be-classified remote sensing image is not input into all embedding networks (i.e., classifier F), but is attributed to a cluster according to the clustering module, and then is input into the second classifier corresponding to the cluster, to obtain the embedding feature and then input into the corresponding coding and decoding network.

[0102] Step B300 includes:

[0103] B310, input the embedding feature into the coding and decoding network corresponding to each cluster respectively, to obtain the classification category and the reconstruction feature corresponding to each cluster respectively.

[0104] B320, determine the reconstruction loss according to the embedding feature and the reconstruction feature.

[0105] B330, when the reconstruction loss corresponding to each cluster is greater than or equal to a preset threshold, the classification result of the to-be-classified remote sensing image is determined as an unknown category.

[0106] B340, when the minimum value of the reconstruction loss in the reconstruction loss corresponding to each cluster is less than the preset threshold, the classification category corresponding to the minimum value of the reconstruction loss is taken as the classification result of the to-be-classified remote sensing image.

[0107] The coding and decoding network includes an encoder, a decoder, and a first classifier. Step B310 specifically includes:

[0108] B311, input the embedding feature into the encoder corresponding to the cluster, to obtain the encoding feature corresponding to the cluster.

[0109] B312, input the encoding feature into the decoder corresponding to the cluster, to obtain the reconstruction feature corresponding to the cluster.

[0110] B313, input the encoding feature into the first classifier corresponding to the cluster, to obtain the classification category corresponding to the cluster.

[0111] Specifically, B310-B340 are consistent with A310-A340, and B311-B313 are consistent with A311-A313. It can be understood that although the processing steps are the same, the classification results may be different under the two implementation manners due to the difference between B200 and A200.

[0112] Since there are two kinds of representative-discriminative open set recognition frameworks based on clustering analysis, the training processes of the representative-discriminative open set recognition frameworks based on clustering analysis are also not completely the same.

[0113] When the representative-discriminative open set recognition framework based on clustering analysis includes an embedding network, a clustering module and a plurality of coding-decoding networks arranged in sequence, the representative-discriminative open set recognition framework based on clustering analysis is trained by using the following steps:

[0114] In step S100, a training image set in an original spatial domain is obtained; wherein the training image set includes training remote sensing images and corresponding labeled categories.

[0115] In step S200, the training remote sensing images in the original spatial domain are input into the embedding network to obtain embedding features of the training remote sensing images, and an embedding space is formed according to the embedding features.

[0116] In step S300, the embedding features in the embedding space are clustered to obtain a clustering module, and a clustering space corresponding to each cluster is formed according to the embedding features of each cluster.

[0117] In step S400, the embedding features in the clustering space of each cluster are input into the coding-decoding network corresponding to the cluster to obtain a classification category and a reconstruction feature corresponding to the training remote sensing image.

[0118] In step S500, a loss function value is determined according to the labeled category, the classification category, the embedding feature and the reconstruction feature.

[0119] In step S600, the network parameters of the embedding network and the coding-decoding network are updated based on the loss function value.

[0120] In step S700, after the step of inputting the training remote sensing images in the original spatial domain into the embedding network to obtain the embedding features of the training remote sensing images is continuously performed, the step of inputting the embedding features in the clustering space of each cluster into the coding-decoding network corresponding to the cluster to obtain the classification category and the reconstruction feature corresponding to the training remote sensing image is performed until a preset training condition is met, and a representative-discriminative open set recognition framework based on clustering analysis is obtained.

[0121] Specifically, there is a training image set X in an original spatial domain, the labeled categories of training remote sensing images in the training image set X can be divided into I, the training image set X is divided into I training remote sensing image subsets according to the labeled categories, X={X1, X2, …, Xi, …, XI} i ,…,X I}, the training remote sensing image subset Xi under the i-th labeled category, i there are I i training remote sensing images, all training remote sensing images in the training image set X, after being mapped through the embedding space, obtain embedding features, and form an embedding space Z, the embedding space ZF is divided into I embedding feature subsets according to the labeled categories, Z={Z1, Z2, …, Zi, …, ZI} i ,…,Z I}, the embedding feature subset Zi under the i-th labeled category, i there are I i embedding features, all embedding features are clustered, specifically, the embedding features in each cluster form a cluster space, then M cluster spaces can be formed, P={P1, P2, …, Pi, …, PM} m ,…,P M}, then the embedding features in the M cluster spaces are respectively input into the corresponding coding-decoding network for iterative training, to obtain the trained M coding-decoding networks and the trained embedding network, in the clustering process, a trained clustering module can be formed, thereby forming a representative-discriminative open set identification framework based on clustering analysis. It can be understood that any unsupervised machine learning algorithm can be used in the clustering process, such as K-means clustering.

[0122] When clustering the embedding features in the embedding space, according to all embedding features of each labeled category, an embedding feature center of the labeled category is determined, when the distance between the embedding feature center and a cluster center is less than a preset distance, all embedding features of the labeled category are classified into a cluster to which the cluster center belongs; or, for each embedding feature, when the distance between the embedding feature and a cluster center is less than a preset distance, the embedding feature is classified into a cluster to which the cluster center belongs.

[0123] Specifically, when clustering the embedding features, there are two clustering methods, one of which considers clustering according to the labeled categories, and the other of which does not consider clustering according to the labeled categories.

[0124] The first clustering manner: considering the annotation category, the embedding features under the same annotation category are attributed to the same cluster, that is, the embedding features under the same annotation category are not attributed to different clusters respectively, so the embedding feature center under each annotation category can be determined first, and then the embedding feature centers under various annotation categories are clustered, specifically, the embedding feature centers under I annotation categories can be clustered into M clusters, M≤I, when the distance between the embedding feature center and the cluster center is less than a preset distance, all embedding features of the annotation category are classified into the cluster center to which the cluster center belongs.

[0125] The second clustering manner: without considering the annotation category, the embedding features under the same annotation category are not necessarily attributed to the same cluster, that is, the embedding features under the same annotation category can be attributed to different clusters respectively, or can be attributed to the same cluster. All embedding features are directly clustered, specifically, all embedding features can be clustered into M clusters (the number of clusters M is irrelevant to the number I of annotation categories), when the distance between the embedding feature and the cluster center is less than a preset distance, the embedding feature is classified into the cluster center to which the cluster center belongs. It should be noted that clustering is performed only once, and step S300 does not need to be repeated. After one clustering is performed, step S200 is performed, and then step S400 is entered.

[0126] When the representative-discriminative open set recognition framework based on clustering analysis is sequentially provided with a clustering module, a plurality of embedding networks and a plurality of encoding-decoding networks, the representative-discriminative open set recognition framework based on clustering analysis is trained by using the following steps:

[0127] T100, obtaining a training image set in an original spatial domain; wherein the training image set includes training remote sensing images and corresponding annotation categories.

[0128] T200, clustering the training remote sensing images in the original spatial domain to obtain a clustering module, and forming a cluster space corresponding to each cluster according to the training remote sensing images in the cluster.

[0129] T300, inputting the training remote sensing images in each cluster space into the embedding network to obtain the embedding features of the training remote sensing images, and forming an embedding space according to the embedding features.

[0130] T400, inputting the embedding features in each cluster space into the encoding-decoding network corresponding to the cluster to obtain the classification category and the reconstruction feature corresponding to the training remote sensing image.

[0131] T500, determining a loss function value according to the annotation category, the classification category, the embedding feature and the reconstruction feature.

[0132] T600. Based on the loss function value, update the network parameters of the embedding network and the encoding / decoding network.

[0133] T700. Continue executing the step of inputting the training remote sensing image in the clustering space of each cluster into the embedding network to obtain the embedding features of the training remote sensing image, until the preset training conditions are met, and obtain the representative-discriminative open set recognition framework based on cluster analysis.

[0134] Specifically, the original spatial domain contains a training image set X. The labeled categories of the training remote sensing images in the training image set X can be divided into I categories. The training image set X is further divided into I subsets of training remote sensing images according to the labeled categories, X = {X1, X2, ..., X...} i ,…,X I}, the training remote sensing image subset X under the i-th labeled category i There is I i One training remote sensing image, Clustering is performed on all training remote sensing images in the training image set X, specifically into M clusters. Each cluster of training remote sensing images forms a clustering space, resulting in M ​​clustering spaces, P = {P1, P2, ..., P...}. m ,…,P M Then, the training remote sensing images within the M clustering spaces are input into the corresponding embedding networks to obtain embedding features, which are then input into the encoding / decoding networks for iterative training. This results in M ​​trained encoding / decoding networks and M trained embedding networks. During the clustering process, trained clustering modules are formed, thus creating a representative-discriminative open-set recognition framework based on clustering analysis. It is understood that any unsupervised machine learning algorithm, such as K-means clustering, can be used during the clustering process. It should be noted that clustering only needs to be performed once; step T200 does not need to be repeated during training.

[0135] Specifically, when clustering training remote sensing images, there are two clustering methods: one considers the labeled categories for clustering, and the other does not consider the labeled categories for clustering.

[0136] The first clustering method: Considering the label categories, training remote sensing images under the same label category will belong to the same cluster. That is, training remote sensing images under the same label category will not belong to different clusters. Therefore, we can first determine the center of the training remote sensing images under each label category based on all the training remote sensing images under that label category. Then, we can cluster the training remote sensing image centers under each label category. Specifically, we can cluster the training remote sensing image centers under I label categories into M clusters, where M≤I. When the distance between the training remote sensing image center and the cluster center is less than a preset distance, all training remote sensing images of that label category are classified into the cluster to which the cluster center belongs.

[0137] The second clustering manner does not consider the annotation categories, and the training remote sensing images under the same annotation category may not belong to the same cluster, that is, the training remote sensing images under the same annotation category may belong to different clusters or belong to the same cluster. All the training remote sensing images are directly clustered, and specifically, all the training remote sensing images can be clustered into M clusters (the number M of clusters is irrelevant to the size of the number I of annotation categories), and when the distance between the training remote sensing image and the cluster center is less than a preset distance, the training remote sensing image is classified into the cluster to which the cluster center belongs.

[0138] The loss function value includes a first loss function value and a second loss function value;

[0139] The first loss function value is:

[0140] L1=λ f L f +λ z L z

[0141] Wherein, L1 represents the first loss function value, L f represents a cross-entropy loss, L z represents a sparsity loss, λ f and λ z represent parameters for balancing the cross-entropy loss and the sparsity loss.

[0142] For the first RDOSR framework, there is only one embedding network, and a cross-entropy loss L f is calculated according to the training remote sensing image and the annotation label, and a sparsity loss L z is calculated according to the embedding feature. For the second RDOSR framework, there are multiple embedding networks, and each cluster corresponds to an embedding network. For each cluster, a corresponding cross-entropy loss L f is calculated according to the training remote sensing image and the annotation label of the cluster, and a sparsity loss L z is calculated according to the embedding feature of the cluster.

[0143] The second loss function value is:

[0144] L2=λ f L r +λ s H s +λ c L c

[0145] Wherein, L2 represents the second loss function value, L r represents a reconstruction loss, H s represents a sparsity loss, L c represents a classification loss, λ f, λ s and λ c denote parameters used to measure the reconstruction loss, the sparsity loss and the classification loss.

[0146] Since there are multiple coding-decoding networks, each cluster corresponds to a coding-decoding network. For each cluster, the reconstruction loss is calculated by the embedding feature and the reconstruction feature of the cluster, and the classification loss L is calculated by the labeled class and the classification class of the cluster. r The sparsity loss H is calculated according to the encoding feature. s .

[0147] The step S600 or the step T600 specifically includes:

[0148] 610, updating the network parameters of the embedding network based on the first loss function value.

[0149] 620, updating the network parameters of the coding-decoding network based on the second loss function value.

[0150] Specifically, the first loss function value and the second loss function value are calculated respectively, and the network parameters of the embedding network are iteratively updated based on the first loss function value, and the network parameters of the coding-decoding network are iteratively updated based on the second loss function value, until a preset training condition is met, to obtain a trained representative-discriminative open-set recognition framework based on clustering analysis. It should be noted that the updating of the network parameters of the embedding network and the updating of the network parameters of the coding-decoding network can be that the embedding network is first trained to update the network parameters of the embedding network, and after the preset requirement is reached, the coding-decoding network is trained to update the network parameters of the coding-decoding network.

[0151] It can be understood that in the training process, the clustering module in this framework is basically unchanged. It is determined according to the original data or the embedding feature, and the number of clusters is determined according to experience. Once determined, it will not change.

[0152] The specific structure of the embedding network, the specific structure of the coding-decoding network and the calculation of each loss can refer to the prior art (Razieh Kaviani Baghbaderani et al., Representative-Discriminative Learning for Open-set Land Cover Classification of Satellite Imagery, European Conference on Computer Vision (ECCV) 2020).

[0153] Based on the open-set image classification method based on clustering analysis according to any one of the above embodiments, the application further provides an embodiment of a computer device:

[0154] The computer device of the present application comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method according to any one of the above embodiments when executing the computer program.

[0155] The open-set image classification method based on cluster analysis according to any one of the above embodiments, the present application also provides an embodiment of a computer readable storage medium:

[0156] The computer readable storage medium of the present application stores a computer program, and the computer program implements the steps of the method according to any one of the above embodiments when executed by a processor.

[0157] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A method of open-set image classification based on cluster analysis, characterized in that, The application is applied to a representative-discriminative open set identification framework based on cluster analysis, and the representative-discriminative open set identification framework based on cluster analysis comprises an embedding network, a clustering module and a plurality of encoding-decoding networks arranged in sequence, or a clustering module, a plurality of embedding networks and a plurality of encoding-decoding networks arranged in sequence. The classification method comprises the steps of: obtaining a to-be-classified remote sensing image; inputting the to-be-classified remote sensing image into the embedding network to obtain embedding features of the to-be-classified remote sensing image, and determining a cluster corresponding to the embedding features based on the clustering module, and determining an encoding-decoding network corresponding to the cluster according to the cluster; or determining a cluster corresponding to the to-be-classified remote sensing image based on the clustering module, determining an embedding network corresponding to the cluster and an encoding-decoding network corresponding to the cluster according to the cluster, and inputting the to-be-classified remote sensing image into the embedding network to obtain embedding features of the to-be-classified remote sensing image; inputting the embedding features into the encoding-decoding network corresponding to the cluster to obtain a classification result of the to-be-classified remote sensing image; wherein the classification result comprises a classification category and an unknown category; The representative-discriminative open set identification framework based on cluster analysis is obtained by training in the following steps: obtaining a training image set in an original spatial domain; wherein the training image set comprises training remote sensing images and corresponding labeled categories; inputting the training remote sensing images in the original spatial domain into an embedding network to obtain embedding features of the training remote sensing images, and the space where the embedding features are located is an embedding space; clustering the embedding features in the embedding space to obtain a clustering module, and forming a cluster space corresponding to each cluster according to the embedding features of each cluster; inputting the embedding features in the cluster space of each cluster into an encoding-decoding network corresponding to the cluster to obtain a classification category and a reconstruction feature corresponding to the training remote sensing image; determining a loss function value according to the labeled category, the classification category, the embedding feature and the reconstruction feature; updating network parameters of the embedding network and the encoding-decoding network based on the loss function value; After the step of inputting the training remote sensing images in the original spatial domain into the embedding network to obtain the embedding features of the training remote sensing images is continuously executed, the step of inputting the embedding features in the cluster space of each cluster into the encoding-decoding network corresponding to the cluster to obtain the classification category and the reconstruction feature corresponding to the training remote sensing image is executed until a preset training condition is met, and a representative-discriminative open set identification framework based on cluster analysis is obtained; Or, the representative-discriminative open set identification framework based on cluster analysis is obtained by training in the following steps: obtaining a training image set in an original spatial domain; wherein the training image set comprises training remote sensing images and corresponding labeled categories; clustering the training remote sensing images in the original spatial domain to obtain a clustering module, and forming a cluster space corresponding to each cluster according to the training remote sensing images of each cluster; inputting the training remote sensing images in the cluster space of each cluster into an embedding network to obtain embedding features of the training remote sensing images, and the space where the embedding features are located is an embedding space; inputting the embedding feature into a coding-decoding network corresponding to the cluster to which the training remote sensing image belongs, to obtain a classification category and a reconstruction feature corresponding to the training remote sensing image; determining a loss function value according to the labeled category, the classification category, the embedding feature, and the reconstruction feature; updating network parameters of the embedding network and the coding-decoding network based on the loss function value; continuing to perform the step of inputting the training remote sensing image in the clustering space of each cluster into the embedding network to obtain the embedding feature of the training remote sensing image until a preset training condition is met, to obtain a representative-discriminative open set recognition framework based on clustering analysis.

2. The open-set image classification method based on cluster analysis according to claim 1, characterized in that, The cluster corresponding to the embedding feature has one or more, or the cluster corresponding to the remote sensing image to be classified has one or more; the step of inputting the embedding feature into the coding-decoding network corresponding to the remote sensing image to be classified to obtain a classification result of the remote sensing image to be classified comprises: inputting the embedding feature into the coding-decoding network corresponding to each cluster respectively to obtain a classification category and a reconstruction feature corresponding to each cluster respectively; determining a reconstruction loss according to the embedding feature and the reconstruction feature; when the reconstruction loss of each cluster is greater than or equal to a preset threshold, the classification result of the remote sensing image to be classified is determined as an unknown category.

3. The open-set image classification method based on cluster analysis according to claim 2, characterized in that, The step of inputting the embedding feature into the coding-decoding network corresponding to the remote sensing image to be classified to obtain a classification result of the remote sensing image to be classified further comprises: when the minimum value of the reconstruction loss of each cluster is less than the preset threshold, the classification category corresponding to the minimum value of the reconstruction loss is taken as the classification result of the remote sensing image to be classified. 4.The open-set image classification method based on cluster analysis of claim 2, wherein, The coding-decoding network comprises an encoder, a decoder, and a first classifier. The step of inputting the embedding feature into the coding-decoding network corresponding to each cluster respectively to obtain a classification category and a reconstruction feature corresponding to each cluster respectively comprises: inputting the embedding feature into the encoder corresponding to the cluster to obtain an encoding feature corresponding to the cluster; inputting the encoding feature into the decoder corresponding to the cluster to obtain a reconstruction feature corresponding to the cluster; inputting the encoding feature into the first classifier corresponding to the cluster to obtain a classification category corresponding to the cluster. 5.The open-set image classification method based on cluster analysis of claim 1, wherein, When clustering the embedding features in the embedding space, an embedding feature center of each labeled category is determined according to all embedding features of the labeled category, and when the distance between the embedding feature center and a cluster center is less than a preset distance, all embedding features of the labeled category are classified into a cluster to which the cluster center belongs; or, for each embedding feature, when the distance between the embedding feature and a cluster center is less than a preset distance, the embedding feature is classified into a cluster to which the cluster center belongs. 6.The open-set image classification method based on cluster analysis of claim 1, wherein, When clustering the training remote sensing images in the original spatial domain, a training remote sensing image center of each annotation category is determined according to all training remote sensing images of the annotation category, and when a distance between the training remote sensing image center and a cluster center is less than a preset distance, all training remote sensing images of the annotation category are classified into a cluster to which the cluster center belongs; or, for each training remote sensing image, when a distance between the training remote sensing image and a cluster center is less than a preset distance, the training remote sensing image is classified into a cluster to which the cluster center belongs. 7.The open-set image classification method based on cluster analysis of claim 1, wherein, The loss function value comprises a first loss function value and a second loss function value. The first loss function value is: wherein, denotes a first loss function value, denotes a cross-entropy loss, denotes a sparsity loss, and denotes a parameter to balance the cross-entropy loss and the sparsity loss; The second loss function value is: wherein, represents a second loss function value, represents a reconstruction loss, represents a sparsity loss, represents a classification loss, , and represent parameters used to measure the reconstruction loss, the sparsity loss, and the classification loss. The updating of the network parameters of the embedding network and the encoding-decoding network based on the loss function value comprises: updating the network parameters of the embedding network based on the first loss function value; updating the network parameters of the encoding-decoding network based on the second loss function value.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images