Method and device for image classification based on image class spatial relationship model

Abstract

The present invention discloses a method and device for image classification based on an image class spatial relationship model, belonging to the field of image recognition technology, the method comprising: step S1: obtaining an image data set, the image data set comprising a data set with known category labels and a data set with unknown category labels; step S2: inputting the image data set into the image class spatial relationship model to perform image classification; step S3: obtaining the image classification result output by the image class spatial relationship model and visualizing it. The present invention can effectively obtain image class relationship information and improve the classification performance of the image new category discovery task.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and in particular relates to a method and device for image classification based on an image class space relationship model. Background Art

[0002] In recent years, computer vision has made remarkable progress in the field of deep learning research. For image recognition and classification, researchers initially proposed the concept of open set recognition (OSR) to deal with the situation where there may be "unseen" samples in the test set that are not covered in the training set. The core of the OSR task is to identify and reject data that does not belong to the distribution of the training set, and classify all unknown categories into a unified "unknown" category. However, this method has limitations when dealing with more specific problems. For example, when faced with unknown instances of a specific category (such as bird species), it is obviously unreasonable to simply classify all unknown birds into the "unknown" category. In other words, although the OSR method may produce correct clustering results, these results often lack clear semantic information, resulting in limited significance in practical applications.

[0003] Recently, the novel class discovery task (NCD) has been proposed to solve the above problems. The novel class discovery task pays more attention to the extraction and sorting of cluster semantic information, ensuring that the final clustering results are consistent with the actual visual semantics, so as to better adapt to the complex scenes of the real world. Early researchers adopted a two-stage method inspired by transfer learning, such as KCL, MCL and DTC. The core idea of ​​these methods is to first train the embedding network on a labeled dataset and then directly apply these networks to the processing of embedded data. For example, KCL and MCL introduced a cross-domain and cross-task transfer learning framework, which uses pairwise similarity to represent classification information and learns clustering networks based on pairwise similarity predictions through different objective functions. Similarly, the DTC method initially trains a feature expansion model on a labeled dataset to capture high-level semantic category information. Afterwards, these extractors are used to initialize the clustering algorithm, and then further optimized and adjusted using unlabeled data to improve the performance of category discovery. Since then, the researchers have optimized the traditional two-stage method and improved it into a single-stage method, which allows simultaneous training of labeled and unlabeled data and obtains more general representations through contrastive learning.

[0004] The latest research shows that although the knowledge transfer of latent space in existing NCD has achieved certain success, it still has limitations and fails to fully reveal the intrinsic connection between known classes and new classes, resulting in inaccurate recognition results of new image categories. Summary of the invention

[0005] In view of the defects existing in the above-mentioned prior art, the present invention provides a method for image classification based on an image-based spatial relationship model, wherein the image-based spatial relationship model is implemented based on a prototype network, and the method comprises:

[0006] Step S1: Acquire an image dataset, wherein the image dataset includes a dataset with known category labels and a dataset with unknown category labels;

[0007] Step S2: Input the image data set into the image class spatial relationship model for image classification, wherein the image class spatial relationship model includes a supervised pre-training module and a discovery learning module; the supervised pre-training module includes an encoder Ω T , prototype network P T and the first softmax layer, the discovery learning module includes an encoder Ω S , prototype network P S and a second softmax layer; the supervised pre-training module serves as a teacher model for knowledge distillation, and the discovery learning module serves as a student model for knowledge distillation;

[0008] The loss function of the image-based spatial relationship model includes a unified cross entropy loss and a knowledge distillation loss;

[0009] Step S3: Obtain the image classification result output by the image class spatial relationship model and visualize it.

[0010] Preferably, in step S1, the image dataset includes a dataset with known category labels. and datasets with unknown class labels Where x, y represent the input image data and the category label corresponding to the input image data, respectively, N is the number of images in the data set with known category labels, and M is the number of images in the data set with unknown category labels; represents the class space corresponding to the known categories, Represents the class space corresponding to the unknown category, where C l Represents the maximum category label in the class space corresponding to the known category, C u Represents the maximum category label in the class space corresponding to the unknown category, and the complete class space corresponding to the image dataset

[0011] Preferably, in step S2, a prototype vector is set for each known category. The prototype network P T and prototype network P S Used to measure sample x i and the prototype vector; where

[0012] The supervised pre-training module is used to obtain image-like spatial relationship information from training data and transfer the image-like spatial relationship information to the discovery learning module through KL divergence;

[0013] The discovery learning module has a two-head network structure, including a known category classification head and a new category discovery header Connect to the prototype network P S To receive image-like spatial relationship information; A self-labeling method is used to cluster new categories.

[0014] Preferably, in step S2, the discovery learning module simultaneously takes the known category label image data and the unknown category label image data as input, and respectively passes the known category classification head and a new category discovery header After that, we get C l and C u Logarithmically, the C l and C u Logarithmic fusion is used to obtain the final image classification result.

[0015] Preferably, in step S2, the loss function of the image-based spatial relationship model includes a unified cross entropy loss and a knowledge distillation loss, specifically:

[0016]

[0017] in, is the loss function of the image-like spatial relationship model, To unify the cross entropy loss, is the knowledge distillation loss and α is a hyperparameter.

[0018] Preferably, the unified cross entropy loss includes a cross entropy loss based on a prototype network and a self-labeling loss, namely:

[0019]

[0020] in, is the cross entropy loss based on the prototype network, is the self-labeling loss; p(x∈μ j |x) represents the image sample x i Belong to the prototype μ j The probability of represents the prior distribution, represents the posterior distribution, B represents the image sample batch size, which consists of image samples of known categories B l and image samples of unknown categories B u composition.

[0021] Preferably, the knowledge distillation loss is specifically:

[0022]

[0023] in, represents the knowledge distillation loss of new class samples of unknown class images, represents the knowledge distillation loss of known category image samples, and η is the coefficient for adjusting the distillation loss of known and unknown categories, which is used to balance the model performance of discovering new categories and detecting known categories.

[0024] Preferably, the and Obtained by the following formula:

[0025]

[0026] Where B represents the batch size of image samples, and They respectively represent the KL divergence corresponding to the unknown category image samples and the known category image samples in the knowledge distillation process.

[0027] Preferably, η is 0.01.

[0028] The present invention also discloses a device for image classification based on an image-like spatial relationship model, comprising a memory and a processor, wherein computer program code is stored in the memory, and the processor is used to execute the computer program code to implement the aforementioned method for image classification based on an image-like spatial relationship model.

[0029] The present invention classifies images using a prototype network-based image class spatial relationship model, and performs spatial modeling based on the distance between image samples and each prototype vector, so as to more effectively obtain class relationship information, thereby improving the interpretability and robustness of the model. In addition, the present invention also performs knowledge distillation on known classes and unknown classes to achieve a balance in model classification performance. In addition, the present invention conducts an in-depth analysis of the model loss function, and uses the joint classification loss of known and unknown class images, the prototype network loss, and the knowledge distillation loss as the loss function, which significantly improves the model training effect and image classification prediction accuracy. Experimental results show that the technical solution of the present invention performs comparable to the existing most advanced methods in the CIFAR-100 dataset, and at the same time achieves excellent performance that exceeds the baseline by 3%-9% in three fine-grained datasets: Stanford Cars, CUB-2002-2011, and FGVC Aircraft. In addition, compared with linear classification, the technical solution of the present invention creates more compact clusters in the latent space, further verifying the effectiveness of this solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0031] Figure 1 is a flowchart illustrating a method for image classification based on an image class spatial relationship model according to an embodiment of the present invention.

[0032] Figure 2 FIG. 2 is a diagram illustrating an image class spatial relationship model according to an embodiment of the present invention.

[0033] Figure 3 is a diagram showing a prototype network according to an embodiment of the present invention.

[0034] Figure 4 FIG. 2 is a diagram showing a visualization comparison result according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0037] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0038] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0039] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0040] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0041] like Figure 1 As shown, a method for image classification based on an image class spatial relationship model, wherein the image class spatial relationship model is implemented based on a prototype network, and the method comprises:

[0042] Step S1: Acquire an image dataset, where the image dataset includes a dataset with known category labels and a dataset with unknown category labels.

[0043] In step S1, the image dataset includes a dataset with known category labels. and datasets with unknown class labels Where x, y represent the input image data and the category label corresponding to the input image data, respectively, N is the number of images in the data set with known category labels, and M is the number of images in the data set with unknown category labels; represents the class space corresponding to the known categories, Represents the class space corresponding to the unknown category, where C l Represents the maximum category label in the class space corresponding to the known category, C u Represents the maximum category label in the class space corresponding to the unknown category, and the complete class space corresponding to the image dataset

[0044] Step S2: Input the image data set into the image class spatial relationship model for image classification. Figure 2 As shown, the image-based spatial relationship model includes a supervised pre-training module and a discovery learning module; the supervised pre-training module includes an encoder ΩT , prototype network P T and the first softmax layer, the discovery learning module includes an encoder Ω S , prototype network P S and a second softmax layer; the supervised pre-training module serves as a teacher model for knowledge distillation, and the discovery learning module serves as a student model for knowledge distillation;

[0045] The loss function of the image-based spatial relationship model includes a unified cross entropy loss and a knowledge distillation loss;

[0046] Step S3: Obtain the image classification result output by the image class spatial relationship model and visualize it.

[0047] Among them, the t-SNE method is used to visualize the image classification results.

[0048] The embodiment of the present invention uses the image class space relationship model based on the prototype network to classify images, and performs spatial modeling based on the distance between the image sample and each prototype vector, so as to more effectively obtain class relationship information, thereby improving the interpretability and robustness of the model. In addition, the present invention also performs knowledge distillation on known classes and unknown classes to achieve a balance in model classification performance.

[0049] According to a specific embodiment of the present invention, the prototype network P T and prototype network P S like Figure 3 As shown. Among them, the prototype network P T and prototype network P S Used to measure sample x i The similarity between and prototype vector is obtained as follows:

[0050] This embodiment first selects to set only one prototype vector for each known image category. In this framework, the embodiment of the present invention uses distance to measure the similarity between the sample and the prototype. The soft assignment of each image sample (x, y) to the prototype vector μ is calculated, and the distance between them can be expressed as x i Belongs to the prototype vector μ j For each image sample instance x in the classification head i , this embodiment measures x i and the corresponding prototype vector μ j The similarity distance between them is:

[0051] Sim(Ω(x i ),μ j )=Sim e (Ω(xi ),μ j )-Sim c (Ω(x i ),μ j )

[0052]

[0053] Sim c (Ω(x i ),μ j )=Ω(x i )·μ j

[0054] Among them, Ω(·) is the encoder before the prototype network, which is used to extract image features; Sim e and Sim c They represent Euclidean distance and cosine distance respectively; the square of the Euclidean distance is divided by the dimension d of the feature space for normalization to avoid the influence of the distance on the model in high-dimensional space.

[0055] Next, calculate each image sample instance x i Belongs to the prototype vector μ j The probability is:

[0056]

[0057] Here, γ represents the regulatory factor.

[0058] Then, the cross entropy (CE) loss can be defined as the loss of the prototype network framework:

[0059]

[0060] After the optimization of the above formula, similar embedded features are attracted to the prototype center, forming a trend of mutual convergence; while heterogeneous features show the characteristics of mutual repulsion. Therefore, the distribution characteristics of the feature embedding obtained by the prototype network in the latent space are more distinct, which is specifically manifested in that the intra-class distribution is more compact and the inter-class distribution is more sparse.

[0061] In a specific embodiment of the present invention, the supervised pre-training module is used to obtain image-like spatial relationship information from training data, and transfer the image-like spatial relationship information to the discovery learning module through KL divergence.

[0062] Specifically, the supervised pre-training module pre-trains the image dataset with known labels to obtain the distribution of known classes. In order to determine the spatial relationship between known class samples, each new class image sample with unknown labels is input into the supervised pre-training module, and the relationship between each new class image sample and the known class is calculated to obtain the image class spatial relationship, and the image class spatial relationship is transferred to the discovery learning module through KL divergence.

[0063] In a specific embodiment of the present invention, the discovery learning module has a two-head network structure, including a known category classification head and a new category discovery header Connect to the prototype network P S To receive image-like spatial relationship information; The self-labeling method is used to cluster new categories. The discovery learning module takes the known category label image data and the unknown category label image data as input, respectively, through the known category classification head and a new category discovery header After that, we get C l and C u Logarithmically, the C l and C u Logarithmic fusion obtains the final image classification result, C l is the number of image categories with known labels, C u is the number of image classifications with unknown labels, L ce is the cross-entropy loss function.

[0064] According to a specific embodiment of the present invention, Figure 2 As shown in the figure, the path of the known label data flow represents the processing process of the known label image data, and the path of the unknown label data flow represents the processing process of the unknown label image data. The mixed logarithmic score represents the mixed logarithmic score obtained after passing through the two classification heads, and finally merged for joint cross entropy (CE) loss calculation. The prototype network serves as the prototype head of the supervised pre-training module and the discovery learning module. Figure 2 The upper middle part is the supervised pre-training module, which serves as the teacher model for knowledge distillation. Figure 2 The lower middle part serves as a student model. Figure 2 The images in are from the Animals and CUB datasets.

[0065] like Figure 2 The image class spatial relationship model shown in the figure, according to the above image classification prediction process, its corresponding loss function includes unified cross entropy loss and knowledge distillation loss, specifically:

[0066]

[0067] in, is the loss function of the image-like spatial relationship model, To unify the cross entropy loss, is the knowledge distillation loss and α is a hyperparameter.

[0068] According to a specific embodiment of the present invention, the unified cross entropy loss consists of two parts, namely, the loss based on the prototype network and the joint loss of clustering of known category images and unknown category images, that is, the self-labeling clustering loss. Here we specifically introduce how to obtain the self-labeling clustering loss.

[0069] In the prior art, clustering and representation learning have become conventional means of processing unlabeled data, especially in situations where labeled data is scarce. However, early methods tended to classify all data points into the same category, resulting in the subsequent classification task outputting constant labels, which led to a degenerate solution. In order to overcome this defect, the self-labeling method introduced the concept of equal distribution constraint. Its core assumption is that the label distribution should be balanced, that is, it is expected that samples are evenly distributed among the labels. This problem can be regarded as an optimal transmission (OT) problem, which can be quickly solved with the help of a variant of Sinkhorn-Knopp (SK). Specifically, an embodiment of the present invention achieves this goal by minimizing the average cross entropy loss by training the model under supervision:

[0070]

[0071] Among them, B is the batch size of image samples, which is determined by the known sample B l and unknown sample B u Composition, p(y i |x i ) is the prior distribution. In an unsupervised setting, the embodiment of the present invention improves the above formula by replacing the label with the posterior distribution q(y|x i ),get:

[0072]

[0073] Where E(·) represents the average cross entropy, p is the expected output probability distribution, and q is the actual output probability distribution.

[0074] At the same time, in order to prevent degenerate solutions, an equipartition constraint is introduced to ensure that B u The samples are evenly distributed in C u Between classes, as follows:

[0075]

[0076] Therefore, in the unsupervised case, the problem of minimizing the average cross entropy loss is expressed as:

[0077] Condition 1: q(y|x i )∈{0,1};

[0078] Condition 2:

[0079]

[0080] For any y that satisfies the above conditions 1 and 2, find the minimum value of E(p,q), that is,

[0081] In the embodiment of the present invention, L=[l 1 ,...l B ] represents the predicted logarithmic matrix calculated for the model, where As the pseudo label estimated by the SK algorithm. Relax to an element of the transport polyhedron:

[0082]

[0083] in, is a vector of all appropriate dimensions. Then, using the matrix L and The Frobenius dot product of optimization This solves the problem:

[0084]

[0085] Finally, using As the optimized pseudo-label, E(p,q) is improved to obtain the final self-labeling clustering loss:

[0086]

[0087] In a specific embodiment of the present invention, the supervised pre-training module calculates the relationship between the new class and the known class by the following formula:

[0088]

[0089] Among them, softmax represents the normalized exponential function, x u Represents a new class sample of an image of an unknown class, Ω T The prototype network P T The previous encoder is used to extract image features. Represents the prototype network P T Calculate image sample x u The probability of belonging to category l, t is the temperature coefficient, is the feature vector space of known image categories.

[0090] The discovery learning module obtains the final prediction result of the new image category according to the following formula:

[0091]

[0092] Among them, softmax represents the normalized exponential function, x u Represents a new class sample of an image of an unknown class, Ω S The prototype network P S The previous encoder is used to extract image features. Represents the prototype network P S Calculate image sample x u The probability of belonging to category l, t is the temperature coefficient, is the feature vector space of known image categories.

[0093] In a specific embodiment of the present invention, for the knowledge distillation process, the knowledge distillation loss is defined as:

[0094]

[0095] In a specific embodiment of the present invention, the goal of knowledge distillation is to ensure that the output distribution of the student model is consistent with the output distribution of the teacher model. Therefore, when a new category sample is input into the classifier head of a known category, what is actually obtained is the overfitting score of the sample in the known category. The key to distillation is to capture the distribution of these overfitting scores, thereby promoting the learning of unknown categories. Then, during the training process, the model is more inclined to learn the knowledge of new categories, which may lead to ignoring the classification performance of known categories. In this embodiment, the knowledge of known classes is extracted in the discovery training phase to solve this problem:

[0096]

[0097] On this basis, the overall form of knowledge distillation loss can be written as:

[0098]

[0099] Where η is a coefficient used to adjust the distillation loss of known and unknown classes. Based on the choice of η, the model can perfectly balance the performance of discovering new categories and detecting known classes.

[0100] In the above formulas, B represents the batch size of image samples. and They respectively represent the KL divergence corresponding to the unknown category image samples and the known category image samples in the knowledge distillation process.

[0101] In order to evaluate the effectiveness and robustness of the method in this paper, the embodiment of the present invention conducted experiments on the CIFAR100 dataset, which is a standard benchmark dataset in computer vision. The embodiment of the present invention divides CIFAR100 into two subsets:

[0102] CIFAR100-80 (where 80% of the classes are known and 20% are new) and CIFAR100-50 (with less supervised information, where the ratio of known classes to new classes is 50:50). The embodiment of the present invention excludes the CIFAR10 and ImageNet datasets from the evaluation because the performance of existing methods on these datasets has reached saturation, which makes it difficult to identify differences in algorithm performance. In addition, the method of the embodiment of the present invention is evaluated on three challenging fine-grained datasets: StanfordCars, CUB-200-2011, and FGVC-Aircraft. For each dataset, the embodiment of the present invention divides it into two parts, one containing known categories and the other containing new categories. The details of the dataset segmentation are as follows:

[0103] Table 1: Dataset segmentation information;

[0104]

[0105] According to the existing evaluation methods in NCD literature, we use the average clustering accuracy (Cluster acc ) is used to evaluate the performance of the method of this embodiment, which is defined as:

[0106]

[0107] where y i and Represent the true value and the predicted value respectively, P is the set of all permutations, and the optimal permutation can be calculated using the Hungarian algorithm.

[0108] The embodiment of the present invention adopts two evaluation protocols to comprehensively evaluate the image-based spatial relationship model, namely task-aware and task-unaware. Under the framework of task-aware evaluation, for images of known label categories, the embodiment of the present invention only focuses on the output results of the known label classifier; and for images of unknown label categories, it only focuses on the output results of the unknown label classifier. This evaluation method has been widely used in previous literature. The accuracy of task-unawareness may be more practical in actual applications because it can test whether the model can effectively distinguish between labeled and unlabeled categories.

[0109] According to the guidance of previous research methods, this study selected ResNet18 as the backbone network of the CIFAR100 dataset for feature extraction. In the initial stage, the embodiment of the present invention pre-trained the network for 100 epochs for known categories to ensure that it has basic classification capabilities. Subsequently, a joint training strategy of 500 epochs was adopted to incorporate known categories and newly added categories into the training process to improve the generalization performance of the network. During the training process, SGD was selected as the optimizer, and an adaptive learning rate strategy was implemented. Specifically, the learning rate starts from a linear growth stage and then undergoes cosine decay to adapt to the learning needs of different stages, thereby optimizing the training effect of the network.

[0110] The embodiment of the present invention makes a detailed comparison between the proposed method and the latest methods under the current two evaluation protocols, covering multiple algorithms such as Cr-KD-NCD, UNO, ComEx, NCL and RankStats. Under the framework of task-aware evaluation, the method of this paper achieves competitive performance on the standard datasets CIFAR100-50 and CIFAR100-80. Specifically, on the CIFAR100-50 dataset, the performance of the embodiment of the present invention is improved by 0.8% compared with Cr-KD-NCD; while on the CIFAR100-80 dataset, the performance is slightly reduced. However, on the three more challenging fine-grained datasets (Stanford Cars, CUB and Aircraft), the embodiment of the present invention shows significant advantages, and its performance surpasses the Cr-KD-NCD algorithm by 8.8%, 7.2% and 5.7% respectively, reflecting the superior performance of the embodiment of the present invention when processing complex datasets.

[0111] Table 2: Evaluation results of different algorithms on various datasets under the task-aware evaluation framework:

[0112] method Method source CIFAR100-50 CIFAR100-80 CUB StanfordCars Aircraft Kmeans Traditional methods 28.3±0.7 56.3±1.7 42.2±0.5 13.1±1.0 18.5±0.3 DTC ICCV2019 35.9±1.0 67.3±1.2 - - - RankStats+ TPAMI2021 44.1±3.7 75.2±4.2 55.3±0.8 36.5±0.6 38.4±0.6 NCL CVPR2021 52.7±1.2 86.6±0.4 48.1±0.9 43.5±1.2 43.0±0.5 ComEx CVPR2022 53.4±0.7 85.7±1.3 - - - UNO ICCV2021 60.4±1.4 90.4±0.2 59.2±0.4 49.8±1.4 52.1±0.7 GCD CVPR2022 - - 56.4±0.3 42.6±0.4 49.5±1.0 Cr-KD-NCD ICCV2023 65.3±0.6 91.2±0.1 65.7±0.6 53.5±0.8 55.8±0.9 This application 66.1±901.2 901.2.9±0.2 72.9±1.2 62.3±0.9 61.5±0.8

[0113] Subsequently, the evaluation metric results under the task-agnostic protocol on the same dataset are presented. In general, the embodiments of the present invention outperform previous methods in both known and new classes. Specifically, on the standard CIFAR100-50 dataset, the embodiments of the present invention improve by 4.2% on new classes and 1.2% on overall accuracy, although the accuracy of known classes is slightly improved by 1.2%. On three challenging fine-grained datasets, the embodiments of the present invention consistently outperform Cr-KD-NCD in classification performance for both known and unknown classes. The overall accuracy of Stanford Cars is improved by 3.3%, CUB by 1.1%, and Aircraft by 2.6%.

[0114] Table 3: Evaluation results of different algorithms on various datasets under the task-aware evaluation framework:

[0115]

[0116] In a specific embodiment of the present invention, the learned feature space is qualitatively analyzed using t-SNE and compared with the previous state-of-the-art methods on the Aircraft dataset, which is a fine-grained dataset containing 50 known classes and 50 unknown classes. Figure 4 It is shown that the specific embodiments of the present invention produce more compact clusters and larger inter-cluster distances than previous methods.

[0117] According to a specific embodiment of the present invention, the impact of η under two evaluation protocols (task-aware and task-agnostic) is comprehensively evaluated on two fine-grained datasets (CUB, Stanford Cars). The evaluation results show that regardless of the hyperparameter settings, the performance of task-awareness greatly exceeds the baseline. Under the task-agnostic protocol, the classification performance of known classes needs to be considered. Therefore, a trade-off must be made between new classes and known classes. Based on the above findings, the embodiment of the present invention recommends setting η to 0.01 because it achieves an excellent balance between learning known classes and discovering new classes.

[0118] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0119] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0120] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0123] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A method for image classification based on an image-based spatial relationship model, wherein the image-based spatial relationship model is implemented based on a prototype network, and the method comprises: Step S1: Acquire an image dataset, wherein the image dataset includes a dataset with known category labels and a dataset with unknown category labels; Step S2: Input the image data set into the image class spatial relationship model for image classification, wherein the image class spatial relationship model includes a supervised pre-training module and a discovery learning module; the supervised pre-training module includes an encoder Ω T , prototype network P T and the first softmax layer, the discovery learning module includes an encoder Ω S , prototype network P S and a second softmax layer; the supervised pre-training module serves as a teacher model for knowledge distillation, and the discovery learning module serves as a student model for knowledge distillation; The loss function of the image-based spatial relationship model includes a unified cross entropy loss and a knowledge distillation loss; Step S3: Obtain the image classification result output by the image class spatial relationship model and visualize it.

2. The method for image classification based on the image class spatial relationship model according to claim 1, characterized in that: In step S1, the image dataset includes a dataset with known category labels. and datasets with unknown class labels Where x, y represent the input image data and the category label corresponding to the input image data, respectively, N is the number of images in the data set with known category labels, and M is the number of images in the data set with unknown category labels; represents the class space corresponding to the known categories, Represents the class space corresponding to the unknown category, where C l Represents the maximum category label in the class space corresponding to the known category, C u Represents the maximum category label in the class space corresponding to the unknown category, and the complete class space corresponding to the image dataset 3. The method for image classification based on image class spatial relationship model according to claim 1, characterized in that: In step S2, a prototype vector is set for each known category. The prototype network P T and prototype network P S Used to measure sample x i and the prototype vector; where The supervised pre-training module is used to obtain image-like spatial relationship information from training data and transfer the image-like spatial relationship information to the discovery learning module through KL divergence; The discovery learning module has a two-head network structure, including a known category classification head and a new category discovery header Connect to the prototype network P S To receive image-like spatial relationship information; A self-labeling method is used to cluster new categories.

4. The method for image classification based on the image class spatial relationship model as claimed in claim 3, characterized in that: In step S2, the discovery learning module simultaneously takes the known category label image data and the unknown category label image data as input, and respectively passes the known category classification head and a new category discovery header After that, we get C l and C u Logarithmically, the C l and C u Logarithmic fusion is used to obtain the final image classification result.

5. The method for image classification based on image class spatial relationship model according to claim 1, characterized in that: In step S2, the loss function of the image-based spatial relationship model includes a unified cross entropy loss and a knowledge distillation loss, specifically: in, is the loss function of the image-like spatial relationship model, To unify the cross entropy loss, is the knowledge distillation loss and α is a hyperparameter.

6. The method for image classification based on image class spatial relationship model according to claim 5, characterized in that: The unified cross entropy loss includes the cross entropy loss based on the prototype network and the self-labeling loss, namely: in, is the cross entropy loss based on the prototype network, is the self-labeling loss; p(x∈μ j |x) represents the image sample x i Belong to the prototype μ j The probability of represents the prior distribution, represents the posterior distribution, B represents the image sample batch size, which consists of image samples of known categories B l and image samples of unknown categories B u composition.

7. The method for image classification based on image class spatial relationship model according to claim 5, characterized in that: The knowledge distillation loss is specifically: in, represents the knowledge distillation loss of new class samples of unknown class images, represents the knowledge distillation loss of known category image samples, and η is the coefficient for adjusting the distillation loss of known and unknown categories, which is used to balance the model performance of discovering new categories and detecting known categories.

8. The method for image classification based on image class spatial relationship model according to claim 7, characterized in that: Said and It is obtained by the following formula: Where B represents the batch size of image samples, and They respectively represent the KL divergence corresponding to the unknown category image samples and the known category image samples in the knowledge distillation process.

9. The method for image classification based on image class spatial relationship model according to claim 7, characterized in that: The value of η is 0.

01.

10. A device for image classification based on an image class spatial relationship model, comprising a memory and a processor, characterized in that: The memory stores computer program code, and the processor is used to execute the computer program code to implement the method for image classification based on the image class spatial relationship model as described in any one of claims 1-9.

Images