Sample feature space enhancement method and device

Abstract

The embodiments of the present disclosure provide a sample feature space enhancement method and a small sample classifier training method, as well as corresponding devices, equipment and storage media. The method of the embodiment of the present disclosure is for an unlabeled data set, and determines a feature encoder and a legacy dictionary through unsupervised pre-training and unsupervised clustering, and uses the feature encoder and the legacy dictionary to obtain richer sample features based on a small number of new class samples to enhance the feature space of the sample. By applying the method of the embodiment of the present disclosure to small sample classification, the semantic changes of the base class samples can be transferred to the new class samples, thereby generating more reliable new class samples with rich semantic changes based on a small number of new class samples, so as to improve the classification ability of the small sample classifier. In addition, the determination of the feature encoder and the legacy dictionary in the embodiment of the present disclosure does not require label information, so the method of the embodiment of the present disclosure can also be applied to other unlabeled or label-scarce tasks.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and computer vision, and more specifically, to a sample feature space enhancement method and a small sample classifier training method, as well as corresponding devices, equipment and storage media. Background Art

[0002] In recent years, inspired by the rapid learning ability of human beings, researchers hope that after the model has learned a large amount of data of a certain category, it only needs a small number of samples to quickly learn when faced with a new category. Therefore, Few-Shot Classification has attracted more and more attention from researchers and has become a research hotspot in many research fields such as computer vision and machine learning.

[0003] At present, small sample classification based on deep learning has achieved certain research results, but the small sample size greatly increases the difficulty of model learning, making the small sample classification task still face great challenges. Small sample classification technology is usually based on supervised pre-training of large labeled data sets, and its success depends on a large number of labeled samples. However, in actual scenarios, large-scale labeled data sets are usually difficult to obtain, while unlabeled data sets are usually more common. In addition, the pre-training data set itself often contains a lot of rich knowledge that can be used to optimize small sample classification (such as data distribution in the pre-training data set), but the existing small sample classification technology only transfers part of the knowledge about the base class in the pre-training data set by using the model weights obtained by pre-training learning, but does not make full use of the knowledge of the pre-training data set.

[0004] Therefore, an effective small-sample classification method is needed that can make better use of unlabeled datasets and achieve better small-sample classification at a lower annotation cost. Summary of the invention

[0005] In order to solve the above problems, the present invention utilizes feature encoders and legacy dictionaries determined by unsupervised pre-training and unsupervised clustering to obtain richer sample features based on a small number of samples, enhance the feature space of the samples, thereby improving the classification ability of subsequent small sample classifiers and greatly reducing the label annotation cost.

[0006] Embodiments of the present disclosure provide a sample feature space enhancement method and a small sample classifier training method, as well as corresponding devices, equipment and storage media.

[0007] An embodiment of the present disclosure provides a sample feature space enhancement method, comprising: acquiring a first number of samples; determining a first number of feature vectors based on the first number of samples using a pretrained feature encoder, the first number of feature vectors corresponding one-to-one to the first number of samples, the pretrained feature encoder being trained based on a pretraining sample set; and determining a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, the legacy dictionary comprising feature information of a plurality of feature vector clusters, the feature vectors in the plurality of feature vector clusters being pre-generated based on the pretraining sample set using the pretrained feature encoder; wherein the feature space of the first number of samples comprises the first number of feature vectors and the second number of enhanced feature vectors, the second number being not less than the first number, wherein one feature vector in the first number of feature vectors is associated with at least one feature vector in the second number of enhanced feature vectors.

[0008] According to an embodiment of the present disclosure, the method further includes: determining a feature vector of each pre-training sample in the pre-training sample set using the feature encoder; clustering all feature vectors of all pre-training samples in the pre-training sample set to form the plurality of feature vector clusters; generating feature information of each feature vector cluster in the plurality of feature vector clusters; and generating the legacy dictionary using the feature information of each feature vector cluster in the plurality of feature vector clusters.

[0009] According to an embodiment of the present disclosure, for each feature vector cluster of the multiple feature vector clusters, the feature information of the feature vector cluster includes a statistical information group of the feature vector cluster, and the statistical information group includes the mean and covariance matrix of all feature vectors in the feature vector cluster.

[0010] According to an embodiment of the present disclosure, based on the first number of feature vectors, determining a second number of enhanced feature vectors using a legacy dictionary includes: for each feature vector in the first number of feature vectors, searching the legacy dictionary for a feature vector cluster having the highest similarity to the feature vector; and determining a plurality of enhanced feature vectors associated with the feature vector based on feature information of the feature vector cluster; wherein the second number of enhanced feature vectors includes a plurality of enhanced feature vectors determined for each feature vector in the first number of feature vectors.

[0011] According to an embodiment of the present disclosure, searching the legacy dictionary for a feature vector cluster having the highest similarity to the feature vector includes: based on a statistical information group of each feature vector cluster in a plurality of feature vector clusters included in the legacy dictionary, calculating the similarity between the feature vector and each feature vector cluster, and determining the feature vector cluster having the highest similarity to the feature vector; wherein the similarity is cosine similarity, and calculating the similarity between the feature vector and each feature vector cluster includes calculating the cosine similarity between the feature vector and the mean in the statistical information group of the feature vector cluster.

[0012] According to an embodiment of the present disclosure, determining a plurality of enhanced feature vectors associated with the feature vector based on the feature information of the feature vector cluster includes: determining a semantic offset distribution associated with the feature vector based on a statistical information group of the feature vector cluster, the semantic offset distribution being a multivariate Gaussian distribution having a zero vector as a mean and a covariance matrix of the feature vector cluster as a covariance; determining a plurality of semantic offset vectors associated with the feature vector based on the semantic offset distribution; and determining the plurality of enhanced feature vectors based on the feature vector and the plurality of semantic offset vectors associated with the feature vector.

[0013] According to an embodiment of the present disclosure, the feature encoder is obtained by performing unsupervised pre-training on the pre-training sample set, and all pre-training samples in the pre-training sample set are unlabeled samples; wherein, an online encoder and a momentum encoder are used to perform unsupervised pre-training on the pre-training sample set, the online encoder includes the feature encoder, and the momentum encoder includes a copy of the online encoder, the parameters of the momentum encoder are obtained based on a momentum update of the parameters of the online encoder, and the momentum update is based on a weighted sum of the parameters of the momentum encoder and the parameters of the online encoder.

[0014] According to an embodiment of the present disclosure, at least two sample views are generated for each pre-training sample; for each pre-training sample in the pre-training sample set, a first error between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample set, and a second error between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample set are determined; and based on the first error and the second error associated with all pre-training samples in the pre-training sample set, the pre-trained feature encoder is determined.

[0015] According to an embodiment of the present disclosure, the feature encoder includes multiple pending parameters; based on the first error and the second error associated with all pre-training samples in the pre-training sample set, determining the pre-trained feature encoder includes: determining current estimated values ​​of the multiple pending parameters; determining the target error of the unsupervised pre-training based on the sum of all first errors associated with all pre-training samples in the pre-training sample set and the sum of all second errors associated with all pre-training samples; when the target error satisfies a predetermined condition, determining the values ​​of the multiple pending parameters of the feature encoder based on the current estimated values ​​of the multiple pending parameters.

[0016] According to an embodiment of the present disclosure, determining a first error between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample set also includes: determining at least one positive sample similarity between the feature vector determined by the online encoder for any sample view of the pre-training sample and the feature vector determined by the momentum encoder for any other sample view of the pre-training sample; determining multiple negative sample similarities between the feature vector determined by the online encoder for any sample view of the pre-training sample and the feature vector determined by the momentum encoder for any sample view of other pre-training samples in the pre-training sample set; and determining the first error based on the at least one positive sample similarity and the multiple negative sample similarities. an error; wherein, determining a second error between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample set, further includes: determining at least one positive sample similarity between the feature vector determined by the momentum encoder for any sample view of the pre-training sample and the feature vector determined by the online encoder for any other sample view of the pre-training sample; determining multiple negative sample similarities between the feature vector determined by the momentum encoder for any sample view of the pre-training sample and the feature vector determined by the online encoder for any sample view of other pre-training samples in the pre-training sample set; and determining the second error based on the at least one positive sample similarity and the multiple negative sample similarities.

[0017] An embodiment of the present disclosure provides a small sample classifier training method, comprising: acquiring a first number of samples; determining a feature space of the first number of samples, the feature space comprising a first number of feature vectors and a second number of enhanced feature vectors; training the small sample classifier based on the feature vectors and enhanced feature vectors in the feature space; wherein the feature space of the first number of samples is determined by the method described above.

[0018] An embodiment of the present disclosure provides a sample feature space enhancement device, comprising: a sample acquisition module, configured to acquire a first number of samples; a feature vector determination module, configured to determine a first number of feature vectors based on the first number of samples using a pretrained feature encoder, the first number of feature vectors corresponding to the first number of samples one-to-one, the pretrained feature encoder being trained based on a pretraining sample set; a feature vector enhancement module, configured to determine a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, the legacy dictionary including feature information of a plurality of feature vector clusters, the feature vectors in the plurality of feature vector clusters being pre-generated based on the pretraining sample set using the pretrained feature encoder; wherein the feature space of the first number of samples includes the first number of feature vectors and the second number of enhanced feature vectors, the second number being not less than the first number, wherein one feature vector in the first number of feature vectors is associated with at least one feature vector in the second number of enhanced feature vectors.

[0019] According to an embodiment of the present disclosure, the device also includes a legacy dictionary generation module, which is configured to: determine a feature vector of each pre-training sample in the pre-training sample set using the feature encoder; cluster all feature vectors of all pre-training samples in the pre-training sample set to form the plurality of feature vector clusters; generate feature information of each feature vector cluster in the plurality of feature vector clusters; and generate the legacy dictionary using the feature information of each feature vector cluster in the plurality of feature vector clusters.

[0020] According to an embodiment of the present disclosure, the feature vector enhancement module determines a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, including: for each feature vector in the first number of feature vectors, searching the legacy dictionary for a feature vector cluster having the highest similarity to the feature vector; and determining a plurality of enhanced feature vectors associated with the feature vector based on feature information of the feature vector cluster; wherein the second number of enhanced feature vectors includes a plurality of enhanced feature vectors determined for each feature vector in the first number of feature vectors.

[0021] According to an embodiment of the present disclosure, the feature encoder is obtained by performing unsupervised pre-training on the pre-training sample set, and all pre-training samples in the pre-training sample set are unlabeled samples; wherein, an online encoder and a momentum encoder are used to perform unsupervised pre-training on the pre-training sample set, the online encoder includes the feature encoder, and the momentum encoder includes a copy of the online encoder, the parameters of the momentum encoder are obtained based on a momentum update of the parameters of the online encoder, and the momentum update is based on a weighted sum of the parameters of the momentum encoder and the parameters of the online encoder.

[0022] An embodiment of the present disclosure provides a sample feature space enhancement device, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the method as described above is performed.

[0023] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the instructions are executed by a processor, the instructions are used to implement the method described above.

[0024] The embodiment of the present disclosure provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method as described above.

[0025] Compared with the traditional method based on large-scale labeled data sets, the method provided by the embodiment of the present disclosure utilizes unsupervised pre-training, thereby avoiding the need for labeled samples in supervised pre-training, thereby greatly reducing the label annotation cost.

[0026] The embodiments of the present disclosure target unlabeled datasets, determine feature encoders and legacy dictionaries through unsupervised pre-training and unsupervised clustering, and use feature encoders and legacy dictionaries to obtain richer sample features based on a small number of new class samples to enhance the feature space of the samples. By applying the method of the embodiments of the present disclosure to small sample classification, the semantic changes of base class samples can be transferred to new class samples, thereby generating more reliable new class samples with rich semantic changes based on a small number of new class samples to enhance the classification ability of the small sample classifier. In addition, the determination of the feature encoder and legacy dictionary in the embodiments of the present disclosure does not require label information, so the method of the embodiments of the present disclosure can also be applied to other unlabeled or label-scarce tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0028] Figure 1A is a flow chart showing a sample feature space enhancement method according to an embodiment of the present disclosure;

[0029] Figure 1B is a schematic flowchart showing sample feature space enhancement according to an embodiment of the present disclosure;

[0030] Figure 2 is a schematic diagram illustrating unsupervised contrastive learning pre-training according to an embodiment of the present disclosure;

[0031] Figure 3A is a flow chart illustrating construction of a legacy dictionary according to an embodiment of the present disclosure;

[0032] Figure 3B is a schematic diagram showing construction of a legacy dictionary according to an embodiment of the present disclosure;

[0033] Figure 4 is a schematic diagram showing sample feature space enhancement according to an embodiment of the present disclosure;

[0034] Figure 5 is a flow chart showing a small sample classifier training method according to an embodiment of the present disclosure;

[0035] Figure 6 is a schematic diagram illustrating small sample classifier training according to an embodiment of the present disclosure;

[0036] Figure 7 is a schematic diagram showing a sample feature space enhancement device according to an embodiment of the present disclosure;

[0037] Figure 8 A schematic diagram of a sample feature space enhancement device according to an embodiment of the present disclosure is shown;

[0038] Fig. 9 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure; and

[0039] Fig.10 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0041] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0043] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0044] The sample feature space enhancement method disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. For example, for the sample feature space enhancement method based on artificial intelligence, it can quickly learn new categories based on a small number of samples in a way similar to how humans quickly learn new objects through a very small number of samples. Artificial intelligence, by studying the design principles and implementation methods of various intelligent machines, enables the sample feature space enhancement method disclosed in the present invention to have the function of migrating the semantic changes of base class samples to new class samples, thereby generating more reliable new class samples with rich semantic changes based on a small number of new class samples to enhance the classification ability of small sample classifiers.

[0045] The sample feature space enhancement method disclosed in the present invention may be based on machine learning. For example, the sample feature space enhancement method disclosed in the present invention may be based on unsupervised learning in machine learning. Unsupervised learning is a type of machine learning technology used to find patterns in data, which aims to find commonalities or associations in a large amount of unlabeled feature data, rather than outputting specific prediction values ​​like supervised learning. For example, the sample feature space enhancement method disclosed in the present invention may be based on a clustering algorithm in machine learning. Clustering algorithms belong to unsupervised learning, among which the K-means clustering algorithm is one of the most famous clustering methods, which aims to find a partitioning scheme of K clusters in an iterative manner so that the cost function corresponding to the clustering result is minimized. Considering that the K-means clustering algorithm has excellent speed and good scalability, in the sample feature space enhancement method disclosed in the present invention, the K-means method may be preferably used to cluster the feature vectors of all unlabeled images of the pre-training data set to generate multiple cluster clusters. However, the present disclosure only uses the above-mentioned K-means clustering method as an example rather than a limitation to perform data clustering, and other clustering algorithms that can achieve similar effects can also be applied to the sample feature space enhancement method of the present disclosure.

[0046] In addition, the sample feature space enhancement method disclosed in the present invention can be based on computer vision (CV) technology. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Therefore, based on computer vision technology, the sample feature space enhancement method disclosed in the present invention can obtain feature information from image samples, and perform data enhancement on the feature information to generate richer feature information with semantic changes.

[0047] Therefore, the sample feature space enhancement method disclosed in the present invention can also be based on data augmentation. Data augmentation is an effective method for compensating for the lack of training data in image classification. It artificially expands the training set by performing multiple transformations on the data and adding new data to the original small sample data set, which is very suitable for small sample classification problems. The sample feature space enhancement method disclosed in the present invention can generate new class samples with rich semantic changes based on a small number of new class samples through data augmentation, thereby improving the performance of small sample classification.

[0048] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as machine learning and computer vision. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0049] Figure 1A is a flow chart showing a sample feature space enhancement method 100 according to an embodiment of the present disclosure. Figure 1B It is a schematic flowchart showing sample feature space enhancement according to an embodiment of the present disclosure.

[0050] Existing small sample classification techniques usually perform supervised pre-training on large labeled data sets. The pre-training tasks usually include meta-learning tasks and classification tasks. Among them, the meta-learning task divides the labeled data set into many sub-tasks, each task contains several (for example, 5) categories, each category contains several small samples for training (for example, 5 samples for each category), and several small samples for evaluation (for example, 15 samples for each category). Meta-learning enables the neural network model to learn new concepts from a small number of samples by allowing it to learn these sub-tasks, thereby having the ability to classify small samples. For the classification task, it regards the labeled data set as a whole, that is, it does not divide the data set, but directly pre-trains the data set for a fully supervised classification task, thereby obtaining a reliable feature encoder. In the small sample learning stage, given a small number of labeled training samples, the feature encoder obtained through previous pre-training is used to extract feature vectors, and then a new classifier is trained based on these feature vectors, thereby having the ability to classify small samples.

[0051] However, the main challenge faced by small sample classification technology is often that there are too few labeled samples, while the training of deep learning models depends on sufficient sample data. In many practical application scenarios, it is difficult to directly obtain a large amount of labeled data, and professional data collection and annotation require manual participation, which is costly. Many tasks are difficult to obtain sufficient and reliable supervised information in a timely manner.

[0052] Moreover, the above-mentioned small sample classification technology only transfers part of the knowledge about the base classes in the pre-training data set by utilizing the model weights obtained by pre-training learning, but does not make full use of the large amount of rich knowledge (such as the data distribution in the pre-training data set) included in the pre-training data set itself that can be used to optimize small sample classification, resulting in a waste of this part of knowledge and is not conducive to the effective training of small sample classifiers.

[0053] Based on this, the present disclosure provides a sample feature space enhancement method, which migrates the data distribution information in the pre-training dataset to enhance the feature space of the sample by performing unsupervised pre-training and unsupervised clustering based on an unlabeled dataset.

[0054] Compared with the traditional method based on large-scale labeled data sets, the method provided by the embodiment of the present disclosure utilizes unsupervised pre-training, thereby avoiding the need for labeled samples in supervised pre-training, thereby greatly reducing the label annotation cost.

[0055] The embodiments of the present disclosure target unlabeled datasets, determine feature encoders and legacy dictionaries through unsupervised pre-training and unsupervised clustering, and use feature encoders and legacy dictionaries to obtain richer sample features based on a small number of new class samples to enhance the feature space of the samples. By applying the method of the embodiments of the present disclosure to small sample classification, the semantic changes of base class samples can be transferred to new class samples, thereby generating more reliable new class samples with rich semantic changes based on a small number of new class samples to enhance the classification ability of the small sample classifier. In addition, the determination of the feature encoder and legacy dictionary in the embodiments of the present disclosure does not require label information, so the method of the embodiments of the present disclosure can also be applied to other unlabeled or label-scarce tasks.

[0056] like Figure 1A As shown, in step 101, a first number of samples may be acquired.

[0057] Optionally, the first number of samples may be a small number of labeled new class samples or base class samples.

[0058] For example, these small number of labeled new class samples or base class samples can be converted into multiple samples via the sample feature space enhancement method disclosed in the present invention for training a small sample classifier, and the converted samples have the same category labels as their corresponding original samples.

[0059] Optionally, these small number of labeled new class samples or base class samples may belong to multiple categories respectively, wherein each category includes several samples (eg, 1, 5 or 10 samples).

[0060] Optionally, the first number of samples may be image samples of different categories. For example, they may be image samples of rare diseases or new diseases with only a small number of samples in medical pathology images (e.g., digital pathology whole slide images (WSI)). Base class samples in medical pathology images are often easy to obtain, while samples of rare diseases and new categories are not easy to obtain and are scarce in number. Their labeling requires experts in certain fields and has a high labeling cost. However, since their classification is closely related to the diagnosis and subsequent treatment of the disease, the automatic and accurate classification and labeling of such samples has important practical significance.

[0061] After obtaining the first number of samples, features of these samples may be extracted for subsequent feature space enhancement.

[0062] In step 102, a pretrained feature encoder may be used to determine a first number of feature vectors based on the first number of samples, wherein the first number of feature vectors corresponds one-to-one to the first number of samples, and the pretrained feature encoder is trained based on a pretraining sample set.

[0063] Optionally, for each sample in the first number of samples, the feature encoder obtained by pre-training can be used to map the high-dimensional input sample data to a low-dimensional feature space, and generate a feature vector of the sample in its feature space. At this time, the sample feature space includes these original feature vectors (i.e., the feature vectors of the first number), and the sample feature space is as follows: Figure 1B The original sample feature space is shown in FIG. Therefore, the subsequent enhancement of the sample feature space can be regarded as the enhancement of the feature vector of the sample therein.

[0064] According to an embodiment of the present disclosure, the feature encoder may be obtained by performing unsupervised pre-training on the pre-training sample set, and all pre-training samples in the pre-training sample set may be unlabeled samples.

[0065] Optionally, the feature encoder may be obtained by unsupervised contrastive learning pre-training based on a pre-training sample set. Specifically, the process of obtaining the feature encoder based on unsupervised contrastive learning pre-training may be as follows: Figure 2 described.

[0066] Optionally, since the purpose of performing neural network pre-training in the sample feature space enhancement method of the present invention is to obtain a better feature encoder, and it is not intended to limit the pre-training to unsupervised pre-training, the unsupervised pre-training adopted by the method of the present invention is only one of many pre-training tasks, which is only used as an example rather than a limitation. Therefore, the unsupervised pre-training in the present invention can also be replaced by other pre-training tasks, such as fully supervised classification task pre-training, image reproduction task pre-training, clustering task pre-training, etc.

[0067] In step 103, a second number of enhanced feature vectors may be determined based on the first number of feature vectors using a legacy dictionary, wherein the legacy dictionary may include feature information of a plurality of feature vector clusters, and the feature vectors in the plurality of feature vector clusters may be pre-generated based on the pre-trained sample set using the pre-trained feature encoder.

[0068] Optionally, the legacy dictionary can be regarded as the legacy of the pre-trained sample set used to construct the legacy dictionary, which is an inheritance of the knowledge included in the unlabeled dataset (such as the data distribution of the unlabeled dataset, etc.). The sample feature space enhancement method disclosed in the present invention can make full use of the knowledge of the unlabeled dataset through the legacy dictionary.

[0069] According to an embodiment of the present disclosure, determining a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary in step 103 may include: for each feature vector in the first number of feature vectors, searching the legacy dictionary for a feature vector cluster having the highest similarity to the feature vector; and determining a plurality of enhanced feature vectors associated with the feature vector based on feature information of the feature vector cluster.

[0070] According to an embodiment of the present disclosure, the second number of enhanced feature vectors includes a plurality of enhanced feature vectors determined for each feature vector in the first number of feature vectors.

[0071] Optionally, after acquiring the feature vector of a new sample, a feature vector cluster that is most similar to the feature vector of the sample may be searched from the legacy dictionary to transfer the knowledge of the feature vector cluster to the knowledge domain including the feature vector of the sample.

[0072] For example, when the newly acquired sample is an image of a dog and the dog does not belong to the base class (recognizable category), a category that is most similar to it (e.g., wolf) can be found from the legacy dictionary, and the knowledge of the category (e.g., the law of change of wolf features) can be applied to the feature vector of the sample to generate multiple enhanced feature vectors rich in semantic changes similar to the found category based on the feature vector of the sample.

[0073] According to an embodiment of the present disclosure, the feature space of the first number of samples may include the first number of feature vectors and the second number of enhanced feature vectors, the second number being not less than the first number, wherein one feature vector in the first number of feature vectors is associated with at least one feature vector in the second number of enhanced feature vectors.

[0074] Optionally, feature enhancement of any sample among the first number of samples may be performed multiple times based on the original feature vector of the sample to generate multiple enhanced feature vectors, and the multiple enhanced samples corresponding to the multiple enhanced feature vectors may have the same category labels as the corresponding original samples.

[0075] Optionally, the feature space of the first number of samples is enhanced based on the legacy dictionary, such as Figure 1B As shown, the enhanced sample feature space may include an original first number of feature vectors and a newly generated second number of enhanced feature vectors.

[0076] Figure 2 is a schematic diagram illustrating unsupervised contrastive learning pre-training according to an embodiment of the present disclosure.

[0077] Optionally, the construction of the legacy dictionary and the feature encoder used in the sample space enhancement can be based on Figure 2 Determined by unsupervised contrastive learning pre-training as shown.

[0078] According to an embodiment of the present disclosure, an online encoder and a momentum encoder can be used to perform unsupervised pre-training on the pre-training sample set, the online encoder can include the feature encoder, and the momentum encoder can include a copy of the online encoder, the parameters of the momentum encoder can be obtained based on the momentum update of the parameters of the online encoder, and the momentum update can be based on the weighted sum of the parameters of the momentum encoder and the parameters of the online encoder.

[0079] like Figure 2 As shown, the online encoder may include a feature encoder f φ , and a mapping head f for implementing a specific training task g and prediction head f q (e.g., for implementing classification tasks), where the function f represents a neural network model, and the subscripts φ, g, and q represent the parameter sets corresponding to each model. The momentum encoder may include a feature encoder and the mapping header in Know They represent the momentum copies of the parameters φ and g in the online encoder respectively, which are obtained by performing momentum updates on their corresponding parameters.

[0080] Optionally, in the parameter updating phase, only the parameters of the online encoder may be updated, and the parameters of the momentum encoder may be updated based on its own parameters in the previous iteration and the updated parameters of the online encoder.

[0081] For example, for a batch of samples, in the kth optimization iteration of the online encoder and momentum encoder, the parameters of the momentum encoder are and The momentum update can be expressed as:

[0082]

[0083] Wherein, m represents momentum, which makes the parameters of the momentum encoder gradually approach the parameters of the online encoder.

[0084] According to an embodiment of the present disclosure, at least two sample views may be generated for each pre-training sample.

[0085] like Figure 2As shown in the figure, taking the unlabeled image samples as an example, each batch of unlabeled image samples x will first be randomly enhanced by data t to obtain two views x1, x2~t(x) (view 1 and view Figure 2 For example, taking a dog as an example, view 1 may be a view of the head, view Figure 2 Can be a view of the tail.

[0086] For example, for any sample in the batch of unlabeled image samples, a random transformation (such as cropping, rotation, and color change) may be performed on the image sample to generate at least two sample views of the sample.

[0087] Optionally, sample views from the same sample can be regarded as positive samples of each other, and for these sample views, sample views from other samples can be regarded as negative samples.

[0088] Optionally, the contrastive learning pre-training of the present disclosure aims to maximize the similarity between positive sample pairs (two views from the same image sample) while minimizing the similarity between negative sample pairs (two views from different image samples).

[0089] According to an embodiment of the present disclosure, for each pre-training sample in the pre-training sample set, a first error between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample set, and a second error between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample set are determined; and the pre-trained feature encoder is determined based on the first error and the second error associated with all pre-training samples in the pre-training sample set.

[0090] According to an embodiment of the present disclosure, determining the above-mentioned first error may also include: determining at least one positive sample similarity between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample; determining multiple negative sample similarities between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any sample view of other pre-training samples in the pre-training sample set; and determining the first error based on the at least one positive sample similarity and the multiple negative sample similarities.

[0091] According to an embodiment of the present disclosure, determining the above-mentioned second error may also include: determining at least one positive sample similarity between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample; determining multiple negative sample similarities between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any sample view of other pre-training samples in the pre-training sample set; and determining the second error based on the at least one positive sample similarity and the multiple negative sample similarities.

[0092] Optionally, views x1 and x2 are fed into the online encoder and momentum encoder, and the momentum encoder and online encoder, respectively, to obtain feature vectors z1 = f q (f g (f φ (x1))), z2=f q (f g (f φ (x2))),

[0093] For example, when views x1, x2 are input into the online encoder and momentum encoder respectively, x1 can be regarded as a query to x2, aiming to maximize the similarity of views from the same sample, which can be expressed as minimizing the contrastive loss function:

[0094]

[0095] Among them, u, v represent two sets of corresponding eigenvectors, v + represents the positive sample feature vector, that is, it corresponds to another view from the same sample as u, v - represents the negative sample feature vector, that is, the view corresponding to other samples in the current batch of samples, τ represents the temperature hyperparameter of the model, It means that the expression in (·) is about v - Perform the summation.

[0096] For example, the first error can be expressed as: The second error can be expressed as:

[0097] According to an embodiment of the present disclosure, the feature encoder may include multiple parameters to be determined.

[0098] According to an embodiment of the present disclosure, determining the pretrained feature encoder based on the first error and the second error associated with all pre-training samples in the pre-training sample set may include: determining current estimated values ​​of the multiple pending parameters; determining the target error of the unsupervised pre-training based on the sum of all first errors associated with all pre-training samples in the pre-training sample set and the sum of all second errors associated with all pre-training samples; and determining values ​​of multiple pending parameters of the feature encoder based on the current estimated values ​​of the multiple pending parameters when the target error satisfies a predetermined condition.

[0099] Optionally, for a batch of unlabeled image samples, the above contrast loss function can be expressed as the sum of the contrast loss functions for each sample as the input of the online encoder:

[0100]

[0101] Among them, ∑u(·) can represent the sum of the expression in (·) with respect to u, that is, the sum of the first errors corresponding to each sample view of all image samples in the batch of unlabeled image samples.

[0102] Optionally, we can also change L in (3) ctr (u, v) is averaged with respect to the number of image samples in the batch to determine the mean of the loss values ​​of the image samples in the batch. The above expression of the present disclosure is only used as an example and can be appropriately adjusted without affecting the optimization result.

[0103] As mentioned above, in the two cases where views x1, x2 are fed into the online encoder and momentum encoder respectively, and the momentum encoder and online encoder respectively, the overall objective function optimized by contrastive learning pre-training can be expressed as minimizing the average contrastive loss function in these two cases:

[0104]

[0105] Before minimizing the total objective function, all eigenvectors may be normalized, that is, z=z / ||z||.

[0106] By minimizing the above objective function, the feature encoder used for subsequent processing of the sample feature space enhancement method disclosed in the present invention can be determined. Figure 2 The mapping head and prediction head used for task-specific training may not be used in subsequent processing.

[0107] Next, the pre-trained feature encoder can be used to build a legacy dictionary. Specifically, Figure 3A is a flow chart illustrating construction of a legacy dictionary according to an embodiment of the present disclosure. Figure 3Bis a schematic diagram illustrating construction of a legacy dictionary according to an embodiment of the present disclosure.

[0108] The sample feature space enhancement method 100 is as follows Figure 1A In addition to steps 101-103 shown in FIG. Figure 3A Step 104 is shown for building a legacy dictionary.

[0109] In step 1041, the feature encoder may be used to determine a feature vector of each pre-training sample in the pre-training sample set.

[0110] Optionally, corresponding feature vectors may be extracted from all samples in the entire pre-training sample set to determine a feature vector set of the pre-training sample set.

[0111] like Figure 3B As shown, the feature encoder f φ Taking the pre-training sample set as input, output the feature vector set of the pre-training sample set for subsequent processing, where the feature encoder f φ It is a neural network model that has been pre-trained.

[0112] In step 1042, all feature vectors of all pre-training samples in the pre-training sample set may be clustered to form the plurality of feature vector clusters.

[0113] Optionally, the feature vector set may be clustered to group the most similar feature vectors into the same feature vector cluster.

[0114] Optionally, a K-means clustering algorithm can be used to cluster all feature vectors of the pre-trained sample set, with distance as the standard for measuring the similarity between feature vectors, that is, the smaller the distance between feature vectors, the higher their similarity, and therefore the more likely they are to be in the same cluster, while feature vectors with lower similarity are divided into different clusters.

[0115] Optionally, the embodiment of the present disclosure uses a K-means clustering algorithm to construct a heritage dictionary, but the K-means clustering algorithm is only used as an example and not a limitation, and any other clustering algorithm that can achieve the same purpose can also be used to cluster the feature vectors of the samples to construct a heritage dictionary. According to the embodiment of the present disclosure, the clustering algorithm can be supervised clustering or unsupervised clustering.

[0116] Therefore, based on the above clustering algorithm, multiple (denoted as C) feature vector clusters based on the similarity principle can be generated.

[0117] In step 1043 , feature information of each feature vector cluster in the plurality of feature vector clusters may be generated.

[0118] According to an embodiment of the present disclosure, for each feature vector cluster of the multiple feature vector clusters, the feature information of the feature vector cluster may include a statistical information group of the feature vector cluster, and the statistical information group may include the mean and covariance matrix of all feature vectors in the feature vector cluster.

[0119] Optionally, statistical information may be determined for each of the plurality of feature vector clusters to determine transferable knowledge (such as potential semantic shift, etc.) in the cluster.

[0120] Optionally, for the i-th feature vector cluster among the multiple feature vector clusters, the prototype c of the feature vector cluster can be calculated i (i.e., mean) and covariance matrix ∑ i , so the characteristic information of the feature vector clustering can include the bigram (c i ,∑ i ).

[0121] In step 1044, the legacy dictionary may be generated using feature information of each feature vector cluster in the plurality of feature vector clusters.

[0122] Optionally, the tuple (c i ,∑ i ) can be used as the i-th item in the legacy dictionary, so the legacy dictionary can be constructed by clustering the corresponding bigrams based on multiple feature vectors

[0123] like Figure 3B As shown, after the feature vector set of the pre-trained sample set is formed into multiple clusters by a specific clustering algorithm, its statistical information (including mean and covariance) constitutes a legacy dictionary. The legacy dictionary can include the prototype of each feature vector cluster in the multiple feature vector clusters (the center of each cluster) and its data distribution (data distribution around the cluster center), reflecting the semantic changes of the feature vectors in the cluster.

[0124] Optionally, the embodiments of the present disclosure use the last layer output of the feature encoder (convolutional neural network) to construct a heritage dictionary, but in fact the outputs of different layers of the convolutional neural network can be used to construct a multi-scale, multi-level heritage dictionary, thereby achieving better sample feature space enhancement.

[0125] Therefore, based on the above step 104, according to an embodiment of the present disclosure, searching the legacy dictionary for the feature vector cluster with the highest similarity to the feature vector in step 103 may include: calculating the similarity between the feature vector and each feature vector cluster based on a statistical information group of each feature vector cluster in a plurality of feature vector clusters included in the legacy dictionary, and determining the feature vector cluster with the highest similarity to the feature vector.

[0126] Figure 4 is a schematic diagram illustrating sample feature space enhancement according to an embodiment of the present disclosure.

[0127] like Figure 4 As shown, in step ①, for each sample feature vector in the original sample feature space, based on the similarity between the feature vector and the statistical information group of multiple feature vector clusters in the legacy dictionary, the feature vector cluster that is most similar to the feature vector can be found in the legacy dictionary ( Figure 4 , which is shown as the association between the feature vector and the feature vector cluster in step ① in the form of an arrow). For example, for the feature vector located at the lower left in the original sample feature space, its most similar feature vector cluster in the legacy dictionary is the cluster shown in the dashed box at the lower left in the legacy dictionary.

[0128] According to an embodiment of the present disclosure, the similarity may be cosine similarity.

[0129] According to an embodiment of the present disclosure, calculating the similarity between the feature vector and each feature vector cluster may include calculating the cosine similarity between the feature vector and a mean value in a statistical information group of the feature vector cluster.

[0130] Optionally, for any feature vector z in the sample feature space, searching for the feature vector cluster with the highest similarity in the legacy dictionary can be expressed as:

[0131]

[0132] in, represents the corresponding i when maximizing the expression in (·), i * Represents the clustering of the found feature vectors.

[0133] After determining the feature vector clusters with the highest similarity to the sample feature vectors in the original sample feature space through step ①, in step ②, data enhancement can be performed on these sample feature vectors based on the feature information of these feature vector clusters.

[0134] According to an embodiment of the present disclosure, determining a plurality of enhanced feature vectors associated with the feature vector based on the feature information of the feature vector cluster in step 103 may include: determining a semantic offset distribution associated with the feature vector based on a statistical information group of the feature vector cluster, the semantic offset distribution being a multivariate Gaussian distribution having a zero vector as a mean and a covariance matrix of the feature vector cluster as a covariance; determining a plurality of semantic offset vectors associated with the feature vector based on the semantic offset distribution; and determining the plurality of enhanced feature vectors based on the feature vector and the plurality of semantic offset vectors associated with the feature vector.

[0135] Optionally, for the feature vector of a new class sample, it can be considered that it has a similar change pattern as a feature vector cluster similar to it, that is, the semantic offset of the feature vector cluster can be transferred to the feature vector of the sample.

[0136] For example, when the new class sample is an image of a dog, we can find the most similar category (e.g., wolf) from the legacy dictionary and transfer the semantic offset of that category to the sample. In other words, we can consider that the changes that occur in samples with the dog category are similar to the changes in samples with the wolf category. That is, through reasonable associations associated with similar categories, we can generate an enhanced feature vector based on the new class sample that is rich in semantic changes similar to those of similar categories.

[0137] like Figure 4 As shown in step ② in , based on the original sample feature space and the feature information (statistical information group) of the feature vector clustering obtained by searching for each feature vector in the original sample feature space, the enhanced sample feature space can be determined to obtain the enhanced sample.

[0138] Optionally, the enhanced sample feature space may include an original feature vector and an enhanced feature vector, wherein the enhanced feature vector may include a semantic offset relative to the original feature vector, wherein the semantic offset is determined based on feature information of a feature vector cluster that is most similar to the original feature vector.

[0139] Optionally, the enhancement of the original feature vector may be additive enhancement, that is, an enhanced feature vector is generated based on the sum of the original feature vector and the semantic offset vector.

[0140] Optionally, the semantic offset vector may be a random sampling of the above semantic offset distribution. For example, for any feature vector z in the original sample feature space and the most similar feature vector cluster i* found thereto, the semantic offset vector may be expressed as in represents the covariance matrix of the above cluster i* with zero vector as mean and eigenvector as cluster is a multivariate Gaussian distribution with covariance. Therefore, the generated enhanced feature vector can be expressed as

[0141] In step ②, for each original feature vector in the original sample feature space, there may be multiple enhanced feature vectors associated therewith in the enhanced sample feature space. These enhanced feature vectors are centered on the original feature vector and have the above-mentioned semantic offset relative to the original feature vector.

[0142] As described above, the sample feature space enhancement method of the embodiment of the present disclosure captures the semantic changes of the data in the pre-training dataset by migrating the model weights of the feature encoder learned from the pre-training dataset and migrating the data distribution in the pre-training dataset, thereby achieving full utilization of the pre-training dataset.

[0143] Optionally, when the sample feature space enhancement method disclosed in the present invention is applied to small sample classification, multiple reliable and semantically variable enhanced sample features can be generated based on the original sample features, so that the small sample classification performance is improved based on the original sample features and the enhanced sample features, as shown by the classification boundary in the enhanced sample feature space in step ② moving from the dotted line to the solid line.

[0144] Figure 5 is a flow chart illustrating a small sample classifier training method 500 according to an embodiment of the present disclosure. Figure 6 is a schematic diagram illustrating small sample classifier training according to an embodiment of the present disclosure.

[0145] like Figure 5 As shown, in step 501, a first number of samples may be acquired.

[0146] Optionally, the first number of samples may be a small number of labeled new class samples or base class samples, for example, they may be image samples of rare diseases or new diseases with only a small number of samples in medical pathology images (such as WSI images).

[0147] Next, sample feature space enhancement may be performed based on the first number of samples obtained for training the small sample classifier.

[0148] In step 502, a feature space of the first number of samples may be determined, the feature space including a first number of feature vectors and a second number of enhanced feature vectors.

[0149] According to an embodiment of the present disclosure, the feature space of the first number of samples may be determined by the above-mentioned feature space enhancement method.

[0150] like Figure 6As shown, determining the feature space of the first number of samples may include the following three steps:

[0151] ① Determine the feature encoder based on the pre-training sample set through unsupervised pre-training;

[0152] ② Using the determined feature encoder to extract feature vectors of the pre-trained sample set to construct a heritage dictionary based on a clustering algorithm;

[0153] ③ Using the determined feature encoder and the constructed legacy dictionary, based on the first number of samples ( Figure 6 Taking a small number of new class samples as an example) to determine its enhanced feature space, so as to train a small sample classifier based on the enhanced feature space.

[0154] Optionally, in step ①, a feature encoder is used to extract feature vectors of a pre-trained sample set for pre-training based on a specific proxy task (wherein the proxy head is a neural network model for performing the specific proxy task), thereby determining the optimal feature encoder.

[0155] After the feature encoder is trained, only the feature encoder part can be retained and the proxy head part can be discarded. The determined feature encoder can be used in steps ② and ③.

[0156] In step ②, similar to reference Figure 3B As described above, a legacy dictionary may be constructed based on the data distribution of feature vectors of the pre-training sample set.

[0157] Based on the feature encoder and the legacy dictionary determined in step ① and step ②, respectively, in step ③, a first number of feature vectors may be extracted based on a first number of samples (e.g., a small number of new class samples), and a second number of enhanced feature vectors may be generated based on the first number of feature vectors and the legacy dictionary to enhance the feature space of the first number of samples, wherein the feature space enhancement portion may be as described in reference Figure 4 described.

[0158] Through the above sample feature space enhancement, the feature space of the first number of samples is expanded from the original first number of feature vectors to include the first number of feature vectors and the newly generated second number of enhanced feature vectors.

[0159] Therefore, in step 503, the small sample classifier may be trained based on the feature vector and the enhanced feature vector in the feature space.

[0160] By applying the above-mentioned sample feature space enhancement method to small sample classification and transferring the semantic changes of base class samples to new class samples, more reliable new class samples with rich semantic changes can be generated based on a small number of new class samples, so that the small sample classifier can be trained based on these samples to improve the classification ability of the small sample classifier.

[0161] Figure 7 is a schematic diagram showing a sample feature space enhancement device 700 according to an embodiment of the present disclosure.

[0162] The sample feature space enhancement device 700 may include a sample acquisition module 701 , a feature vector determination module 702 and a feature vector enhancement module 703 .

[0163] According to an embodiment of the present disclosure, the data acquisition module 701 may be configured to acquire a first number of samples.

[0164] Optionally, the first number of samples may be a small number of labeled new class samples or base class samples.

[0165] For example, these small number of labeled new class samples or base class samples may belong to multiple categories, respectively, wherein each category includes several samples (eg, 1, 5, or 10 samples).

[0166] For example, the first number of samples may be image samples of different categories, such as image samples of rare diseases or new diseases of which only a small number of samples are available in medical pathology images.

[0167] The feature vector determination module 702 can be configured to use a pre-trained feature encoder to determine a first number of feature vectors based on the first number of samples, wherein the first number of feature vectors corresponds one-to-one to the first number of samples, and the pre-trained feature encoder is trained based on a pre-training sample set.

[0168] Optionally, for each sample in the first number of samples, a feature encoder obtained through pre-training may be used to map high-dimensional input sample data to a low-dimensional feature space, and generate a feature vector for the sample in the feature space.

[0169] According to an embodiment of the present disclosure, the feature encoder is obtained by performing unsupervised pre-training on the pre-training sample set, and all pre-training samples in the pre-training sample set are unlabeled samples.

[0170] Optionally, the feature encoder can be obtained by pre-training with unsupervised contrastive learning based on a pre-training sample set, such as Figure 2 The described method is obtained by pre-training based on unsupervised contrastive learning.

[0171] The feature vector enhancement module 703 can be configured to determine a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, wherein the legacy dictionary includes feature information of multiple feature vector clusters, and the feature vectors in the multiple feature vector clusters are pre-generated based on the pre-trained sample set using the pre-trained feature encoder.

[0172] Optionally, after acquiring the feature vector of a new sample, a feature vector cluster that is most similar to the feature vector of the sample may be searched from the legacy dictionary to transfer the knowledge of the feature vector cluster to the knowledge domain including the feature vector of the sample.

[0173] For example, when the newly acquired sample is an image of a dog and the dog does not belong to the base class (recognizable category), a category that is most similar to it (e.g., wolf) can be found from the legacy dictionary, and the knowledge of the category (e.g., the law of change of wolf features) can be applied to the feature vector of the sample to generate multiple enhanced feature vectors rich in semantic changes similar to the found category based on the feature vector of the sample.

[0174] According to an embodiment of the present disclosure, the feature space of the first number of samples includes the first number of feature vectors and the second number of enhanced feature vectors, the second number is not less than the first number, wherein one feature vector of the first number of feature vectors is associated with at least one feature vector of the second number of enhanced feature vectors.

[0175] Optionally, after the feature space of the first number of samples is enhanced based on the legacy dictionary, the enhanced sample feature space may include the original first number of feature vectors and the newly generated second number of enhanced feature vectors.

[0176] In addition, according to an embodiment of the present disclosure, the sample feature space enhancement device 700 may further include a legacy dictionary generation module 704 .

[0177] According to an embodiment of the present disclosure, the legacy dictionary generation module 704 may be configured to perform the following steps: Figure 3A The method described is used to generate a legacy dictionary.

[0178] Optionally, the legacy dictionary can be regarded as the legacy of the pre-trained sample set used to construct the legacy dictionary, which is an inheritance of the knowledge included in the unlabeled dataset (such as the data distribution of the unlabeled dataset, etc.). The sample feature space enhancement method disclosed in the present invention can make full use of the knowledge of the unlabeled dataset through the legacy dictionary.

[0179] According to yet another aspect of the present disclosure, a sample feature space enhancement device is also provided. Figure 8A schematic diagram of a sample feature space enhancement device 2000 according to an embodiment of the present disclosure is shown.

[0180] like Figure 8 As shown, the sample feature space enhancement device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer readable code, and when the computer readable code is run by the one or more processors 2010, the sample feature space enhancement method described above may be executed.

[0181] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0182] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0183] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig. 9 The architecture of the computing device 3000 shown in FIG. Fig. 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the sample feature space enhancement method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig. 9 One or more components of a computing device are shown.

[0184] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Fig.10 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0185] like Fig.10 As shown, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the sample feature space enhancement method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0186] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the sample feature space enhancement method according to the embodiments of the present disclosure.

[0187] Embodiments of the present disclosure provide a sample feature space enhancement method and a small sample classifier training method, as well as corresponding devices, equipment and storage media.

[0188] Compared with the traditional method based on large-scale labeled data sets, the method provided by the embodiment of the present disclosure utilizes unsupervised pre-training, thereby avoiding the need for labeled samples in supervised pre-training, thereby greatly reducing the label annotation cost.

[0189] The embodiments of the present disclosure target unlabeled datasets, determine feature encoders and legacy dictionaries through unsupervised pre-training and unsupervised clustering, and use feature encoders and legacy dictionaries to obtain richer sample features based on a small number of new class samples to enhance the feature space of the samples. By applying the method of the embodiments of the present disclosure to small sample classification, the semantic changes of base class samples can be transferred to new class samples, thereby generating more reliable new class samples with rich semantic changes based on a small number of new class samples to enhance the classification ability of the small sample classifier. In addition, the determination of the feature encoder and legacy dictionary in the embodiments of the present disclosure does not require label information, so the method of the embodiments of the present disclosure can also be applied to other unlabeled or label-scarce tasks.

[0190] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0191] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0192] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A sample feature space enhancement method, comprising: Acquire a first number of samples, wherein the first number of samples are image samples of different categories; Determining a first number of feature vectors based on the first number of samples using a pre-trained feature encoder, wherein the first number of feature vectors corresponds one-to-one to the first number of samples, wherein the feature encoder is obtained by performing unsupervised pre-training on a pre-training sample set, and all pre-training samples in the pre-training sample set are unlabeled samples; and determining a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, the legacy dictionary including feature information of a plurality of feature vector clusters, the feature vectors in the plurality of feature vector clusters being pre-generated based on the pre-trained sample set using the pre-trained feature encoder; The feature space of the first number of samples includes the first number of feature vectors and the second number of enhanced feature vectors, and the second number is not less than the first number. Wherein, one feature vector of the first number of feature vectors is associated with at least one feature vector of the second number of enhanced feature vectors.

2. The method of claim 1, further comprising: Determine a feature vector of each pre-training sample in the pre-training sample set using the feature encoder; Clustering all feature vectors of all pre-training samples in the pre-training sample set to form the plurality of feature vector clusters; generating feature information of each feature vector cluster in the plurality of feature vector clusters; as well as The legacy dictionary is generated using feature information of each of the plurality of feature vector clusters.

3. The method of claim 2, wherein: For each feature vector cluster of the plurality of feature vector clusters, the feature information of the feature vector cluster includes a statistical information group of the feature vector cluster, wherein the statistical information group includes a mean and a covariance matrix of all feature vectors in the feature vector cluster.

4. The method of claim 2, wherein: Based on the first number of feature vectors, determining a second number of enhanced feature vectors using a legacy dictionary comprises: For each feature vector in the first number of feature vectors, searching the legacy dictionary for a feature vector cluster having the highest similarity to the feature vector; and Determining a plurality of enhanced feature vectors associated with the feature vector based on feature information of the feature vector cluster; The second number of enhanced feature vectors includes a plurality of enhanced feature vectors determined for each feature vector in the first number of feature vectors.

5. The method according to claim 3 or 4, wherein: Finding the feature vector cluster with the highest similarity to the feature vector from the legacy dictionary includes: Based on the statistical information group of each feature vector cluster in the plurality of feature vector clusters included in the legacy dictionary, calculating the similarity between the feature vector and each feature vector cluster, and determining the feature vector cluster with the highest similarity to the feature vector; Wherein, the similarity is cosine similarity, Calculating the similarity between the feature vector and each feature vector cluster includes calculating the cosine similarity between the feature vector and the mean value in the statistical information group of the feature vector cluster.

6. The method according to claim 3 or 4, wherein: Determining a plurality of enhanced feature vectors associated with the feature vector based on the feature information of the feature vector cluster comprises: Determining, based on the statistical information set of the feature vector cluster, a semantic shift distribution associated with the feature vector, the semantic shift distribution being a multivariate Gaussian distribution having a zero vector as a mean and a covariance matrix of the feature vector cluster as a covariance; Based on the semantic offset distribution, determining a plurality of semantic offset vectors associated with the feature vector; and The plurality of enhanced feature vectors are determined based on the feature vector and the plurality of semantic offset vectors associated with the feature vector.

7. The method of claim 1, wherein: An online encoder and a momentum encoder are used to perform unsupervised pre-training on the pre-training sample set, wherein the online encoder includes the feature encoder, and the momentum encoder includes a copy of the online encoder, and the parameters of the momentum encoder are obtained based on a momentum update of the parameters of the online encoder, and the momentum update is based on a weighted sum of the parameters of the momentum encoder and the parameters of the online encoder.

8. The method of claim 7, wherein: Generate at least two sample views for each pre-training sample; For each pre-training sample in the pre-training sample set, determining a first error between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample set, and a second error between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample set; as well as The pre-trained feature encoder is determined based on the first error and the second error associated with all pre-training samples in the pre-training sample set.

9. The method of claim 8, wherein: The feature encoder includes a plurality of parameters to be determined; Determining the pretrained feature encoder based on the first error and the second error associated with all pretraining samples in the pretraining sample set includes: Determining current estimated values ​​of the plurality of undetermined parameters; Determining a target error for the unsupervised pre-training based on a sum of all first errors associated with all pre-training samples in the pre-training sample set and a sum of all second errors associated with all pre-training samples; In a case where the target error satisfies a predetermined condition, values ​​of a plurality of undetermined parameters of the feature encoder are determined based on current estimated values ​​of the plurality of undetermined parameters.

10. The method of claim 8, wherein: Determining a first error between a feature vector determined by the online encoder for any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample set, further comprising: Determine at least one positive sample similarity between a feature vector determined by the online encoder for the any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any other sample view of the pre-training sample; Determine multiple negative sample similarities between a feature vector determined by the online encoder for the any sample view of the pre-training sample and a feature vector determined by the momentum encoder for any sample view of other pre-training samples in the pre-training sample set; and Determining the first error based on the at least one positive sample similarity and the plurality of negative sample similarities; Wherein, determining a second error between a feature vector determined by the momentum encoder for any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample set further includes: Determine at least one positive sample similarity between a feature vector determined by the momentum encoder for the any sample view of the pre-training sample and a feature vector determined by the online encoder for any other sample view of the pre-training sample; Determine a plurality of negative sample similarities between a feature vector determined by the momentum encoder for the any sample view of the pre-training sample and a feature vector determined by the online encoder for any sample view of other pre-training samples in the pre-training sample set; and The second error is determined based on the at least one positive sample similarity and the plurality of negative sample similarities.

11. A small sample classifier training method, comprising: Acquire a first number of samples, wherein the first number of samples are image samples of different categories; determining a feature space of the first number of samples, the feature space comprising a first number of feature vectors and a second number of enhanced feature vectors; Training the small sample classifier based on the feature vector and the enhanced feature vector in the feature space; The feature space of the first number of samples is determined by the method according to any one of claims 1-10.

12. A sample feature space enhancement device, comprising: A sample acquisition module is configured to acquire a first number of samples, wherein the first number of samples are image samples of different categories; a feature vector determination module, configured to determine a first number of feature vectors based on the first number of samples using a pre-trained feature encoder, the first number of feature vectors corresponding to the first number of samples in a one-to-one manner, wherein the feature encoder is obtained by performing unsupervised pre-training on a pre-training sample set, and all pre-training samples in the pre-training sample set are unlabeled samples; a feature vector enhancement module configured to determine a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, the legacy dictionary including feature information of a plurality of feature vector clusters, the feature vectors in the plurality of feature vector clusters being pre-generated based on the pre-trained sample set using the pre-trained feature encoder; The feature space of the first number of samples includes the first number of feature vectors and the second number of enhanced feature vectors, and the second number is not less than the first number. Wherein, one feature vector of the first number of feature vectors is associated with at least one feature vector of the second number of enhanced feature vectors.

13. The device of claim 12, wherein: The apparatus further comprises a legacy dictionary generating module, wherein the legacy dictionary generating module is configured to: Determine a feature vector of each pre-training sample in the pre-training sample set using the feature encoder; Clustering all feature vectors of all pre-training samples in the pre-training sample set to form the plurality of feature vector clusters; generating feature information of each feature vector cluster in the plurality of feature vector clusters; as well as The legacy dictionary is generated using feature information of each of the plurality of feature vector clusters.

14. The device of claim 13, wherein: The feature vector enhancement module determines a second number of enhanced feature vectors based on the first number of feature vectors using a legacy dictionary, comprising: For each feature vector in the first number of feature vectors, searching the legacy dictionary for a feature vector cluster having the highest similarity to the feature vector; and Determining a plurality of enhanced feature vectors associated with the feature vector based on feature information of the feature vector cluster; The second number of enhanced feature vectors includes a plurality of enhanced feature vectors determined for each feature vector in the first number of feature vectors.

15. The device of claim 12, wherein: The feature encoder is obtained by performing unsupervised pre-training on the pre-training sample set, and all pre-training samples in the pre-training sample set are unlabeled samples; An online encoder and a momentum encoder are used to perform unsupervised pre-training on the pre-training sample set, the online encoder includes the feature encoder, and the momentum encoder includes a copy of the online encoder, the parameters of the momentum encoder are obtained based on the momentum update of the parameters of the online encoder, and the momentum update is based on the weighted sum of the parameters of the momentum encoder and the parameters of the online encoder.

16. A sample feature space enhancement device, comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method of any one of claims 1-11 is performed.

17. A computer program product, comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 11.

18. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

Images