Small sample learning image classification method and device, electronic device

Abstract

The present invention discloses a small sample learning image classification method and device, and electronic equipment, the method comprising: extracting high-dimensional features of the image to be predicted; inputting the high-dimensional features of the image to be predicted into a trained classification model to obtain the predicted category of the image to be predicted; wherein the classification model is trained based on an augmented data set, the augmented data set includes real samples and generated samples generated by a generative adversarial network based on the real samples; the generative adversarial network includes a diversity regularizer, a ternary loss regularizer, and a classification regularizer, the diversity regularizer is used to keep the distance between noise vectors greater than a distance threshold, the ternary loss regularizer is used to reduce the difference between samples of the same type and expand the difference between samples of different types, and the classification regularizer is used to keep the distribution of generated samples consistent with the distribution of real samples based on the Pearson correlation coefficient. The present invention can improve the classification effect and classification accuracy of images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically relates to a small sample learning image classification method and device, and electronic equipment. Background Art

[0002] In the wide application of artificial intelligence and machine learning, image classification tasks have become an important research topic. Traditional deep learning models rely on a large amount of labeled data for training to achieve high classification accuracy. However, in many practical application scenarios, obtaining enough labeled data is time-consuming, expensive, and even infeasible in some cases. As an emerging research direction, few-sample learning aims to achieve good classification results with a small number of training samples.

[0003] Small sample learning has a wide range of applications in many fields, including but not limited to medical diagnosis, autonomous driving, security monitoring, and natural language processing. In these applications, efficient and accurate classification with limited labeled data is of great significance for improving system performance and reducing data acquisition costs. Therefore, developing a method and system that can achieve efficient classification under small sample conditions has important research value and application prospects.

[0004] However, the current small sample learning methods each have problems such as complex training process, high computational overhead, limited generalization ability, restricted application scenarios, easy model overfitting and unstable training. As a result, when the model trained by this method is used to classify images, the classification effect is poor and the classification accuracy is poor. Summary of the invention

[0005] The embodiments of the present invention provide a small sample learning image classification method and device, and an electronic device, which can solve the problems of poor classification effect and low classification accuracy of the current small sample learning image classification method.

[0006] In a first aspect, an embodiment of the present invention provides a small sample learning image classification method, the method comprising:

[0007] Extract high-dimensional features of the image to be predicted;

[0008] Input the high-dimensional features of the image to be predicted into the trained classification model to obtain the predicted category of the image to be predicted;

[0009] The classification model is trained based on the augmented data set, which includes real samples and generated samples. The generated samples are pseudo samples generated by the generative adversarial network based on the real samples and the noise vector for the last time.

[0010] Among them, the generative adversarial network includes a diversity regularizer, a ternary loss regularizer and a classification regularizer. The diversity regularizer is used to keep the distance between noise vectors greater than the distance threshold. The ternary loss regularizer is used to reduce the difference between similar samples and expand the difference between heterogeneous samples. The classification regularizer is used to keep the distribution of generated samples consistent with that of query samples based on the Pearson correlation coefficient. The query samples have the same distribution as the real samples and the same types of images.

[0011] In a second aspect, an embodiment of the present invention provides a small sample learning image classification device, the device comprising a processing unit; the processing unit is used to extract high-dimensional features of the image to be predicted; the high-dimensional features of the image to be predicted are input into a trained classification model to obtain a predicted category of the image to be predicted;

[0012] The classification model is trained based on the augmented data set, which includes real samples and generated samples. The generated samples are pseudo samples generated by the generative adversarial network based on the real samples and the noise vector for the last time.

[0013] Among them, the generative adversarial network includes a diversity regularizer, a ternary loss regularizer and a classification regularizer. The diversity regularizer is used to keep the distance between noise vectors greater than the distance threshold. The ternary loss regularizer is used to reduce the difference between similar samples and expand the difference between heterogeneous samples. The classification regularizer is used to keep the distribution of generated samples consistent with that of query samples based on the Pearson correlation coefficient. The query samples have the same distribution as the real samples and the same types of images.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store computer programs; the processor can be used to execute the computer program (instructions) stored in the memory to implement the method of the first aspect above.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: the present invention amplifies the data set by generating an adversarial network instead of using a meta-learning method, thereby avoiding the problems of complex training process and high computational overhead brought about by this method; ensures that the noise vectors between the generated samples have a certain distance through a diversity regularizer, thereby ensuring that there are sufficient differences between the generated samples, avoiding the generator from falling into mode collapse and increasing the diversity of generated samples; minimizes the distance between samples of the same class and maximizes the distance between samples of different classes through a ternary loss regularizer, thereby enhancing the classification model's ability to distinguish features of different categories and reducing the risk of overfitting; the classification regularizer can maintain the linear relationship between generated samples through the Pearson correlation coefficient, so that the features of the generated samples are statistically consistent with the features of the real samples, and reduces the deviation and noise in the data to improve the stability and reliability of the classification model; thereby improving the classification effect and classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the structure of a generative adversarial network provided by an embodiment of the present invention;

[0017] Figure 2 A flowchart of a training method for generating an adversarial network provided by an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of the structure of a feature extractor provided by an embodiment of the present invention;

[0019] Figure 4 An implementation process of a classification model training method provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of a data preprocessing process provided by an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of a classification model training process provided by an embodiment of the present invention;

[0022] Figure 7 A flowchart of an implementation of a small sample learning image classification method provided by an embodiment of the present invention;

[0023] Figure 8 A schematic diagram of the structure of a small sample learning image classification device provided by an embodiment of the present invention;

[0024] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0029] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0031] There are roughly three popular small sample learning methods currently, namely meta-learning, metric learning and data augmentation.

[0032] Meta-learning usually requires a large number of meta-tasks to train the model so that the model can quickly adapt to new tasks and obtain a more adaptable model. However, in practical applications, building a large number of meta-tasks also faces the problem of data scarcity, and the process of using meta-tasks for training is complex and computationally expensive.

[0033] Metric learning optimizes the distance between samples in a high-dimensional feature space by learning a metric space so that the distance between samples of the same type is closer and the distance between samples of different types is farther. Although metric learning can improve the classification performance of the model, its performance is limited when the differences between samples are large. Importantly, metric learning relies heavily on the generalization ability of the model and is difficult to adapt to complex practical application scenarios.

[0034] Data augmentation can generate new training samples to enhance the generalization ability of the model through techniques such as generative adversarial networks. Although this method effectively expands the training data set, the generated samples lack sufficient differentiation, which can easily lead to model overfitting. At the same time, generative adversarial networks often face problems such as mode collapse and unstable training during training, and the reliability and practicality of the model are poor.

[0035] In view of this, the present invention provides a small sample learning image classification method, which can avoid the problems of complex training process and high computational overhead brought by this method by amplifying the data set through a generative adversarial network instead of using a meta-learning method; ensure that the noise vectors between the generated samples have a certain distance through a diversity regularizer, thereby ensuring that there are sufficient differences between the generated samples, which can avoid the generator from falling into mode collapse and increase the diversity of generated samples; minimize the distance between samples of the same class and maximize the distance between samples of different classes through a ternary loss regularizer, thereby enhancing the classification model's ability to distinguish features of different categories and reducing the risk of overfitting; the classification regularizer can maintain the linear relationship between generated samples through the Pearson correlation coefficient, so that the features of the generated samples are statistically consistent with the features of the real samples, and reduce the deviation and noise in the data to improve the stability and reliability of the classification model; thereby improving the classification effect and classification accuracy.

[0036] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0037] The small sample learning image classification method provided in the embodiment of the present invention can be applied to electronic devices such as mobile terminals, personal laptops, supercomputers, etc. The embodiment of the present invention does not impose any restrictions on the specific type of electronic devices.

[0038] Figure 1 Shown is a schematic diagram of the structure of a generative adversarial network provided by an embodiment of the present invention.

[0039] For example, see Figure 1 , the generative adversarial network 100 may include a generator 110 , a discriminator 120 , a diversity regularizer 130 , a ternary loss regularizer 140 , and a classification regularizer 150 .

[0040] Specifically, the generator 110 can generate a pseudo sample according to the input noise vector and the real sample, and the discriminator 120 can evaluate the authenticity of the real sample and the pseudo sample to obtain a discrimination result. The diversity regularizer 130 can calculate the distance between the noise vectors, and by adding the diversity regularization loss in the loss function, the distance between any two noise vectors is greater than the distance threshold, so as to generate more diverse pseudo samples during the training process of the network 100. The ternary loss regularizer 140 can calculate the distance between the anchor sample and its corresponding positive and negative samples, and by adding the triplet loss in the loss function, minimize the distance between the anchor sample and its corresponding positive sample, and maximize the distance between its corresponding negative sample, thereby reducing the difference between the same sample and expanding the difference between the heterogeneous samples, and enhancing the authenticity of the pseudo sample to deceive the discriminator. The classification regularizer 150 can calculate the probability that the pseudo sample and the query sample belong to the same category based on the Pearson correlation coefficient. Since the query sample and the real sample have the same distribution and the same type of image, the distribution of the generated pseudo sample can be kept consistent with the distribution of the real sample by adding the classification loss in the loss function.

[0041] In one possible implementation, the generative adversarial network 100 may be a conditional optimal transmission distance generative adversarial network.

[0042] Exemplarily, the conditional optimal transmission distance generative adversarial network can use the optimal transmission distance as the loss function, and its training process is more stable than that of the traditional generative adversarial network.

[0043] The optimal transmission distance is the minimum cost required to convert one distribution to another. Therefore, the goal of the optimal transmission distance generative adversarial network is to minimize the distribution difference between the generated samples and the real samples, thereby producing higher quality samples without the need to use the sigmoid activation function. The calculation of the optimal transmission distance requires multiple iterations between the generator 110 and the discriminator 120 to ensure that the distance between the generated samples and the discriminator 120 is small enough. The conditional optimal transmission distance generative adversarial network is an improved version of the optimal transmission distance generative adversarial network, on the basis of which the gradient loss is introduced to solve the problems caused by its use of the weight clipping method. In the conditional optimal transmission distance generative adversarial network, adding the gradient penalty term to the loss of the discriminator 120 can provide a more stable training process for the network, improve the quality of the generated samples, and avoid the common training instability and crash problems in traditional generative adversarial networks.

[0044] For example, the discriminator 120 can use a 1-Lipschitz function to evaluate the authenticity of the sample. A gradient penalty term is introduced to implement the constraint on the 1-Lipschitz function. In the conditional optimal transmission distance generative adversarial network, the gradient of the discriminant result is directly constrained, requiring that the norm of the discriminant result at all points does not exceed 1.

[0045] Exemplarily, the gradient penalty term may satisfy the following formula:

[0046] (1.1),

[0047] in, is the gradient penalty term, Express expectations, is the gradient operator, is the interpolation sample, which is the interpolation between the real sample and the pseudo sample. is the distribution of interpolation samples, Indicates that the interpolation samples obey The distribution of Indicates L 2 norm, is the discriminator for the interpolation samples and the categories of the interpolation samples , the context of the interpolated sample The discrimination result of .

[0048] In one example, the optimization goal of the generator 110 may be:

[0049] (1.2),

[0050] in, Express expectations, Represents the distribution from the latent space Extract the noise vector from The generator is based on the noise vector The generated samples are It represents the discriminator’s judgment result on the sample. Minimize .

[0051] Optionally, Can be a standard normal distribution.

[0052] In one example, the optimization objective of the discriminator 120 may be:

[0053] (1.3),

[0054] in, is the distribution of the real samples, For real samples, Indicates that the real sample obeys distributed, is a pseudo sample, is the distribution of pseudo samples, Indicates that the pseudo sample obeys distributed, , are the discriminator’s discrimination results for real samples and fake samples, respectively. Represents maximization , is the weight of the gradient penalty term.

[0055] Figure 2 The flowchart of the training method of a generative adversarial network provided by an embodiment of the present invention is shown. As an example but not a limitation, the training method of the generative adversarial network 100 may include steps S201-S209, which may be applied to the above electronic device, and each step is described below.

[0056] S201, extracting features of the original sample image to obtain a real sample and a query sample.

[0057] In a possible implementation, a trained feature extractor can be used to extract high-dimensional features of the original sample image. The high-dimensional features of the original sample image are then divided into a support set and a query set according to a certain ratio. The real samples in the support set can be used to input into the network 100 to generate pseudo samples, and the query samples in the query set are used to calculate the loss of the network 100.

[0058] Exemplarily, the distribution of real samples and query samples and the types of images included are the same to ensure fairness and accuracy when the network 100 is subsequently tested.

[0059] Specifically, the real samples in the support set need to be evenly distributed and cover all category features of the image.

[0060] In one example, see Figure 3 , the feature extractor 300 may include a residual network submodule and an attention mechanism submodule.

[0061] Exemplarily, the residual network submodule is a deep neural network model that can adopt the ResNet architecture. The introduction of the residual network can solve the problem of gradient disappearance in the deep network. The residual network submodule can gradually extract important features in the original sample image through multiple convolutional layers and pooling layers.

[0062] Exemplarily, the attention mechanism submodule can enhance the feature extractor's extraction effect on key features by weighted summation, helping the feature extractor to focus on important areas in the image.

[0063] Exemplarily, a data set containing multiple categories of images and having a sufficiently large amount of data can be used to train the feature extractor 300 to obtain a trained feature extractor. The images in the data set can be labeled with their categories. The trained feature extractor can extract a high-dimensional feature vector that effectively expresses the main information of the data through multiple convolutional layers and pooling layers.

[0064] S202, the generator generates pseudo samples for this round of training based on real samples and noise vectors.

[0065] S203, the discriminator evaluates the authenticity of the pseudo samples and real samples of this round of training to obtain the discrimination result of this round of training.

[0066] S204, the diversity regularizer calculates the distance between the noise vectors to obtain the output result of the diversity regularizer in this round of training.

[0067] In one example, the diversity regularizer can calculate and output the Euclidean distance between noise vectors. These distance values ​​can be used to calculate the diversity regularization loss. The diversity regularization loss can constrain the distance between noise vectors to be greater than a distance threshold.

[0068] For example, the diversity regularization loss can satisfy the following formula:

[0069] (1.4),

[0070] in, is the diversity regularization loss, is the total number of true samples, is the output of the diversity regularizer, indicating the Noise vector and Noise vector The distance between is the distance threshold.

[0071] in:

[0072] (1.5),

[0073] , are the noise vectors , No. A quantity.

[0074] Formula (1.4) indicates that the diversity regularization loss is generated only when the distance between noise vectors is less than the preset threshold, thus encouraging the noise vectors to maintain at least Adding the diversity regularization loss to the total loss of training the generator can encourage the generator to generate more diverse pseudo samples or noise vectors during training.

[0075] During the training process of the generative adversarial network, the generator can generate highly similar pseudo samples, resulting in insufficient diversity in the augmented dataset. The diversity regularizer imposes constraints on the noise vector to encourage the generator to generate samples with different features, ensuring that the pseudo samples have sufficient diversity in the feature space; it enables the generator to cover a wider range of sample spaces, avoiding the situation where it only generates a few sample categories and ignores other possible samples, resulting in training crashes.

[0076] S205, the ternary loss regularizer calculates the distance between the anchor point sample and its corresponding positive sample and negative sample, and obtains the output result of the current round of training of the ternary loss regularizer.

[0077] In one example, the triplet loss regularizer can select one sample from the real sample and the fake sample as the anchor sample each time, and then select samples with similar categories as positive samples and samples with different categories as negative samples from the real sample and the fake sample. The distance between the anchor sample and the positive sample and the negative sample corresponding to the anchor sample is calculated and output. The triplet loss can be set by these sample distance values. The triplet loss is a loss function used to learn the differences between data points. The samples generated by the generator can be made more realistic by the triplet loss.

[0078] Exemplarily, the triplet loss may satisfy the following formula:

[0079] (1.6),

[0080] in, is the triplet loss, , is the output of the ternary loss regularizer, is the total number of true samples, For the Anchor point samples With Positive samples of anchor samples The distance between For the The anchor sample and Negative samples of anchor samples The distance between is the marginal parameter.

[0081] In a generative adversarial network, the goal of the generator is to deceive the discriminator as much as possible to generate realistic samples. Simply minimizing the loss of the discriminator may cause the generator to create overly simple pseudo samples that cannot fully reproduce the diversity and complexity of real samples. Therefore, the generated samples can be regularized using triplet loss, which aims to minimize the distance between the generated samples and the real samples while maximizing the distance between the generated samples and the noise vector, ensuring that the generated samples are closer to the real samples in the feature space, which helps the generator generate more realistic samples to deceive the discriminator.

[0082] S206, the classification regularizer determines the probability that the pseudo sample of this round of training and the query sample belong to the same category based on the Pearson correlation coefficient, and obtains the output result of the classification regularizer for this round of training.

[0083] Exemplarily, the classification regularizer may be a non-parametric few-shot learning classifier.

[0084] In one example, the classification regularizer may calculate and output the probability that the pseudo sample in the current round of training belongs to the same category as the query sample based on the Pearson correlation coefficient.

[0085] For example, the classification loss can satisfy the following formula:

[0086] (1.7),

[0087] in, is the classification loss, is the total number of true samples, For the A pseudo sample, is the embedding representation of the query sample, is the Pearson correlation coefficient, For the A pseudo sample, is the output of the classification regularizer.

[0088] in:

[0089] (1.8),

[0090] in, Representation vector , The Pearson correlation coefficient, For vector No. Quantity, For vector , Length, For vector No. Quantity, , The vectors are , The mean of .

[0091] For example, , ; Represents a query set Medium, Category Sample query for With Pseudo samples The probability of belonging to the same category.

[0092] S207, determining the discriminator loss of this round of training according to the discrimination result of this round of training.

[0093] In one example, the discrimination result of this round of training can be substituted into the discriminator loss function to obtain the discriminator loss of this round of training.

[0094] For example, the discriminator loss function can be expressed as:

[0095] (1.9),

[0096] in, is the discriminator loss, is the distribution of the real samples, For real samples, Indicates that the real sample obeys distributed, is the category of the real sample, is the context of the real sample, Representation Discriminator The discrimination results of the real samples, the categories of the real samples and the context of the real samples, is a pseudo sample, is the distribution of pseudo samples, Indicates that the pseudo sample obeys distributed, is the category of the pseudo sample, is the context of the pseudo sample, It represents the discriminator's judgment result on the pseudo sample, the category of the pseudo sample and the context of the pseudo sample. is the weight of the gradient penalty term.

[0097] S208, determining the generator loss of this round of training according to the output results of this round of training of the diversity regularizer, the output results of this round of training of the ternary loss regularizer, the output results of this round of training of the classification regularizer, the pseudo samples of this round of training, and the discrimination results of this round of training.

[0098] In one example, the generator loss for this round of training can be obtained based on the generator loss function.

[0099] Exemplarily, the generator loss function can be expressed as:

[0100] (1.10),

[0101] in, is the generator loss, The generator is based on the noise vector The pseudo samples generated, is the distribution of the noise vector, For the generator according to another noise vector The pseudo samples generated, , Represent the noise vectors and the noise vector obey distributed, , , They are triplet loss weight, diversity loss weight, and classification loss weight, respectively. , , They are triplet loss, diversity regularization loss, and classification loss respectively.

[0102] S209, determining whether a preset training end condition is met.

[0103] Exemplarily, the preset training end condition may be that the generator loss and discriminator loss of this round of training are respectively less than the generator loss threshold and the discriminator loss threshold, or the training reaches a preset training number threshold.

[0104] In one example, if the preset training end condition is met, the current network parameters of the generator and the discriminator can be fixed to obtain a trained generative adversarial network. The pseudo sample generated by the generator for the last time is determined as the generated sample and added to the support set to obtain an augmented data set.

[0105] Exemplarily, generated samples of the same category and real samples in the support set may be mixed together to obtain an augmented data set.

[0106] In another example, if the preset training end condition is not met, the network parameters of the generator and the discriminator can be updated according to the generator loss and the discriminator loss of this round of training, respectively, and steps S202-S209 are executed for the next round of training.

[0107] Figure 4The illustrated implementation process of a classification model training method provided by an embodiment of the present invention is as follows. As an example but not a limitation, the classification model training method may include steps S401-S402, which may be applied to the above electronic device, and each step is described below.

[0108] S401, preprocessing the amplified data set to obtain a training set, a test set, and a validation set.

[0109] For details, see Figure 5 , the amplified data set can be preprocessed through steps S4011-S4016.

[0110] S4011, receiving the amplified data set from a memory or a communication interface.

[0111] S4012, converting the image in the amplified data set into a preset format, such as a preset size and a preset color depth, to obtain a sample image in the preset format.

[0112] Exemplarily, the preset size may be 224×224 pixels, and the preset color depth may be RGB format.

[0113] S4013, normalize the image in the preset format and scale the pixel values ​​in the image to between 0 and 1. This helps to eliminate the brightness and contrast differences between different images and improve the stability and convergence speed of model training.

[0114] S4014, performing data enhancement processing on the sample image in the preset format, such as random cropping, rotation, flipping, etc. to obtain an enhanced sample image. The data enhancement processing is used to prevent the classification model from overfitting and improve the generalization ability of the classification model.

[0115] S4015, cropping the format and size of the enhanced sample image into a format suitable for feature extractor input, and then inputting it into the feature extractor to obtain a high-dimensional feature vector of the enhanced sample image.

[0116] Exemplarily, the feature extractor may be the feature extractor 300 described above.

[0117] S4016, these high-dimensional feature vectors are randomly divided into training sets, validation sets and test sets according to a certain proportion, while ensuring the consistency and diversity of sample distribution in each sample set.

[0118] S402: Train the classification model according to the samples in the training set to obtain a trained classification model.

[0119] In one possible implementation, see Figure 6, the extracted high-dimensional feature vector can be input into the classification model for forward propagation calculation to perform classification prediction. The loss of the classification model is calculated based on the prediction results, and then back propagation is performed based on the loss of the classification model to calculate the gradient and update the parameters of the classification model to minimize the loss function. The performance of the classification model is then evaluated using the validation set to verify the classification error and accuracy. The parameters and structure of the model are adjusted according to the validation results to ensure good performance of the model on the validation data. When the stopping condition is met, stop training, otherwise continue training.

[0120] In one example, the loss of the classification model may include a cross entropy loss, which can be used to measure the prediction error of the classification model.

[0121] In one example, an optimization algorithm (eg, an adaptive moment estimation algorithm or a stochastic gradient descent algorithm) may be used to adjust model parameters.

[0122] In one example, similarly, the stopping condition during classification model training may also include loss convergence or reaching a preset number of training times.

[0123] Exemplarily, after training is completed, the classification model can also be tested based on samples in the test set.

[0124] Since the samples in the training set are obtained by amplifying 100 pairs of real samples through the generative adversarial network, they have strong diversity and balanced data distribution. Training the classification model with the training set can improve the generalization ability of the classification model and the classification accuracy.

[0125] Figure 7 The flowchart shown is a method for implementing a small sample learning image classification method provided by an embodiment of the present invention. As an example but not a limitation, the small sample learning image classification method may include steps S701-S702. The method may be applied to the above electronic device, and each step is described below.

[0126] S701, extracting high-dimensional features of the image to be predicted.

[0127] Exemplarily, the image to be predicted may be an image acquired through a memory or received through a communication interface, and may come from a data source such as a sensor, a camera, or a database.

[0128] In one example, the image to be predicted may be input into a feature extractor to extract high-dimensional features of the image to be predicted.

[0129] Exemplarily, the feature extractor may be the feature extractor 300 described above.

[0130] Exemplarily, before the image to be predicted is input into the feature extractor, data preprocessing may be performed on the image to be predicted by a method similar to steps S4011 - S4015 .

[0131] S702, inputting the high-dimensional features of the image to be predicted into the trained classification model to obtain the predicted category of the image to be predicted.

[0132] Exemplarily, the classification model can classify and output a prediction result according to the high-dimensional features of the image to be predicted.

[0133] Specifically, the classification model can match the high-dimensional features of the image to be predicted with the high-dimensional feature vectors of each sample in the amplified data set stored during training, and determine the category of the sample with the highest similarity as the predicted category of the image to be predicted.

[0134] Exemplarily, after the predicted category of the image to be predicted is obtained, it can be output to the user or stored.

[0135] The generative adversarial network provided by the present invention can avoid the problem of unstable training and easy overfitting of the generative adversarial network through three regularizers; and through these three regularizers, the generative adversarial network can generate more diverse samples with certain differences; at the same time, the amount of calculation when amplifying samples is reduced. By obtaining an amplified data set through such a generative adversarial network and training a classification model based on the data set, samples with strong diversity can improve the generalization ability of the classification model, and certain differences between samples can enable the classification model to learn more features of images in different scenes, thereby improving the generalization ability of the model. Therefore, using a classification model trained by an amplified data set to predict image categories can improve classification effect and classification accuracy.

[0136] Figure 8 The structure diagram of a small sample learning image classification device provided by an embodiment of the present invention is shown. As an example but not a limitation, the device 800 may include a processing unit 810.

[0137] The processing unit 810 may be used to: extract high-dimensional features of the image to be predicted; input the high-dimensional features of the image to be predicted into a trained classification model to obtain a predicted category of the image to be predicted;

[0138] The classification model is trained based on the augmented data set, which includes real samples and generated samples. The generated samples are pseudo samples generated by the generative adversarial network based on the real samples and the noise vector for the last time.

[0139] Among them, the generative adversarial network includes a diversity regularizer, a ternary loss regularizer and a classification regularizer. The diversity regularizer is used to keep the distance between noise vectors greater than the distance threshold. The ternary loss regularizer is used to reduce the difference between similar samples and expand the difference between heterogeneous samples. The classification regularizer is used to keep the distribution of generated samples consistent with that of query samples based on the Pearson correlation coefficient. The query samples have the same distribution as the real samples and the same types of images.

[0140] The generative adversarial network provided by the present invention can avoid the problem of unstable training and easy overfitting of the generative adversarial network through three regularizers; and through these three regularizers, the generative adversarial network can generate more diverse samples with certain differences; at the same time, the amount of calculation when amplifying samples is reduced. By obtaining an amplified data set through such a generative adversarial network and training a classification model based on the data set, samples with strong diversity can improve the generalization ability of the classification model, and certain differences between samples can enable the classification model to learn more features of images in different scenes, thereby improving the generalization ability of the model. Therefore, using a classification model trained by an amplified data set to predict image categories can improve the classification effect and classification accuracy of the classification device.

[0141] Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig. 9 The electronic device 900 shown may include: at least one processor 910 ( Fig. 9 Only one processor is shown in the figure), a memory 920, and a computer program 930 stored in the memory 920 and executable on the at least one processor 910, wherein the processor 910 implements the steps of any of the above-mentioned method embodiments when executing the computer program 930.

[0142] The electronic device 900 may be a processing device such as a robot that can implement the above method. The embodiment of the present invention does not impose any limitation on the specific type of the electronic device.

[0143] Those skilled in the art will understand that Fig. 9 This is only an example of the electronic device 900 and does not constitute a limitation on the electronic device. The electronic device 900 may include more or fewer components than shown in the figure, or may combine certain components, or may include different components. For example, the electronic device 900 may also include an input and output interface.

[0144] The processor 910 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASTC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0145] The memory 920 may be an internal storage unit in some embodiments, such as a hard disk or a memory. The memory 920 may also be an external storage device in other embodiments, such as a plug-in hard disk, a smart memory card (SmartMemory Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 920 may also include both an internal storage unit and an external storage device. The memory 920 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 920 may also be used to temporarily store data that has been output or is to be output.

[0146] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0147] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0148] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0149] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A small sample learning image classification method, characterized in that: include: Extract high-dimensional features of the image to be predicted; Inputting the high-dimensional features of the image to be predicted into a trained classification model to obtain a predicted category of the image to be predicted; The classification model is trained according to the amplified data set, the amplified data set includes real samples and generated samples, and the generated samples are pseudo samples generated by the generative adversarial network for the last time according to the real samples and the noise vector; The generative adversarial network includes a diversity regularizer, a ternary loss regularizer and a classification regularizer, the diversity regularizer is used to keep the distance between the noise vectors greater than a distance threshold, the ternary loss regularizer is used to reduce the difference between the same type of samples and expand the difference between different types of samples, and the classification regularizer is used to keep the distribution of the generated samples consistent with the distribution of the query samples based on the Pearson correlation coefficient, and the query samples have the same distribution and the same types of images as the real samples; The generative adversarial network further includes a generator and a discriminator, and the training method of the generative adversarial network includes: Extracting features of the original sample image to obtain the real sample and the query sample; The generator generates a pseudo sample for this round of training according to the real sample and the noise vector; The discriminator evaluates the authenticity of the pseudo samples and the real samples of the current round of training to obtain the discrimination result of the current round of training; The diversity regularizer calculates the distance between the noise vectors to obtain the output result of the current round of training of the diversity regularizer; The ternary loss regularizer determines the distance between the anchor sample and the positive sample corresponding to the anchor sample, and the distance between the anchor sample and the negative sample corresponding to the anchor sample, and obtains the output result of the current round of training of the ternary loss regularizer, wherein the anchor sample is one of the real sample and the pseudo sample; The classification regularizer determines the probability that the pseudo sample of the current round of training and the query sample belong to the same category based on the Pearson correlation coefficient, and obtains the output result of the current round of training of the classification regularizer; Determine the discriminator loss of this round of training according to the discrimination result of this round of training; Determine the generator loss of this round of training according to the output result of this round of training of the diversity regularizer, the output result of this round of training of the ternary loss regularizer, the output result of this round of training of the classification regularizer, the pseudo samples of this round of training, and the discrimination result of this round of training; Determine whether a preset training end condition is met, and if not, update the network parameters of the discriminator and the generator according to the discriminator loss of the current round of training and the generator loss of the current round of training respectively; If satisfied, the network parameters of the generator and the discriminator are fixed to obtain a trained generative adversarial network; Among them, the generator loss satisfies the following formula: ； in, is the generator loss, The generator is based on the noise vector The pseudo samples generated, is the distribution of the noise vector, The generator is based on another noise vector The pseudo samples generated, , Represent the noise vectors and the noise vector obey distributed, , , They are triplet loss weight, diversity loss weight, and classification loss weight, respectively. , , They are triplet loss, diversity regularization loss, and classification loss. Represents the discriminator's response to pseudo samples , the pseudo sample Category And the pseudo sample Context The judgment result of Represents the discriminator's response to pseudo samples , the pseudo sample Category And the pseudo sample Context The discrimination result of express expectations; Wherein, the triplet loss is determined according to the output result of the triplet loss regularizer, the diversity regularization loss is determined according to the output result of the diversity regularizer, and the classification loss is determined according to the output result of the classification regularizer; The diversity regularization loss satisfies the following formula: ； in, is the total number of true samples, is the output result of the diversity regularizer, indicating the Noise vector and Noise vector The distance between is the distance threshold; The classification loss satisfies the following formula: ； in, is the total number of true samples, For the A pseudo sample, is the embedding representation of the query sample, is the Pearson correlation coefficient, For the A pseudo sample, is the output result of the classification regularizer.

2. The method according to claim 1, characterized in that The generative adversarial network is a conditional optimal transmission distance generative adversarial network, and the discriminator loss in the generative adversarial network includes a gradient penalty term, and the gradient penalty term satisfies the following formula: ； in, is the gradient penalty term, Express expectations, is the gradient operator, is an interpolation sample, wherein the interpolation sample is an interpolation between the real sample and the pseudo sample, is the distribution of the interpolation samples, Indicates that the interpolation samples obey distributed, represents the L2 norm, is the discriminator's evaluation of the interpolated sample and the category of the interpolated sample , the context of the interpolated sample The discrimination result of .

3. The method according to claim 2, characterized in that The discriminator loss satisfies the following formula: ； in, is the discriminator loss, is the distribution of the real samples, is the real sample, Indicates that the real sample obeys distributed, is the category of the real sample, is the context of the real sample, represents the discriminator's discriminant result on the real sample, the category of the real sample and the context of the real sample, is the pseudo sample, is the distribution of the pseudo samples, Indicates that the pseudo sample obeys distributed, is the category of the pseudo sample, is the context of the pseudo sample, represents the discrimination result of the discriminator on the pseudo sample, the category of the pseudo sample and the context of the pseudo sample, is the weight of the gradient penalty term.

4. The method according to claim 1, characterized in that The triplet loss satisfies the following formula: ； in, , is the output result of the ternary loss regularizer, is the total number of true samples, For the Anchor point samples With the said Positive samples of anchor samples The distance between For the The anchor samples are related to the Negative samples of anchor samples The distance between is the marginal parameter.

5. A small sample learning image classification device, characterized in that: The method comprises a processing unit, wherein the processing unit is used to extract high-dimensional features of the image to be predicted; input the high-dimensional features of the image to be predicted into a trained classification model to obtain a predicted category of the image to be predicted; The classification model is trained according to the amplified data set, the amplified data set includes real samples and generated samples, and the generated samples are pseudo samples generated by the generative adversarial network for the last time according to the real samples and the noise vector; The generative adversarial network includes a diversity regularizer, a ternary loss regularizer and a classification regularizer, the diversity regularizer is used to keep the distance between the noise vectors greater than a distance threshold, the ternary loss regularizer is used to reduce the difference between the same type of samples and expand the difference between different types of samples, and the classification regularizer is used to keep the distribution of the generated samples consistent with the distribution of the query samples based on the Pearson correlation coefficient, and the query samples have the same distribution and the same types of images as the real samples; The generative adversarial network further includes a generator and a discriminator, and the training method of the generative adversarial network includes: Extracting features of the original sample image to obtain the real sample and the query sample; The generator generates a pseudo sample for this round of training according to the real sample and the noise vector; The discriminator evaluates the authenticity of the pseudo samples and the real samples of the current round of training to obtain the discrimination result of the current round of training; The diversity regularizer calculates the distance between the noise vectors to obtain the output result of the current round of training of the diversity regularizer; The ternary loss regularizer determines the distance between the anchor sample and the positive sample corresponding to the anchor sample, and the distance between the anchor sample and the negative sample corresponding to the anchor sample, and obtains the output result of the current round of training of the ternary loss regularizer, wherein the anchor sample is one of the real sample and the pseudo sample; The classification regularizer determines the probability that the pseudo sample of the current round of training and the query sample belong to the same category based on the Pearson correlation coefficient, and obtains the output result of the current round of training of the classification regularizer; Determine the discriminator loss of this round of training according to the discrimination result of this round of training; Determine the generator loss of this round of training according to the output result of this round of training of the diversity regularizer, the output result of this round of training of the ternary loss regularizer, the output result of this round of training of the classification regularizer, the pseudo samples of this round of training, and the discrimination result of this round of training; Determine whether a preset training end condition is met, and if not, update the network parameters of the discriminator and the generator according to the discriminator loss of the current round of training and the generator loss of the current round of training respectively; If satisfied, the network parameters of the generator and the discriminator are fixed to obtain a trained generative adversarial network; Among them, the generator loss satisfies the following formula: ； in, is the generator loss, The generator is based on the noise vector The pseudo samples generated, is the distribution of the noise vector, The generator is based on another noise vector The pseudo samples generated, , Represent the noise vectors and the noise vector obey distributed, , , They are triplet loss weight, diversity loss weight, and classification loss weight, respectively. , , They are triplet loss, diversity regularization loss, and classification loss. Represents the discriminator's response to pseudo samples , the pseudo sample Category And the pseudo sample Context The judgment result of Represents the discriminator's response to pseudo samples , the pseudo sample Category And the pseudo sample Context The discrimination result of express expectations; Wherein, the triplet loss is determined according to the output result of the triplet loss regularizer, the diversity regularization loss is determined according to the output result of the diversity regularizer, and the classification loss is determined according to the output result of the classification regularizer; The diversity regularization loss satisfies the following formula: ； in, is the total number of true samples, is the output result of the diversity regularizer, indicating the Noise vector and Noise vector The distance between is the distance threshold; The classification loss satisfies the following formula: ； in, is the total number of true samples, For the A pseudo sample, is the embedding representation of the query sample, is the Pearson correlation coefficient, For the A pseudo sample, is the output result of the classification regularizer.

6. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

Images