Image classification method and device

Abstract

The embodiment of the present application provides an image classification method and device, the method comprising: obtaining an object image to be classified, a foreground target extractor and a final classifier; inputting the object image into the foreground target extractor to obtain a foreground target image; inputting the foreground target image into the final classifier to obtain a classification result; the final classifier is composed of an image feature extractor and a small sample object classifier; the image feature extractor is obtained by training with a large sample of foreground target images of known types of objects as training data; the small sample object classifier is obtained by training with an expanded foreground target image of a new type of object as training data; the expansion method comprises: using a posture conversion generator to convert the foreground target image of the new type of object into an expanded foreground target image. The embodiment scheme effectively solves the classification problem of fine-grained images under small sample conditions and effectively improves the classification accuracy.

Description

Technical Field

[0001] This article relates to image recognition technology, and in particular to an image classification method and device. Background Art

[0002] Fine-grained image classification has a wide range of research needs and application scenarios in both industry and academia. Its purpose is to identify different subcategories from a large category, such as distinguishing different birds and different models of cars. It has been a hot research issue in the field of computer vision in recent years. The rapid development of deep learning technology has greatly improved the accuracy of fine-grained image classification. This technology generally requires a large number of labeled samples for model training. However, in many practical application scenarios, such as mechanical fault detection, medical image recognition, deep-sea biological recognition, etc., due to the small amount of data in some categories and the high cost of labeling, there are often few labeled samples, that is, the problem of small sample fine-grained image classification. This problem is more challenging than the fine-grained image classification problem or the ordinary small sample image classification problem, because it inherits the difficulties of these two types of problems: one is the difficulty of large intra-class variance and small inter-class variance in fine-grained image classification; the other is the difficulty of too few samples in small sample image classification to train deep learning models. Summary of the invention

[0003] The embodiments of the present application provide an image classification method and device, which can effectively solve the classification problem of fine-grained images under small sample conditions and effectively improve the classification accuracy.

[0004] The present application provides an image classification method, which may include:

[0005] Obtain an image of an object to be classified and a pre-trained foreground object extractor and a final classifier;

[0006] Inputting the object image to be classified into the foreground target extractor to obtain the foreground target image of the object image to be classified;

[0007] Inputting the foreground target image of the object image to be classified into the final classifier, and the final classifier outputs the classification result of the object image to be classified;

[0008] Wherein, the final classifier is composed of a pre-trained image feature extractor and a small sample object classifier;

[0009] The image feature extractor is obtained by training with a large sample of foreground target images of known types of objects as training data;

[0010] The small sample object classifier is obtained by training with the expanded foreground target image of the new type of object as training data; the expansion method includes: using a posture conversion generator to perform posture conversion on the foreground target image of the new type of object to obtain an expanded foreground target image.

[0011] In an exemplary embodiment of the present application, the foreground object extractor may include: a salient object detection model and an image operation module including a plurality of image operations;

[0012] The image operation may include one or more of the following in sequence: a pixel-level logic operation, a multiplication operation, a mask operation, and an amplification operation;

[0013] The salient object detection model is used to obtain a salient map of an input image of the salient object detection model, wherein the salient map is a feature map containing the posture of a foreground object; and the image operation module is used to perform image processing on the salient map.

[0014] In an exemplary embodiment of the present application, acquiring the foreground object extractor may include: acquiring the salient object detection model;

[0015] The acquiring of the salient target detection model may include: training a constructed convolutional neural network using a first training set consisting of images of the known type of objects and salient maps corresponding to the images to acquire the salient target detection model.

[0016] In an exemplary embodiment of the present application, the method for acquiring the image feature extractor may include:

[0017] Based on a first image of a large sample of a known type of object and the foreground target extractor, a foreground target image of the first image is generated, and the foreground target image is used to train a convolutional neural network to obtain the image feature extractor.

[0018] In an exemplary embodiment of the present application, the method for obtaining the small sample thing classifier may include:

[0019] Generate a new sample of the new type of object based on a first image of a large sample of a known type of object, a second image of a small sample of a new type of object, a pre-trained foreground object extractor, and a pre-trained pose transformation generator;

[0020] Expanding the new sample into the small sample to form an expanded sample set about the new type of thing;

[0021] The small sample thing classifier is obtained by training with the expanded sample set.

[0022] In an exemplary embodiment of the present application, the generating of a new sample about the new type of object based on a first image of a known type of object of a large sample, a second image of a new type of object of a small sample, a pre-trained foreground object extractor, and a pre-trained posture transformation generator may include:

[0023] Setting type labels for the first image set and the second image set respectively, and obtaining the foreground object extractor and the posture conversion generator;

[0024] Inputting the first image set and the second image set with type labels into a pre-trained foreground object extractor respectively, and obtaining a first foreground object image set about the known type of object and a second foreground object image set about the new type of object respectively;

[0025] Inputting the first foreground object image set and the second foreground object image set into the posture conversion generator to convert the posture of the new type of object into the posture of the known type of object; and the posture conversion generator outputs a posture conversion image of the new type of object;

[0026] The acquired multiple posture conversion images are used as new samples of the new type of object.

[0027] In an exemplary embodiment of the present application, the method for acquiring the posture conversion generator may include:

[0028] Extracting a multiple birth training set from a second training set consisting of a large sample of first images of objects of known types;

[0029] The constructed autoencoder neural network is trained using the multiple birth training set to obtain the posture conversion generator.

[0030] In an exemplary embodiment of the present application, the multiple birth training set includes: a quadruplet training set;

[0031] The extracting of a multiple birth training set from a second training set consisting of a large sample of first images of objects of known type may include:

[0032] Acquire multiple quadruplet data pairs {A1, A2, B1, B2} from the second training set to form the quadruplet training set;

[0033] The quadruplet data pair {A1, A2, B1, B2} satisfies: A1 and A2 are from one known type, B1 and B2 are from another known type, the similarity between the saliency map of A1 and the saliency map of B1 is greater than or equal to a preset similarity threshold, the similarity between the saliency map of A2 and the saliency map of B2 is greater than or equal to the similarity threshold; the posture transformation from A1 to A2 is consistent with the posture transformation from B1 to B2.

[0034] In an exemplary embodiment of the present application, the step of training the constructed autoencoder network using the multiple birth training set to obtain the posture conversion generator may include:

[0035] Taking {A1, A2, B1} as input and B2 as the target output of the posture transformation generator, As the actual output of the posture transformation generator;

[0036] According to the target output B2, the actual output The preset loss function is optimized based on the category label y of the target output B2, and the training is determined to be completed when the loss value of the loss function meets the preset requirement, and the trained autoencoder network is used as the posture conversion generator.

[0037] In an exemplary embodiment of the present application, the loss function may include:

[0038] Among them, Loss is the loss value, is the mean square error between the actual output and the target output, For the general The cross entropy loss as the input of the posture transformation generator; λ is a hyperparameter for adjusting the ratio between the mean square error and the cross entropy loss.

[0039] In an exemplary embodiment of the present application, the small sample thing classifier may be a classifier based on cosine distance.

[0040] An embodiment of the present application also provides an image classification device, which may include a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed by the processor, any one of the above-mentioned image classification methods is implemented.

[0041] Compared with the related art, the embodiment of the present application may include: obtaining an object image to be classified and a pre-trained foreground target extractor and a final classifier; inputting the object image to be classified into the foreground target extractor to obtain the foreground target image of the object image to be classified; inputting the foreground target image of the object image to be classified into the final classifier, and the final classifier outputs the classification result of the object image to be classified; wherein the final classifier is composed of a pre-trained image feature extractor and a small sample object classifier; the image feature extractor is obtained by training with a large sample of foreground target images of known types of objects as training data; the small sample object classifier is obtained by training with an expanded foreground target image of a new type of object as training data; the expansion method includes: using a posture conversion generator to perform posture conversion on the foreground target image of the new type of object to obtain an expanded foreground target image.

[0042] Through this embodiment scheme, the classification problem of fine-grained images under small sample conditions is effectively solved, and the classification accuracy is effectively improved.

[0043] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0045] Figure 1 This is a flow chart of the image classification method according to an embodiment of the present application;

[0046] Figure 2 A basic structural diagram of the Baseline++ method of an embodiment of the present application;

[0047] Figure 3 The overall architecture diagram and workflow diagram of the FOT method of the embodiment of the present application;

[0048] Figure 4 A schematic diagram of the process of processing an image by a foreground object extractor according to an embodiment of the present application;

[0049] Figure 5 This is a flow chart of a method for obtaining a small sample thing classifier according to an embodiment of the present application;

[0050] Figure 6 This is a flow chart of a method for acquiring a posture conversion generator according to an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the network structure of the posture conversion generator of the embodiment of the present application;

[0052] Figure 8 This is a schematic diagram of an example of a new type of image generated according to the FOT method in an embodiment of the present application;

[0053] Fig. 9 This is a block diagram of the image classification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0055] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0056] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0057] In the exemplary embodiments of the present application, the following analysis is performed for the two difficulties raised in the background technology:

[0058] Regarding the first difficulty, current fine-grained image classification methods often add additional annotation boxes and local area location information to assist in identifying important areas in the image, or capture similar information through deep learning methods, but this still requires a large number of labeled samples to achieve, and cannot be well applied to the environment of small sample image classification.

[0059] Regarding the second difficulty, the current small sample image classification methods often use image cropping, mirror transformation, appropriate rotation and other methods to solve the problem of too few samples through sample expansion, which is effective in general small sample image classification problems, but the effect is very limited when fine-grained images are used as objects. To explore the root cause of this problem, we carefully observed and found two phenomena: First, images of different categories in fine-grained image classification have similar backgrounds. For example, the living habits of different birds are not very different, and their images often have forests, water, sky and other backgrounds; second, the key features used to distinguish categories are all located on the foreground targets of the image. For example, distinguishing birds often depends on local features of the body such as beaks, colors, and tail feathers. The first phenomenon shows that the image background plays a more negative role in the small sample fine-grained image classification problem, and the second phenomenon shows that the image foreground target plays a more positive role in the small sample fine-grained image classification problem. It can be seen that the mainstream sample expansion method currently used in small sample image classification does not distinguish between the image foreground and background. It is difficult to achieve good results by directly expanding the sample by transforming the entire image, because while expanding the sample, it also expands the negative effect of the background.

[0060] Based on the above analysis, the present application embodiment provides an image classification method, such as Figure 1 As shown, the method may include steps S101-S103:

[0061] S101, obtaining an image of an object to be classified and a pre-trained foreground object extractor and a final classifier;

[0062] S102, inputting the object image to be classified into the foreground target extractor to obtain the foreground target image of the object image to be classified;

[0063] S103, inputting the foreground target image of the object image to be classified into the final classifier, and the final classifier outputs the classification result of the object image to be classified;

[0064] Wherein, the final classifier is composed of a pre-trained image feature extractor and a small sample object classifier;

[0065] The image feature extractor is obtained by training with a large sample of foreground target images of known types of objects as training data;

[0066] The small sample object classifier is obtained by training with the expanded foreground target image of the new type of object as training data; the expansion method includes: using a posture conversion generator to perform posture conversion on the foreground target image of the new type of object to obtain an expanded foreground target image.

[0067] In an exemplary embodiment of the present application, the large sample object classifier refers to a classifier that can only classify objects of known types with large samples. The small sample object classifier refers to a classifier that can classify objects of new types with small samples.

[0068] In an exemplary embodiment of the present application, a foreground target extractor is used to extract a foreground target image of an input object image to be classified, an image feature extractor is used to extract a feature map of the foreground target image, and a small sample object classifier is used to perform type classification based on the input feature map.

[0069] In an exemplary embodiment of the present application, a small sample fine-grained image classification method based on foreground target posture transformation is proposed. The scheme is mainly based on a trained final classification model. When a test picture of a new type of object (or an image of an object to be classified) is subjected to foreground target image extraction by a foreground target extractor, the foreground target image can be used as the input of the final classification model. After image feature extraction of the foreground target image by an image feature extractor and classification of the extracted feature image by a small sample object classifier, the category label corresponding to the test picture can be obtained. In the above manner, the classification problem of fine-grained images under small sample conditions can be effectively solved, and the classification accuracy can be effectively improved.

[0070] In the exemplary embodiment of the present application, the small sample image classification method based on foreground pose transformation (FOT for short) proposed in the present embodiment is based on the small sample image benchmark method Baseline++ as the basic architecture, and adds two new modules, foreground target extractor and pose transformation generator, to form a complete new method. The basic structure of the Baseline++ method is as follows: Figure 2 As shown in the figure, it can be divided into two stages: 1. The training stage (the first stage) uses a sufficient number of samples of known object types to train an image feature extractor and a large sample object classifier; 2. The fine-tuning stage (the second stage) fixes the image feature extractor unchanged, expands the limited number of new types of samples, and uses the expanded samples to train a small sample object classifier.

[0071] In an exemplary embodiment of the present application, the overall architecture of the FOT method proposed in this embodiment can be as follows: Figure 3 As shown, the workflow may include:

[0072] (I) Extract foreground objects. Use the foreground object extractor to process known types and new types, and obtain the enlarged version of the foreground objects after removing the background of all images. The detailed process can be seen as follows: Figure 4 shown.

[0073] (ii) Training on known types. Known types have a large number of labeled samples. Using the known type images processed in the previous step (i.e., foreground target images), we train an image feature extraction network based on a convolutional neural network (CNN) and a classifier for known types.

[0074] (III) Training the posture conversion generator. Construct a multiple-child training set (e.g., a quadruplet data set) that meets the requirements based on the known type, and train the training posture conversion generator generator based on this multiple-child training set.

[0075] (IV) Producing new types of samples. To address the problem of insufficient new types of samples, each new type of sample can be used to find samples with similar saliency maps in the known classes, and the posture transformation of the known sample can be transferred to the new type of sample, thereby achieving the generation of additional new type of samples.

[0076] (V) Training small sample object classifiers. After expanding the generated new type samples into new types, the new type classifier is trained using the expanded new type data set.

[0077] (VI) After the above steps, a complete classifier (i.e., the final classifier) ​​can be obtained. The complete classifier can be composed of an image feature extractor and a small sample object classifier connected in sequence. When a new class test image (i.e., the image of the object to be classified) is used as input, the corresponding category label can be obtained after image feature extraction and new type classification.

[0078] In an exemplary embodiment of the present application, it can be seen from the above steps that the scheme for obtaining the final classifier may include: first designing a foreground target extractor to extract foreground target images from known types and new types of object images, wherein the foreground target extractor is composed of a saliency detection model and several image operations; using the acquired foreground target images of known types for training to obtain a neural network-based image feature extractor; constructing a quadruplet data set (or quadruplet training set), using the quadruplet data set to train the autoencoder neural network structure to obtain a posture transformation generator; using the posture transformation generator to transform the posture of the new type of image to generate additional samples (i.e., new samples about the new type of object); using the expanded data set to train to obtain a classifier for the new type of object image (i.e., the small sample object classifier), and the image feature extractor and the small sample object classifier are connected in sequence to form the final classifier.

[0079] In the exemplary embodiments of the present application, the methods for obtaining the multiple components of the final classifier are respectively described in detail below.

[0080] In an exemplary embodiment of the present application, the foreground object extractor may include: a salient object detection model and an image operation module including a plurality of image operations;

[0081] The image operation may include one or more of the following in sequence: a pixel-level logic operation, a multiplication operation, a mask operation, and an amplification operation;

[0082] The salient object detection model is used to obtain a salient map of an input image of the salient object detection model, wherein the salient map is a feature map containing the posture of a foreground object; and the image operation module is used to perform image processing on the salient map.

[0083] In an exemplary embodiment of the present application, the foreground object extractor may be composed of a pre-trained salient object detection model (BASNet) f, a pixel-level logical operation σ, a multiplication operation A mask operation g and an enlargement operation Composition, such as Figure 4 shown.

[0084] In an exemplary embodiment of the present application, the pre-trained salient object detection model adopts a BASNet network model, and other salient object detection models may also be used.

[0085] In an exemplary embodiment of the present application, acquiring the foreground object extractor may include: acquiring the salient object detection model;

[0086] The acquiring of the salient target detection model may include: training a constructed convolutional neural network using a first training set consisting of images of the known type of objects and salient maps corresponding to the images to acquire the salient target detection model.

[0087] In an exemplary embodiment of the present application, in practical applications, a three-channel RGB (red, green, blue) image X is given as input, and the final foreground target image Y can be obtained through five processing steps:

[0088] (i) Obtain the saliency map f(X) of the input image X using the pre-trained BASNet model;

[0089] (ii) Perform pixel-level logical operations on the saliency map to obtain a single-channel 0-1 logical map σ(f(X)). The execution process of this operation is to give a threshold γ. When the mean of the three channels of a certain pixel is greater than γ, the pixel value is set to 1, otherwise it is set to 0.

[0090] (III) Multiply the single-channel 0-1 logic map with the original image to convert the input image into a black background image

[0091] (iv) Use mask operation to crop the image to obtain only the area containing the foreground object (such as a rectangular area) The size and position of the mask box can be automatically read from the logic diagram in the previous step;

[0092] (V) Using the zoom operation, the area containing the foreground target is enlarged to a uniform size as the input of the subsequent deep learning model

[0093] In an exemplary embodiment of the present application, based on the above steps, the input-output relationship can be expressed as:

[0094] In an exemplary embodiment of the present application, the method for acquiring the image feature extractor may include:

[0095] Based on a first image of a large sample of a known type of object and the foreground target extractor, a foreground target image of the first image is generated, and the foreground target image is used to train a convolutional neural network to obtain the image feature extractor.

[0096] In an exemplary embodiment of the present application, the structure of the image feature extractor is generally a convolutional neural network (CNN), and its training data set uses foreground target images of known types.

[0097] In an exemplary embodiment of the present application, Figure 5As shown, the method for obtaining the small sample thing classifier may include steps S201-S203:

[0098] S201, based on a first image of a known type of object with a large sample, a second image of a new type of object with a small sample, a pre-trained foreground object extractor and a pre-trained posture transformation generator, generate a new sample about the new type of object.

[0099] In an exemplary embodiment of the present application, Figure 6 As shown, the method for acquiring the posture conversion generator may include steps S301-S302:

[0100] S301, extracting a multiple-birth training set from a second training set consisting of a large sample of first images of objects of known types.

[0101] In an exemplary embodiment of the present application, the multiple birth training set includes: a quadruplet training set;

[0102] The extracting of a multiple birth training set from a second training set consisting of a large sample of first images of objects of known type may include:

[0103] Acquire multiple quadruplet data pairs {A1, A2, B1, B2} from the second training set to form the quadruplet training set;

[0104] The quadruplet data pair {A1, A2, B1, B2} satisfies: A1 and A2 are from one known type, B1 and B2 are from another known type, the similarity between the saliency map of A1 and the saliency map of B1 is greater than or equal to a preset similarity threshold, the similarity between the saliency map of A2 and the saliency map of B2 is greater than or equal to the similarity threshold; the posture transformation from A1 to A2 is consistent with the posture transformation from B1 to B2.

[0105] In an exemplary embodiment of the present application, the posture conversion generator is an auto-encoder neural network, and its architecture is as follows: Figure 7As shown in Figure 1, it is a deep learning model. In addition to the network structure itself, the focus is on the training of the model. Its training data set uses a multiple training set (such as a quadruplet training set) constructed from known types. To this end, a multiple training set construction algorithm based on saliency maps is proposed on the basis of known types (base classes). That is, the training set of the deep learning model consists of a large number of quadruplet data pairs {A1, A2, B1, B2}, where A1 and A2 are from one known type, B1 and B2 are from another known type, and the saliency map of A1 is similar to that of B1, and the saliency map of A2 is similar to that of B2. Since the saliency map can represent the posture of the foreground target, the posture transformation from A1 to A2 is consistent with the posture transformation from B1 to B2; the formal description is as follows:

[0106] find all{A1,A2,B1,B2},

[0107]

[0108] in, Represents two different known types, They represent the saliency maps of A1, A2, B1, and B2 respectively. Distance() uses Euclidean distance, and α and β represent two thresholds for measuring the size of the Euclidean distance.

[0109] S302: Use the multiple birth training set to train the constructed autoencoder neural network to obtain the posture conversion generator.

[0110] In an exemplary embodiment of the present application, the step of training the constructed autoencoder network using the multiple birth training set to obtain the posture conversion generator may include:

[0111] Taking {A1, A2, B1} as input and B2 as the target output of the posture transformation generator, As the actual output of the posture transformation generator;

[0112] According to the target output B2, the actual output The preset loss function is optimized based on the category label y of the target output B2, and the training is determined to be completed when the loss value of the loss function meets the preset requirement, and the trained autoencoder network is used as the posture conversion generator.

[0113] In an exemplary embodiment of the present application, the loss function may include:

[0114] Among them, Loss is the loss value, is the mean square error between the actual output and the target output, For the general The cross entropy loss as the input of the posture transformation generator; λ is a hyperparameter for adjusting the ratio between the mean square error and the cross entropy loss.

[0115] In the exemplary embodiment of the present application, after data training, the posture transformation generator can learn how to transfer the posture transformation from A1 to A2 to B1, thereby learning to obtain a posture transformation similar to B2.

[0116] In an exemplary embodiment of the present application, the generating of a new sample about the new type of object based on a first image of a known type of object of a large sample, a second image of a new type of object of a small sample, a pre-trained foreground object extractor, and a pre-trained posture transformation generator may include:

[0117] Setting type labels for the first image set and the second image set respectively, and obtaining the foreground object extractor and the posture conversion generator;

[0118] Inputting the first image set and the second image set with type labels into a pre-trained foreground object extractor respectively, and obtaining a first foreground object image set about the known type of object and a second foreground object image set about the new type of object respectively;

[0119] Inputting the first foreground object image set and the second foreground object image set into the posture conversion generator to convert the posture of the new type of object into the posture of the known type of object; and the posture conversion generator outputs a posture conversion image of the new type of object;

[0120] The acquired multiple posture conversion images are used as new samples of the new type of object.

[0121] In an exemplary embodiment of the present application, the following steps can be followed when applying the posture conversion generator to perform posture conversion on new class samples: for each training sample Z1 in the new type, find a sample X1 with a similar saliency map from the known type, combine it with other samples X2 of the type where the sample X1 belongs, and combine them together [X1, X2, Z1] as the input of the posture conversion generator, so as to obtain a new sample sequence Z1 after the morphological transformation Repeat this process to get enough new class samples.

[0122] S202: Expand the new sample into the small sample to form an expanded sample set about the new type of thing.

[0123] S203: Using the expanded sample set for training to obtain the small sample object classifier.

[0124] In an exemplary embodiment of the present application, the small sample thing classifier may be a classifier based on cosine distance, or other types of classifiers may be used.

[0125] In the exemplary embodiments of the present application, the detailed implementation of the embodiment scheme of the present application is further described below around an implementation case. Obviously, the case described in this section is only an implementation case of the present invention, not all implementation cases. Based on the implementation case of the present application, all other implementation cases obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0126] In an exemplary embodiment of the present application, taking the fine-grained image classification benchmark datasets Cub birds, Stanford Dogs, and Stanford Cars as examples, two small sample fine-grained classification strategies are mainly adopted. One is 5-way 1-shot, that is, for 5 new classes (i.e., new types), there is only 1 image for each type in the training set; the other is 5-way 5-shot, that is, for 5 new classes, there are only 5 images for each class in the training set. During testing, regardless of 5-way 1-shot or 5-way 5-shot, 15 images are selected as test images for each class.

[0127] In the exemplary embodiment of the present application, in order to illustrate the effectiveness of the method of the present embodiment, comparative experiments were conducted on three benchmark datasets with two benchmark methods, Baseline and Baseline++, four mainstream small sample image classification methods, MatchingNet, ProtoNet, MAML, and RelationNet, and four state-of-the-art fine-grained small sample image classification methods, PCM, PABN, CovaMNet, and DN4, and the results are shown in Table 1. Table 1 compares the experimental results of different methods on three fine-grained image benchmark datasets, where the dark gray area represents the best result in each column, and the light gray area represents the second best result in each column.

[0128] Table 1

[0129]

[0130]

[0131] In the exemplary embodiments of the present application, it can be seen from Table 1 that, based on the Baseline++ method, the method (FOT) proposed in the embodiment of the present application has an average improvement of 7.20% and 5.99% on 5-way 1-shot and 5-way 5-shot; it has significantly surpassed the current mainstream small sample image classification method; compared with the current most advanced fine-grained small sample image classification method, it has achieved the highest results on the Cub birds and Stanford Dogs data sets, and is second only to the DN4 method on Stanford Cars, showing high classification performance and generalization performance.

[0132] In an exemplary embodiment of the present application, as a sample expansion method, the FOT method proposed in the embodiment of the present application can also be combined with the current small sample image classification method as an auxiliary method to improve the performance of the current small sample image classification method. Table 2 shows the experimental results of combining the FOT method with the four current mainstream small sample image classification methods.

[0133] Table 2

[0134]

[0135]

[0136] In an exemplary embodiment of the present application, as a data expansion method, the FOT method proposed in the embodiment of the present application can be conveniently applied as an auxiliary module to any existing small sample image classification method. As can be seen from Table 2, the FOT module can effectively improve the performance of the mainstream small sample image classification method. In fact, all the methods listed in Table 1 can further improve the performance through the FOT module.

[0137] In the exemplary embodiments of the present application, in order to further describe the execution effect of the method of the embodiment of the present application in a more vivid way, as shown in FIG. Figure 8 As shown in the figure, some examples of new types of images generated by the FOT method proposed in the embodiment of the present application are shown, and each column represents a different quadruplet data pair Z1 represents an image of a new type, X1 is an image of a known type that is similar to its saliency map, and X2 is another image of the same known type as X1. is the actual generated image. It can be seen that In terms of category, it has the characteristics of Z1 (such as the color of the bird, the characteristics of key parts, etc.), and in terms of posture, it has the characteristics of X2.

[0138] The present application also provides an image classification device 1, such as Fig. 9As shown, it may include a processor 11 and a computer-readable storage medium 12, wherein the computer-readable storage medium 12 stores instructions, and when the instructions are executed by the processor 11, any one of the above-mentioned image classification methods is implemented.

[0139] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. An image classification method, characterized in that: The method comprises: Obtain an image of an object to be classified and a pre-trained foreground object extractor and a final classifier; Inputting the object image to be classified into the foreground target extractor to obtain the foreground target image of the object image to be classified; Inputting the foreground target image of the object image to be classified into the final classifier, and the final classifier outputs the classification result of the object image to be classified; Wherein, the final classifier is composed of a pre-trained image feature extractor and a small sample object classifier; The image feature extractor is obtained by training with a large sample of foreground target images of known types of objects as training data; The small sample object classifier is obtained by training with the expanded foreground target image of the new type of object as training data; the expansion method includes: using a posture conversion generator to perform posture conversion on the foreground target image of the new type of object to obtain an expanded foreground target image; The method for obtaining the small sample thing classifier includes: Generate a new sample of the new type of object based on a first image of a large sample of a known type of object, a second image of a small sample of a new type of object, a pre-trained foreground object extractor, and a pre-trained pose transformation generator; Expanding the new sample into the small sample to form an expanded sample set about the new type of thing; The expanded sample set is used for training to obtain the small sample thing classifier; The method of generating a new sample of the new type of object based on a first image of a known type of object with a large sample, a second image of a new type of object with a small sample, a pre-trained foreground object extractor and a pre-trained posture transformation generator comprises: Setting type labels for the first image and the second image respectively, and obtaining the foreground object extractor and the posture conversion generator; Inputting the first image and the second image with type labels into a pre-trained foreground object extractor respectively, and obtaining a first foreground object image set about the known type of object and a second foreground object image set about the new type of object respectively; Inputting the first foreground object image set and the second foreground object image set into the posture conversion generator to convert the posture of the new type of object into the posture of the known type of object; and the posture conversion generator outputs a posture conversion image of the new type of object; Using the acquired multiple posture conversion images as new samples of the new type of things; The method for acquiring the posture conversion generator includes: Extracting a multiple birth training set from a second training set consisting of a large sample of first images of objects of known types; The constructed autoencoder neural network is trained using the multiple birth training set to obtain the posture conversion generator; The multiple birth training set includes: a quadruplet training set; The extracting of a multiple birth training set from a second training set consisting of a large sample of first images of objects of known type comprises: Obtain multiple quadruplet data pairs from the second training set , constituting the quadruplet training set; Among them, the quadruplet data pair satisfy: From a known type, From another known type, The salient graph and The similarity of the saliency maps is greater than or equal to the preset similarity threshold. The salient graph and The similarity of the saliency maps of is greater than or equal to the similarity threshold; arrive The posture change from arrive The posture changes are consistent; The step of training the constructed autoencoder network using the multiple birth training set to obtain the posture conversion generator includes: Will As input, As the target output of the posture transformation generator, As the actual output of the posture transformation generator; Output according to the target , the actual output and the target output The preset loss function is optimized based on the category label y, and the training is determined to be finished when the loss value of the loss function meets the preset requirement, and the trained autoencoder network is used as the posture conversion generator.

2. The image classification method according to claim 1, characterized in that: The foreground object extractor includes: a salient object detection model and an image operation module including multiple image operations; The image operation includes one or more of the following in sequence: pixel-level logic operation, multiplication operation, mask operation and amplification operation; The salient object detection model is used to obtain a salient map of an input image of the salient object detection model, wherein the salient map is a feature map containing the posture of a foreground object; and the image operation module is used to perform image processing on the salient map.

3. The image classification method according to claim 2, characterized in that: Acquiring the foreground object extractor includes: acquiring the salient object detection model; The obtaining of the salient target detection model comprises: training a constructed convolutional neural network using a first training set consisting of an image of the known type of object and a salient map corresponding to the image to obtain the salient target detection model.

4. The image classification method according to claim 1, characterized in that: The method for obtaining the image feature extractor comprises: Based on a first image of a large sample of a known type of object and the foreground target extractor, a foreground target image of the first image is generated, and the foreground target image is used to train a convolutional neural network to obtain the image feature extractor.

5. The image classification method according to claim 1, characterized in that: The loss function includes: ； in, is the loss value, is the mean square error between the actual output and the target output, For the general Cross entropy loss as input to the pose transformation generator; is a hyperparameter that adjusts the ratio between the mean square error and the cross entropy loss.

6. The image classification method according to any one of claims 1 to 4, characterized in that: The small sample thing classifier is a classifier based on cosine distance.

7. An image classification device, comprising a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, characterized in that: When the instructions are executed by the processor, the image classification method according to any one of claims 1 to 4 is implemented.

Images