Training method and device of classification model, electronic equipment and storage medium

Abstract

The application provides a classification model training method and device, electronic equipment and a computer readable storage medium. The classification model training method comprises: obtaining a plurality of sample images; performing feature extraction processing according to the sample images and sample distribution labels to obtain basic feature information and augmented feature information of the sample images; performing classification prediction processing according to the basic feature information and the augmented feature information to obtain first classification results and second classification results of the sample images; updating model parameters of a preset model according to the first classification results, the second classification results and the category labels until the preset model converges, and taking the preset model as a trained classification model. In the application, the classification accuracy of the classification model for a few-sample category can be improved, and the overall classification accuracy of the classification model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a classification model training method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] In recent years, due to the rapid development of artificial intelligence, computer vision-based image classification processing has been widely applied. In order to improve the accuracy of image classification, existing image classification models need to rely on a large number of sample images for training.

[0003] However, in reality, there may be cases where samples are few or difficult to obtain, which will result in uneven sample distribution during training of the classification model (i.e., the sample data set is a data long-tail distribution), which is specifically manifested as: some classes have a large number of sample images, and some classes have a small number of sample images. Since the classes with a small number of samples cannot be generalized, the classification accuracy of the trained classification model will be low when classifying images of classes with a small number of samples, thereby reducing the classification accuracy of the classification model. SUMMARY

[0004] The present application provides a classification model training method, device, electronic device, and computer readable storage medium, aiming to solve the problem of reduced training accuracy of the classification model due to the data long-tail distribution of the sample data set.

[0005] In a first aspect, the present application provides a classification model training method, which comprises:

[0006] obtaining a plurality of sample images, wherein the sample images have been labeled with a class label and a sample distribution label, and the sample distribution label is used to indicate whether the sample image is head data or tail data;

[0007] performing feature extraction processing according to the sample images and the sample distribution label to obtain basic feature information and augmented feature information of the sample images;

[0008] performing classification prediction processing according to the basic feature information and the augmented feature information to obtain a first classification result and a second classification result of the sample images;

[0009] updating model parameters of a preset model according to the first classification result, the second classification result, and the class label until the preset model converges, and then taking the preset model as a trained classification model, wherein the classification model is used for classifying images.

[0010] In a possible implementation of the present application, the feature extraction processing is performed according to the sample image and the sample distribution label, to obtain basic feature information and augmented feature information of the sample image, and the feature extraction processing includes:

[0011] The feature extraction processing is performed according to the sample image, to obtain image feature information of the sample image;

[0012] The first mean information and the first variance information of the image feature information are obtained;

[0013] The basic feature information and the augmented feature information are determined according to the first mean information, the first variance information, and the sample distribution label.

[0014] In a possible implementation of the present application, the basic feature information and the augmented feature information are determined according to the first mean information, the first variance information, and the sample distribution label, and the determination includes:

[0015] The first mean information and the first variance information are taken as the basic feature information;

[0016] The image feature information is classified according to the sample distribution label, to obtain tail image feature information and head image feature information of the sample image;

[0017] The first variance information corresponding to the head image feature information is taken as second variance information of the tail image feature information; and the first mean information corresponding to the tail image feature information is taken as second mean information of the tail image feature information;

[0018] The second mean information and the second variance information are taken as the augmented feature information.

[0019] In a possible implementation of the present application, the classification prediction processing is performed according to the basic feature information and the augmented feature information, to obtain a first classification result and a second classification result of the sample image, and the classification prediction processing includes:

[0020] The sample image is classified and predicted according to the first mean information and the first variance information, to obtain the first classification result;

[0021] The sample image is classified and predicted according to the second mean information and the second variance information, to obtain the second classification result.

[0022] In a possible implementation of the present application, the first variance information corresponding to the head image feature information is taken as the second variance information of the tail image feature information, and the taking includes:

[0023] The second variance information is determined based on the first variance information corresponding to the head image feature information and a preset buffer formula, wherein the buffer formula is:

[0024] δ' tail =αδ tail +(1-α)δ head

[0025] Where, δ' tail This represents the second variance information, δ tail δ represents the first variance information corresponding to the tail image feature information. head The first variance information corresponding to the head image feature information is represented by α, where α is a hyperparameter.

[0026] In one possible implementation of this application, the step of updating the model parameters of a preset model based on the first classification result, the second classification result, and the category label until the preset model converges, and then using the preset model as a trained classification model, includes:

[0027] Based on the first classification result and the category label, determine the first classification loss value of the preset model;

[0028] Based on the second classification result and the category label, determine the second classification loss value of the preset model;

[0029] The total loss value of the preset model is determined based on the first classification loss value and the second classification loss value;

[0030] The model parameters are updated based on the total loss value until the preset model converges, at which point the preset model is used as the trained classification model.

[0031] In one possible implementation of this application, the method further includes:

[0032] Acquire the target image;

[0033] The classification model is invoked to perform feature extraction processing on the target image, thereby obtaining the target feature information of the target image;

[0034] The classification model is invoked to perform classification prediction processing based on the target feature information to obtain the target category of the target image.

[0035] Secondly, this application provides a training apparatus for a classification model, the training apparatus for the classification model comprising:

[0036] The acquisition unit is configured to acquire a plurality of sample images, wherein the sample images are labeled with a category label and a sample distribution label, and the sample distribution label is used to indicate whether the sample image is head data or tail data.

[0037] The feature extraction unit is configured to perform feature extraction processing on the sample images and the sample distribution label to obtain basic feature information and augmented feature information of the sample images.

[0038] The prediction unit is configured to perform classification prediction processing on the basic feature information and the augmented feature information to obtain a first classification result and a second classification result of the sample images.

[0039] The update unit is configured to update model parameters of a preset model according to the first classification result, the second classification result, and the category label, and when the preset model converges, the preset model is used as a trained classification model, wherein the classification model is used to classify images.

[0040] In a possible implementation of the present application, the feature extraction unit is specifically configured to:

[0041] perform feature extraction processing on the sample images to obtain image feature information of the sample images;

[0042] obtain first mean information and first variance information of the image feature information;

[0043] determine the basic feature information and the augmented feature information according to the first mean information, the first variance information, and the sample distribution label.

[0044] In a possible implementation of the present application, the feature extraction unit is specifically configured to:

[0045] use the first mean information and the first variance information as the basic feature information;

[0046] perform classification processing on the image feature information according to the sample distribution label to obtain tail image feature information and head image feature information of the sample images;

[0047] use first variance information corresponding to the head image feature information as second variance information of the tail image feature information, and use first mean information corresponding to the tail image feature information as second mean information of the tail image feature information;

[0048] use the second mean information and the second variance information as the augmented feature information.

[0049] In a possible implementation of the present application, the prediction unit is specifically further configured to:

[0050] perform classification prediction processing on the sample image according to the first mean information and the first variance information, to obtain the first classification result;

[0051] perform classification prediction processing on the sample image according to the second mean information and the second variance information, to obtain the second classification result.

[0052] In a possible implementation of the present application, the feature extraction unit is specifically further configured to:

[0053] determine the second variance information according to first variance information corresponding to the head image feature information and a preset buffer formula, where the buffer formula is:

[0054] δ' tail = αδ tail + (1-α)δ head

[0055] wherein δ' tail represents the second variance information, δ tail represents first variance information corresponding to the tail image feature information, and δ head represents first variance information corresponding to the head image feature information, and α is a hyperparameter.

[0056] In a possible implementation of the present application, the updating unit is specifically further configured to:

[0057] determine a first classification loss value of the preset model according to the first classification result and the category label;

[0058] determine a second classification loss value of the preset model according to the second classification result and the category label;

[0059] determine a total loss value of the preset model according to the first classification loss value and the second classification loss value;

[0060] update the model parameters according to the total loss value, and when the preset model converges, take the preset model as a trained classification model.

[0061] In a possible implementation of the present application, the training apparatus of the classification model further includes a training unit, and the training unit is specifically configured to:

[0062] obtain a target image;

[0063] invoke the classification model to perform feature extraction processing on the target image, to obtain target feature information of the target image.

[0064] calling the classification model, performing classification prediction processing according to the target feature information, to obtain a target category of the target image.

[0065] In a third aspect, the present application also provides an electronic device, which comprises a processor and a memory, and the memory stores a computer program, and the processor executes the steps of the training method of any one of the classification models provided by the present application when calling the computer program in the memory.

[0066] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is loaded by a processor to execute the steps of the training method of the classification model.

[0067] According to the feature extraction processing of the plurality of sample images labeled with the sample distribution label for indicating whether the sample image is head data or tail data, the basic feature information and the augmented feature information of the sample image are obtained; according to the basic feature information and the augmented feature information, the classification prediction processing is performed to obtain the first classification result and the second classification result of the sample image; according to the first classification result, the second classification result and the category label, the model parameters of the preset model are updated until the preset model converges, and the preset model is taken as the trained classification model; on the one hand, since the feature space of the tail data in the sample image can be augmented by the sample distribution label, the augmented feature information is obtained and used for model training, so that the classification generalization of the trained classification model on the tail sample data (i.e. few samples) category is improved, thereby improving the classification accuracy of the classification model on the few sample category, and further improving the accuracy of the entire classification model. On the other hand, since the feature augmentation is performed at the feature space level, it is not necessary to rely on the number of sample images. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0069] Figure 1 is a scene schematic diagram of the training system of the classification model provided by the embodiments of the present application;

[0070] Figure 2 is a flowchart of the training method of the classification model provided by the embodiments of the present application;

[0071] Figure 3 is a framework schematic diagram of the preset model provided by the embodiments of the present application;

[0072] Figure 4 is another framework schematic diagram of the preset model provided in the embodiments of the present application;

[0073] Figure 5 is an embodiment flow schematic diagram of step S20 provided in the embodiments of the present application;

[0074] Figure 6 is a schematic diagram of the feature augmentation process provided in the embodiments of the present application;

[0075] Figure 7 is an embodiment structure schematic diagram of the training device of the classification model provided in the embodiments of the present application;

[0076] Figure 8 is an embodiment structure schematic diagram of the electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0078] In the description of the embodiments of the present application, it should be understood that the terms “first”, “second” are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of “multiple” is two or more than two, unless otherwise specifically limited.

[0079] The following description is presented to enable any person skilled in the art to practice and use the present application. In the following description, for the purpose of explanation, details are set forth. It will be apparent to those skilled in the art from this disclosure that the application can be practiced without the specific details presented. In other instances, well-known processes have not been described in detail in order to avoid obscuring the description of the embodiments of the present application. Therefore, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0080] The embodiment of the present application provides a classification model training method and device, electronic equipment and a computer readable storage medium. The classification model training device can be integrated in the electronic equipment, which can be a server or a terminal or the like.

[0081] First, before introducing the embodiment of the present application, the related content of the application background of the embodiment of the present application is introduced.

[0082] Computer vision technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics, so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, and tries to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and map building, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0083] Deep neural networks have made great progress in target classification and recognition tasks, but they perform poorly when dealing with long-tail distributed data, specifically, the generalization performance of classification on classes with only a small number of training samples is poor.

[0084] The existing technology solves the problem of long-tail data in three ways: resampling, sample generation and weight allocation. Resampling is to repeatedly sample classes with only a small number of training samples, sample generation is to generate a batch of additional samples for the few-sample class through a generative adversarial network, and resampling and sample generation both increase the number of samples of the few-sample class, balancing the number of classes; Weight allocation considers different training numbers of classes, usually sets appropriate weights on the training loss function, and improves the attention of the classifier to the few-sample class. The methods based on resampling and weight allocation are limited by the existing few-sample, and the learned classifier has limited performance. The generative adversarial network based method effectively improves the performance of the classifier, but the training samples required by this method are too high.

[0085] Based on the above-mentioned defects of the existing related technology, the embodiment of the present application provides a classification model training method, which at least partially overcomes the defects of the existing related technology.

[0086] The execution subject of the training method of the classification model provided in the embodiments of the present application can be a training device of a classification model provided in the embodiments of the present application, or a server device, a physical host, or a user equipment (UE) or other types of electronic devices integrated with the training device of the classification model. The training device of the classification model can be implemented in a hardware or software manner, and the UE can be a terminal device such as a smartphone, a tablet computer, a notebook computer, a palm computer, a desktop computer, or a personal digital assistant (PDA).

[0087] The electronic device can operate in a standalone manner or in a device cluster manner. By applying the training method of the classification model provided in the embodiments of the present application, the classification accuracy of the classification model for a few-sample class can be improved, and thus the overall classification accuracy of the classification model can be improved.

[0088] Referring to Figure 1 , Figure 1 is a scene schematic diagram of a training system of a classification model provided in the embodiments of the present application. The training system of the classification model can include an electronic device 100 integrated with a training device of a classification model. For example, the electronic device can obtain a plurality of sample images; perform feature extraction processing according to the sample images and sample distribution labels to obtain basic feature information and augmented feature information of the sample images; perform classification prediction processing according to the basic feature information and the augmented feature information to obtain first classification results and second classification results of the sample images; update model parameters of a preset model according to the first classification results, the second classification results, and the class labels until the preset model converges, and then use the preset model as a trained classification model.

[0089] In addition, as Figure 1 indicated, the training system of the classification model can also include a memory 200 for storing data such as image data and video data.

[0090] It should be noted that Figure 1 the scene schematic diagram of the training system of the classification model shown is only an example, and the training system of the classification model and the scene described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not limit the technical solutions provided in the embodiments of the present application. It is known to those skilled in the art that, as the training system of the classification model evolves and new business scenarios appear, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0091] Next, the training method of the classification model provided by the embodiments of the present application will be introduced. In the embodiments of the present application, an electronic device is taken as the execution subject. In order to simplify and facilitate the description, the execution subject will be omitted in the subsequent method embodiments. The training method of the classification model comprises: obtaining a plurality of sample images; performing feature extraction processing according to the sample images and the sample distribution labels to obtain basic feature information and augmented feature information of the sample images; performing classification prediction processing according to the basic feature information and the augmented feature information to obtain first classification results and second classification results of the sample images; updating model parameters of a preset model according to the first classification results, the second classification results and the category labels until the preset model converges, and taking the preset model as a trained classification model.

[0092] Referring to Figure 2 , Figure 2 is a flowchart of the training method of the classification model provided by the embodiments of the present application. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described herein can be performed in an order different from that shown herein. The training method of the classification model comprises steps S10-S40, wherein:

[0093] S10, obtaining a plurality of sample images.

[0094] The sample images have been labeled with category labels and sample distribution labels. The sample distribution labels are used to indicate whether the sample images are head data or tail data.

[0095] The head data (also referred to as head class or head class data herein) refers to sample images with a larger number of samples corresponding to a category. The tail data (also referred to as tail class or tail class data herein) refers to sample images with a smaller number of samples corresponding to a category.

[0096] The objects contained in the sample images and the categories of the objects can be set according to specific requirements, which are not limited in the embodiments of the present application.

[0097] For example, in order to enable the model to classify vehicles into a brand-new category, a slight scratch category and a serious damage category according to images, images of vehicles in brand-new, slight scratch and serious damage (such as damage caused by serious accidents) states need to be collected as sample images for training the model. Among them, images of vehicles in brand-new and slight scratch states are relatively easy to collect, that is, the number of samples in the brand-new category is relatively large, and the number of samples in the slight scratch category is also relatively large. However, images of vehicles in a serious damage state are relatively difficult to collect in actual situations, that is, the number of samples in the serious damage category is relatively small. Therefore, the sample distribution labels of the sample images in the brand-new category, the slight scratch category and the serious damage category can be labeled as head data, head data and tail data, respectively.

[0098] S20, performing feature extraction processing according to the sample image and the sample distribution label, to obtain basic feature information and augmented feature information of the sample image.

[0099] The basic feature information refers to image features obtained after feature extraction of the sample image. The augmented feature information refers to features obtained after augmentation processing of the basic feature information according to the sample distribution label.

[0100] In order to better understand the embodiments of the present application, the training concept of the classification model in the embodiments of the present application and the network structure and function of the preset model involved will be introduced first.

[0101] In the first aspect, the training concept of the classification model.

[0102] The embodiments of the present application do not increase the richness of the sample image itself, but aims to augment the image features of the tail sample data, and train the model using the augmented image features of the tail sample data, so as to realize the classification generalization of the classification model on the tail sample data (i.e. few samples) category, thereby improving the classification accuracy of the classification model on the few sample category, and further improving the accuracy of the entire classification model.

[0103] In the second aspect, the network structure and function of the preset model.

[0104] The preset model can include a feature extraction layer, a feature augmentation layer, and a classification prediction layer.

[0105] The feature extraction layer is configured to output basic feature information of an image. The feature extraction layer takes an image (such as a single RGB picture) as input, performs feature extraction processing on the image, and outputs an image feature f with a dimension of D.

[0106] The feature augmentation layer is configured to output augmented image features f' according to image features f. The feature augmentation layer takes image features f as input, performs feature augmentation processing on image features f, and outputs augmented image features f'.

[0107] The classification prediction layer is configured to output a first classification result of an image according to image features f, and a second classification result of the image according to augmented image features f'.

[0108] Please refer to Figure 3 , Figure 3 is a framework schematic diagram of the preset model provided in the embodiments of the present application. In some embodiments, the network framework of the preset model can include, for example Figure 3The two branches (regular branch, augmented branch) are shown. The regular branch can sequentially include a first feature extraction layer and a first classification prediction layer. The first feature extraction layer is configured to perform feature extraction processing on the sample image to output basic feature information of the sample image. The first classification prediction layer is configured to perform classification prediction processing on the basic feature information of the sample image to output a first classification result of the sample image. The augmented branch can sequentially include a second feature extraction layer, a feature augmentation layer, and a second classification prediction layer. The second feature extraction layer is configured to perform feature extraction processing on the sample image to output basic feature information of the sample image. The feature augmentation layer is configured to perform feature augmentation processing on the basic feature information of the sample image to output augmented feature information of the sample image. The second classification prediction layer is configured to perform classification prediction processing on the basic feature information or the augmented feature information of the sample image to output a second classification result of the sample image.

[0109] In some embodiments, the first feature extraction layer and the second feature extraction layer can have the same structure, and the weight parameters of the two can be shared to ensure that the basic feature information output by the two is the same. Figure 3

[0110] Please refer to Figure 4 , Figure 4 is another framework diagram of the preset model provided in the embodiments of the present application. In some embodiments, the network framework of the preset model can include a feature extraction layer and two branches (regular branch, augmented branch) as shown in Figure 4 The regular branch can sequentially include a first classification prediction layer. The first classification prediction layer is configured to perform classification prediction processing on the basic feature information of the sample image to output a first classification result of the sample image. The augmented branch can sequentially include a feature augmentation layer and a second classification prediction layer. The feature augmentation layer is configured to perform feature augmentation processing on the basic feature information of the sample image to output augmented feature information of the sample image. The second classification prediction layer is configured to perform classification prediction processing on the basic feature information or the augmented feature information of the sample image to output a second classification result of the sample image.

[0111] Figure 3 The regular branch shown in Figure 4 is configured to output the first classification result, and the augmented branch is configured to output the second classification result.

[0112] ​It can be understood that the above introduction of the network structure and functions of the preset model is only for better understanding of the embodiments of the present application. Those skilled in the art can adjust the network structure of the preset model on the basis of the above network structure examples according to the functions required to be implemented by the preset model. For example, the feature extraction layer outputs preliminary image features, and a feature preprocessing layer is newly added after the feature extraction layer, and the feature preprocessing layer inputs the preliminary image features output by the feature extraction layer and performs feature processing on the preliminary image features to obtain basic feature information that can be used for classification prediction.

[0113] Please refer to Figure 4 , Figure 4 is an embodiment flowchart of step S20 provided in the embodiments of the present application. In some embodiments of the present application, step S20 can specifically include steps S21-S23, wherein:

[0114] S21, performing feature extraction processing on the sample image to obtain image feature information of the sample image.

[0115] The image feature information refers to the image spatial feature obtained after performing feature extraction processing on the sample image.

[0116] For example, the sample image is input into the conventional branch and the augmented branch as shown in Figure 3 , the first feature extraction layer in the conventional branch performs feature extraction processing on the sample image to obtain the image feature information of the sample image. The second feature extraction layer in the augmented branch also performs feature extraction processing on the sample image to also obtain the image feature information of the sample image.

[0117] S22, obtaining first mean information and first variance information of the image feature information.

[0118] The first mean information refers to the mean value of the image feature information of each category obtained by fitting. The first variance information refers to the variance of the image feature information of each category obtained by fitting.

[0119] For example, for image feature information of different categories, a Gaussian distribution is used to fit the mean value and variance of the image feature information of each category. Specifically, the first mean information of the image feature information is determined by the following formula (1), and the first variance information of the image feature information is determined by the following formula (2), wherein formula (1) and formula (2) are respectively:

[0120]

[0121]

[0122] wherein μi denotes the mean of the i-th class (i.e., first mean information), δ i denotes the variance of the i-th class (i.e., first variance information), N i denotes the sample size of the i-th class, denotes the image feature information of the j-th sample of the i-th class, T denotes the current iteration number.

[0123] S23, determining the basic feature information and the augmented feature information according to the first mean information, the first variance information, and the sample distribution label.

[0124] Please refer to Figure 6 , Figure 6 is a schematic diagram of a feature augmentation process provided in an embodiment of the present application.

[0125] As Figure 6 shown, in order to realize the augmentation of the image features of the tail sample data, in some embodiments of the present application, by keeping the mean of the image feature information of the tail class (i.e., first mean information) unchanged, directly migrating the variance of the image feature information of the head class (i.e., first variance information) to the variance of the image feature information of the tail class (i.e., second variance information), the augmented image feature information can be obtained.

[0126] As can be seen from the above, since the variance and the mean of the image feature information are used as the basic feature information and the augmented feature information for image classification prediction, since the variance and the mean information can fully reflect the distribution of the sample data, and the variance can reflect the degree of change of the sample data, therefore, using the mean and the variance information is more conducive to feature augmentation.

[0127] At this time, step S23 can specifically include: taking the first mean information and the first variance information as the basic feature information; performing classification processing on the image feature information according to the sample distribution label to obtain tail image feature information and head image feature information of the sample image; taking the first variance information corresponding to the head image feature information as the second variance information of the tail image feature information; and taking the first mean information corresponding to the tail image feature information as the second mean information of the tail image feature information; taking the second mean information and the second variance information as the augmented feature information.

[0128] Wherein, the tail image feature information refers to the first mean information and the first variance information of the image feature information of the tail class sample image. The head image feature information refers to the first mean information and the first variance information of the image feature information of the head class sample image.

[0129] Specifically, in one aspect, the first mean information and the first variance information of the image feature information are directly taken as the basic feature information of the sample image. For example, the first mean information and the first variance information of the image feature information of the tail class are μ tail , δ tail , respectively. The first mean information and the first variance information of the image feature information of the head class are μ head , δ head , respectively. Then, μ tail , δ tail can be taken as the basic feature information of the tail class, and μ head , δ head can be taken as the basic feature information of the head class.

[0130] On the other hand, first, according to the indication of the sample distribution label, the image feature information is classified to obtain the tail image feature information and the head image feature information of the sample image. For example, the first mean information and the first variance information of the image feature information of the tail class are μ tail , δ tail , respectively, i.e., the tail image feature information is μ tail , δ tail . The first mean information and the first variance information of the image feature information of the head class are μ head , δ head , respectively, i.e., the head image feature information is μ head , δ head .

[0131] Then, the first variance information (such as δ head ) corresponding to the head image feature information is taken as the second variance information (such as δ' tail ) of the tail image feature information, and the first mean information (such as μ tail ) corresponding to the tail image feature information is taken as the second mean information (such as μ' tail ) of the tail image feature information.

[0132] Finally, the second mean information and the second variance information are taken as the augmented feature information. For example, the second variance information (such as δ' tail ) and the second mean information (such as μ' tail ) of the tail image feature information are taken as the augmented feature information (μ' tail , δ' tail )

[0133] As can be seen from the above, specifically, the basic feature information refers to the mean and variance of the image feature information.

[0134] It can be seen from the above that by taking the first variance information corresponding to the head image feature information as the second variance information of the tail image feature information, taking the first mean information corresponding to the tail image feature information as the second mean information of the tail image feature information, and taking the second mean information and the second variance information as the augmented feature information, the increase of the sample space feature is realized, the diversity of the feature space of the sample is made, and the generalization on the tail sample data (i.e. few samples) category is made.

[0135] Since the first variance information (such as δ head ) corresponding to the head image feature information is directly taken as the second variance information (such as δ' tail ) of the tail image feature information, it is easy to cause a huge change in data. In order to prevent the huge change in data, in some embodiments of the present application, buffering is performed by the following formula (3), that is, the step of "taking the first variance information corresponding to the head image feature information as the second variance information of the tail image feature information" can specifically include: determining the second variance information according to the first variance information corresponding to the head image feature information and a preset buffering formula, wherein the buffering formula is:

[0136] δ' tail = αδ tail + (1-α)δ head Formula (3)

[0137] Wherein, δ' tail represents the second variance information, δ tail represents the first variance information corresponding to the tail image feature information, δ head represents the first variance information corresponding to the head image feature information, and α is a hyperparameter.

[0138] As can be seen from the above, by setting the hyperparameter α through the preset buffering formula, the huge change in data can be slowed down, and the adverse effects of the huge change in data on model training are avoided.

[0139] S30, performing classification prediction processing according to the basic feature information and the augmented feature information to obtain a first classification result and a second classification result of the sample image.

[0140] The first classification result refers to the category of the sample image obtained by performing classification prediction processing on the basic feature information output by the first feature extraction layer as shown in formula (1) or the feature extraction layer as shown in formula (2) through the first classification prediction layer of the preset model. Figure 3 Figure 4 The second classification result refers to the category of the sample image obtained by performing classification prediction processing on the augmented feature information output by the second feature extraction layer as shown in formula (3) or the feature extraction layer as shown in formula (4) through the second classification prediction layer of the preset model.

[0141] The second classification result refers to the category of the sample image obtained by performing classification prediction processing on the augmented feature information output by the second feature extraction layer as shown in formula (3) or the feature extraction layer as shown in formula (4) through the second classification prediction layer of the preset model. Figure 3 ​​Figure 4 The basic feature information output by the feature extraction layer (as shown) is used for classification and prediction to obtain the category of the sample image. Alternatively, it can be obtained through the second classification prediction layer of a preset model, based on (such as...) Figure 3 The second classification prediction layer shown, or as... Figure 4 The augmented feature information output by the second classification prediction layer (shown) is used for classification prediction to obtain the category of the sample image.

[0142] like Figure 3 As shown, in some embodiments, a first classification result of the sample image can be obtained by performing classification prediction processing on the basic feature information output by the first feature extraction layer through a first classification prediction layer. A second classification result of the sample image can be obtained by performing classification prediction processing on the basic feature information output by the second feature extraction layer through a second classification prediction layer. A second classification result of the sample image can also be obtained by performing classification prediction processing on the augmented feature information output by the feature augmentation layer through a second classification prediction layer.

[0143] like Figure 4 As shown, in some embodiments, a first classification prediction layer performs classification prediction processing on the basic feature information output by the feature extraction layer to obtain a first classification result for the sample image. A second classification prediction layer performs classification prediction processing on the basic feature information output by the feature extraction layer to obtain a second classification result for the sample image. A second classification result for the sample image can also be obtained by performing classification prediction processing on the augmented feature information output by the feature augmentation layer using the second classification prediction layer.

[0144] Since the basic feature information refers to the mean and variance of image feature information, the mean and variance of image feature information can be used for classification regression to predict the category of the sample image. Specifically, in some embodiments, step S30 may include: performing classification prediction processing on the sample image based on the first mean information and the first variance information to obtain the first classification result; and performing classification prediction processing on the sample image based on the second mean information and the second variance information to obtain the second classification result.

[0145] As can be seen from the above, variance and mean information can fully reflect the distribution of sample data, and variance can reflect the degree of change in sample data. By using the variance and mean of image feature information as basic and augmented feature information for image classification prediction, feature augmentation can be directly performed using mean and variance information.

[0146] S40, updating model parameters of a preset model according to the first classification result, the second classification result and the category label, and taking the preset model as a trained classification model when the preset model converges, wherein the classification model is used for classifying images.

[0147] It can be seen from the above that, by performing feature extraction processing on a plurality of sample images according to sample distribution labels annotated for indicating that the sample images are head data or tail data, basic feature information and augmented feature information of the sample images are obtained; by performing classification prediction processing according to the basic feature information and the augmented feature information, a first classification result and a second classification result of the sample images are obtained; by updating model parameters of a preset model according to the first classification result, the second classification result and the category label, and taking the preset model as a trained classification model when the preset model converges, on the one hand, the feature space of tail data in the sample images can be augmented by the sample distribution labels, and the augmented feature information is used for model training, so that the classification generalization of the trained classification model on the tail sample data (i.e. few samples) category is improved, thereby improving the classification accuracy of the classification model on the few sample category, and further improving the accuracy of the entire classification model. On the other hand, since the feature augmentation is performed at the feature space level, it is not necessary to rely on the number of sample images.

[0148] In some embodiments of the present application, step S40 can specifically include the following steps S41-S44:

[0149] S41, determining a first classification loss value of the preset model according to the first classification result and the category label.

[0150] For example, the loss value of the conventional branch of the preset model can be determined according to formula (4) as follows, that is, the first classification loss value of the preset model can be determined according to formula (4) as follows, wherein formula (4) is:

[0151]

[0152] wherein L1 is the first classification loss value, x i is the actual category corresponding to the category label of the i-th sample, y i is the category of the i-th sample (i.e. the first classification result), e is a natural constant, w is the j-th unit weight of the full connection layer (i.e. the classification prediction layer), N is the sample size, and b is the balance parameter.

[0153] S42, determining a second classification loss value of the preset model according to the second classification result and the category label.

[0154] To fully reflect the degree of change of the sample data, in some embodiments, the second classification loss value of the preset model is determined by a loss function of data variance. For example, the loss value of the augmented branch of the preset model can be determined according to formula (5) as follows, that is, the second classification loss value of the preset model can be determined according to formula (5) as follows, wherein formula (5) is:

[0155]

[0156] wherein L2 is the second classification loss value, x i is the actual category corresponding to the category label of the i-th sample, y i is the category of the i-th sample (i.e., the second classification result), e is a natural constant, w is the j-th unit weight of the full connection layer (i.e., the classification prediction layer), N is the sample size, b is the balance parameter, s is the hyperparameter, δ tail is the variance after augmentation (i.e., the second variance information).

[0157] S43, determining the total loss value of the preset model according to the first classification loss value and the second classification loss value.

[0158] As shown in Figure 3 or Figure 4 The total loss value of the preset model can be determined by the loss function in formula (6) as follows, wherein formula (6) is:

[0159] L = (1 - λ) L1 + λ L2 formula (6)

[0160] wherein, T is the current iteration number of model training, Tmax is the maximum iteration number, L is the total loss value of the preset model, L1 is the first classification loss value, and L2 is the second classification loss value.

[0161] S44, updating the model parameters according to the total loss value, and taking the preset model as a trained classification model when the preset model converges.

[0162] For example, the model parameters of the preset model are updated by back propagation according to the total loss value of the preset model, and the preset model takes the above total loss value as the final objective function and iteratively trains until the error of the total loss value is minimized. Until the model converges, the updated preset model can be taken as a trained classification model, and the trained classification model can classify images.

[0163] From the above, it can be seen that the design of the preset model double branch (including the conventional branch and the augmented branch) is in the form of game equilibrium, so that the network of the augmented branch occupies an important component in the classification loss. As can be seen from formula (6), the preset model has little difference from the single branch (only the conventional) model at the beginning of training, and with the increase of the number of iterations, the augmented branch focuses on the feature space balance of the class and is more highlighted.

[0164] In some embodiments of the present application, the trained classification model can also be used for image classification, that is, the training method of the classification model of the embodiments of the present application can also include: obtaining a target image; calling the classification model to perform feature extraction processing on the target image to obtain target feature information of the target image; calling the classification model to perform classification prediction processing according to the target feature information to obtain a target class of the target image.

[0165] Among them, the target image refers to the image to be classified. The target feature information refers to the image feature obtained by performing feature extraction processing on the target image by the classification model. The target class refers to the class of the target image obtained by performing classification prediction on the target image by the classification model.

[0166] For ease of understanding, please continue to refer to Figure 3 The network framework of the trained classification model corresponds to the network framework of the preset model. For example, in the actual prediction stage of the classification model, the target image can be classified by the conventional branch in Figure 3 . Specifically, the target image is input into the classification model, the first feature extraction layer in the classification model performs feature extraction processing on the target image to obtain the image feature (i.e. the target feature information) of the target image; then, the first classification prediction layer performs regression classification according to the target feature information output by the first feature extraction layer to predict the target class of the target image.

[0167] From the above, it can be seen that the target image can be classified by using the conventional branch in the trained classification model, and since the classification model is obtained by training the tail class by the double branch of the preset model, the classification model can also achieve accurate classification for images of classes with small sample sizes, thereby improving the classification accuracy of the classification model for images of classes with small sample sizes, and further improving the classification accuracy of the classification model.

[0168] In addition, it can be seen from the above that the augmented branch mentioned in the embodiments of the present application can be embedded in the existing mainstream network for optimizing the performance of the existing classification model in the face of uneven sample distribution. Moreover, without making any changes to the existing classification model, only by adding the augmented branch and retraining the conventional branch (i.e. the existing classification model as the conventional branch), the plug-and-play feature is achieved.

[0169] In order to better implement the training method of the classification model in the embodiments of the present application, on the basis of the training method of the classification model, the embodiments of the present application further provide a training device of a classification model, as shown in Figure 7 The training device of the classification model 700 includes:

[0170] The acquisition unit 701 is configured to acquire a plurality of sample images, wherein the sample images have been labeled with a class label and a sample distribution label, and the sample distribution label is used to indicate whether the sample image is head data or tail data.

[0171] The feature extraction unit 702 is configured to perform feature extraction processing according to the sample images and the sample distribution label, to obtain basic feature information and augmented feature information of the sample images.

[0172] The prediction unit 703 is configured to perform classification prediction processing according to the basic feature information and the augmented feature information, to obtain a first classification result and a second classification result of the sample images.

[0173] The update unit 704 is configured to update model parameters of a preset model according to the first classification result, the second classification result and the class label, until the preset model converges, and then the preset model is used as a trained classification model, wherein the classification model is used for classifying images.

[0174] In some embodiments of the present application, the feature extraction unit 702 is specifically configured to:

[0175] perform feature extraction processing according to the sample images, to obtain image feature information of the sample images;

[0176] obtain first mean information and first variance information of the image feature information;

[0177] determine the basic feature information and the augmented feature information according to the first mean information, the first variance information and the sample distribution label.

[0178] In some embodiments of the present application, the feature extraction unit 702 is specifically configured to:

[0179] the first mean information and the first variance information as the basic feature information;

[0180] perform classification processing on the image feature information according to the sample distribution label, to obtain tail image feature information and head image feature information of the sample image;

[0181] take first variance information corresponding to the head image feature information as second variance information of the tail image feature information, and take first mean information corresponding to the tail image feature information as second mean information of the tail image feature information;

[0182] take the second mean information and the second variance information as the augmented feature information.

[0183] In some embodiments of the present application, the prediction unit 703 is specifically further configured to:

[0184] perform classification prediction processing on the sample image according to the first mean information and the first variance information, to obtain the first classification result;

[0185] perform classification prediction processing on the sample image according to the second mean information and the second variance information, to obtain the second classification result.

[0186] In some embodiments of the present application, the feature extraction unit 702 is specifically further configured to:

[0187] determine the second variance information according to first variance information corresponding to the head image feature information and a preset buffer formula, wherein the buffer formula is:

[0188] δ' tail = αδ tail + (1-α)δ head

[0189] wherein δ' tail represents the second variance information, δ tail represents first variance information corresponding to the tail image feature information, δ head represents first variance information corresponding to the head image feature information, and α is a hyperparameter.

[0190] In some embodiments of the present application, the update unit 704 is specifically further configured to:

[0191] determine a first classification loss value of the preset model according to the first classification result and the category label;

[0192] determine a second classification loss value of the preset model according to the second classification result and the category label.

[0193] determine a total loss value of the preset model according to the first classification loss value and the second classification loss value;

[0194] update the model parameters according to the total loss value until the preset model converges, and take the preset model as a trained classification model.

[0195] In some embodiments of the present application, the training apparatus 700 of the classification model further comprises a training unit (not shown in the figure), which is specifically configured to:

[0196] obtain a target image;

[0197] invoke the classification model to perform feature extraction processing on the target image to obtain target feature information of the target image;

[0198] invoke the classification model to perform classification prediction processing according to the target feature information to obtain a target category of the target image.

[0199] In specific implementation, each of the above units can be implemented as an independent entity, or can be combined as the same or several entities. The specific implementation of each of the above units can be referred to the method embodiments above, and will not be described here.

[0200] The training apparatus of the classification model can perform the method of training the classification model of the present application as described above. Figures 1 to 6 The training apparatus of the classification model can perform the method of training the classification model of the present application as described above. Figures 1 to 6 The training apparatus of the classification model can perform the method of training the classification model of the present application as described above.

[0201] In addition, in order to better implement the method of training the classification model in the embodiments of the present application, based on the method of training the classification model, the embodiments of the present application further provide an electronic device, which is described in detail below. Figure 8 , Figure 8 Fig. 1 shows a structural schematic diagram of an electronic device according to an embodiment of the present application. Specifically, the electronic device provided by the embodiments of the present application comprises a processor 801, which is configured to implement the steps of the method of training the classification model of the present application as described above when executing a computer program stored in a memory 802. Figures 1 to 6 The training apparatus of the classification model can perform the method of training the classification model of the present application as described above. Figure 7 The training apparatus of the classification model can perform the method of training the classification model of the present application as described above.

[0202] For example, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory 802 and executed by the processor 801 to complete the embodiments of the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which is used to describe the execution process of the computer program in the computer device.

[0203] The electronic device can include, but is not limited to, the processor 801, the memory 802. Those skilled in the art can understand that the schematic diagram is only an example of the electronic device and does not constitute a limitation on the electronic device, and can include more or fewer components than the schematic diagram, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc. The processor 801, the memory 802, the input / output device, and the network access device are connected through the bus.

[0204] The processor 801 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the electronic device, and connects all parts of the electronic device through various interfaces and lines.

[0205] The memory 802 can be used to store computer programs and / or modules, and the processor 801 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 802, and calling data stored in the memory 802. The memory 802 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the electronic device (such as audio data, video data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0206] Those skilled in the art can clearly understand the specific working process of the training device of the classification model, the electronic device and the corresponding units described above for the convenience and brevity of description, which can be referred to as Figures 1 to 6 The description of the training method of the classification model corresponding to any embodiment will not be repeated here.

[0207] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0208] Therefore, the embodiments of the present application provide a computer readable storage medium, which stores a plurality of instructions. The instructions can be loaded by a processor to execute the embodiments of the present application, such as Figures 1 to 6 The specific operations of the steps in the training method of the classification model corresponding to any embodiment can be referred to as Figures 1 to 6 The description of the training method of the classification model corresponding to any embodiment will not be repeated here.

[0209] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0210] Due to the instructions stored in the computer readable storage medium, the embodiments of the present application, such as Figures 1 to 6 The steps in the training method of the classification model corresponding to any embodiment, so the embodiments of the present application, such as Figures 1 to 6The beneficial effects that can be achieved by the training method of the classification model in any embodiment are described in detail in the foregoing description, and will not be repeated here.

[0211] The training method of the classification model, the device, the electronic equipment and the computer readable storage medium provided by the embodiments of the present application are described in detail above, the principles and implementation modes of the present application are described in this paper, and the above embodiment description is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A method for training a classification model, the method comprising: The method comprises: obtaining a plurality of sample images, wherein the sample images have been labeled with a category label and a sample distribution label, the sample distribution label being used to indicate whether the sample image is head data or tail data, and the number of samples corresponding to the category in the head data is greater than the number of samples corresponding to the category in the tail data; performing feature extraction processing on the sample images to obtain image feature information of the sample images; obtaining first mean information and first variance information of the image feature information; taking the first mean information and the first variance information as basic feature information; performing classification processing on the image feature information according to the sample distribution label to obtain tail image feature information and head image feature information of the sample images; taking first variance information corresponding to the head image feature information as second variance information of the tail image feature information, and taking first mean information corresponding to the tail image feature information as second mean information of the tail image feature information; taking the second mean information and the second variance information as augmented feature information; performing classification prediction processing on the sample images according to the first mean information and the first variance information to obtain a first classification result; performing classification prediction processing on the sample images according to the second mean information and the second variance information to obtain a second classification result; updating model parameters of a preset model according to the first classification result, the second classification result, and the category label, until the preset model converges, and taking the preset model as a trained classification model, wherein the classification model is used to classify images. 2.The method of claim 1, wherein, The first variance information corresponding to the head image feature information is taken as the second variance information of the tail image feature information, comprising: determining the second variance information according to the first variance information corresponding to the head image feature information and a preset buffer formula, wherein the buffer formula is: delta tail = alpha delta tail + (1 - alpha) delta head wherein, δ tail represents the second variance information, δ tail represents the first variance information corresponding to the tail image feature information, δ head represents the first variance information corresponding to the head image feature information, and a is a hyperparameter.

3. The method of claim 1, wherein the training of the classification model is performed by using a machine learning algorithm. The model parameters of the preset model are updated according to the first classification result, the second classification result, and the category label, until the preset model converges, and the preset model is taken as a trained classification model, comprising: determining a first classification loss value of the preset model according to the first classification result and the category label; determining a second classification loss value of the preset model according to the second classification result and the category label; determining a total loss value of the preset model according to the first classification loss value and the second classification loss value; updating the model parameters according to the total loss value, until the preset model converges, and taking the preset model as a trained classification model. 4.The method of any one of claims 1-3, wherein, The method further comprises: obtaining a target image; calling the classification model to perform feature extraction processing on the target image to obtain target feature information of the target image; calling the classification model to perform classification prediction processing on the target feature information to obtain a target category of the target image.

5. A training device for a classification model, characterized in that, The training device of the classification model comprises: An acquisition unit is configured to acquire a plurality of sample images, wherein the sample images are labeled with a category label and a sample distribution label, and the sample distribution label is used to indicate whether the sample image is head data or tail data; in the sample images, the number of samples corresponding to the category of the head data is greater than the number of samples corresponding to the category of the tail data; A feature extraction unit is configured to perform feature extraction processing on the sample images to obtain image feature information of the sample images; acquire first mean information and first variance information of the image feature information; take the first mean information and the first variance information as basic feature information; perform classification processing on the image feature information according to the sample distribution label to obtain tail image feature information and head image feature information of the sample images; take first variance information corresponding to the head image feature information as second variance information of the tail image feature information; and take first mean information corresponding to the tail image feature information as second mean information of the tail image feature information; take the second mean information and the second variance information as augmented feature information; A prediction unit is configured to perform classification prediction processing on the sample images according to the first mean information and the first variance information to obtain a first classification result; and perform classification prediction processing on the sample images according to the second mean information and the second variance information to obtain a second classification result. An update unit is configured to update model parameters of a preset model according to the first classification result, the second classification result and the category label, until the preset model converges, and take the preset model as a trained classification model, wherein the classification model is used to classify images.

6. An electronic device, comprising: A processor and a memory are included, the memory stores a computer program, and the processor executes the computer program in the memory to perform the training method of the classification model according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is loaded by a processor to perform the steps in the training method of the classification model according to any one of claims 1 to 4.

8. A computer program product, characterised in that, A computer program is included, and the computer program is loaded by a processor to perform the steps in the training method of the classification model according to any one of claims 1 to 4.

Images