Enhancement of sample image data, model training, classification method, device and equipment

Abstract

The application discloses a sample image data enhancement method, an image classification model training method, an image classification method, an image classification device, an image classification equipment and a storage medium, to solve the problem of how to strengthen the role of unlabeled sample image data in the model training process, so as to avoid the imbalance phenomenon of model prediction for categories. The enhancement method comprises the following steps: from a sample image data set, difficult sample image data satisfying a preset classification difficulty condition is selected; a data enhancement manner matched with a classification task of a first image classification model to be trained is used to perform data enhancement on the difficult sample image data, so as to obtain an augmented sample image data set; wherein, the augmented sample image data set is used for model training of the first image classification model to be trained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method for enhancing sample image data, a method for training an image classification model, an image classification method, an apparatus, a device, and a storage medium. Background Technology

[0002] In semi-supervised object detection tasks, the number of labeled sample images in the dataset is often relatively small. Therefore, using such a dataset to train a deep learning model will face the problem of "limited supervision". This can easily lead to an imbalance in the training deep learning model when making class predictions. That is, the model is more likely to identify common classes or easy-to-detect targets, while missing rare classes or difficult-to-detect targets.

[0003] To address the aforementioned issues, it is urgent to find a way to enhance the role of unlabeled sample image data in the training process in order to avoid the aforementioned imbalance phenomenon. Summary of the Invention

[0004] This application provides a method for enhancing sample image data to address the issue of how to strengthen the role of unlabeled sample image data in the model training process, thereby avoiding the problem of imbalance in model prediction of categories.

[0005] This application also provides a training method for an image classification model to address the problem of how to enhance the role of unlabeled sample image data in the training process, so as to avoid the problem of imbalance in class prediction.

[0006] This application also provides an image classification method, as well as an apparatus, device, and computer-readable storage medium corresponding to the sample image data enhancement method, the image classification model training method, and the image classification method, respectively.

[0007] The embodiments of this application adopt the following technical solutions:

[0008] A method for enhancing sample image data, comprising:

[0009] From the sample image dataset, select difficult sample image data that meets the preset classification difficulty conditions; the difficult sample image data is the unlabeled sample image in the sample image dataset that is input into the first image classification model to be trained, and the unsupervised loss obtained satisfies the unlabeled sample image corresponding to the preset classification difficulty conditions;

[0010] The difficult sample image data is augmented using a data augmentation method that matches the classification task of the first image classification model to be trained, to obtain an augmented sample image dataset.

[0011] The augmented sample image dataset is used for model training of the first image classification model to be trained.

[0012] A method for training an image classification model, comprising:

[0013] Obtain a sample image dataset; the sample image dataset includes labeled sample image data, unlabeled sample image data, and first augmented sample image data obtained based on the unlabeled sample image data;

[0014] The sample image dataset is input into the first image classification model to be trained; iterative training is performed, and the parameters of the first image classification model to be trained are adjusted according to the supervised loss and unsupervised loss of the first image classification model to be trained for the sample image dataset; until the iteration conditions of the first image classification model are met, the trained first image classification model is obtained.

[0015] The first augmented sample image data is obtained using a sample image data enhancement method as described above.

[0016] An image classification method, comprising:

[0017] Obtain the target image data to be classified;

[0018] The target image data is input into the trained first image classification model to obtain the image classification result of the target image data;

[0019] The first image classification model is trained using the image classification model training method described above.

[0020] An enhancement device for sample image data, comprising:

[0021] The difficult sample selection unit selects difficult sample image data that meets the preset classification difficulty conditions from the sample image dataset; the difficult sample image data is the unlabeled sample image data in the sample image dataset that is input into the first image classification model to be trained, and the unsupervised loss obtained is the unlabeled sample image data that meets the preset classification difficulty conditions.

[0022] The augmentation unit is used to perform data augmentation on the difficult sample image data by adopting a data augmentation method that matches the classification task of the first image classification model to be trained, so as to obtain an augmented sample image dataset.

[0023] The augmented sample image dataset is used for model training of the first image classification model to be trained.

[0024] An image classification device, comprising:

[0025] The acquisition unit acquires the target image data to be classified.

[0026] A classification unit is used to input the target image data into a trained first image classification model to obtain the classification result of the first image classification model for the target image data;

[0027] The first image classification model is trained using the image classification model training method described above.

[0028] A computing device includes: a memory and a processor, wherein,

[0029] The memory is used to store computer programs;

[0030] The processor, coupled to the memory, is used to execute the computer program stored in the memory for performing the methods described above.

[0031] A computer-readable storage medium storing a computer program that, when executed by a computer, enables the implementation of the above-described method.

[0032] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0033] By selecting unlabeled sample image data from the sample image dataset that meet the preset classification difficulty conditions for unsupervised loss, performing data augmentation, and using the data-augmented difficult sample image data for training the first image classification model to be trained, the training of the first image classification model (such as a deep learning model) based on the difficult sample image data can be more thorough. As a result, the model trained in the end can have high classification accuracy for "classification difficulties" such as rare categories and hard-to-detect targets.

[0034] Meanwhile, the solution provided in this application focuses on difficult sample image data that requires data augmentation, rather than treating all unlabeled sample image data in the sample image dataset equally. Augmenting the difficult sample image data and then training the first image classification model based on the augmented sample image data significantly improves the accuracy of the first image classification model in recognizing "difficult-to-classify" targets. Therefore, the solution provided in this application is a preferred solution that can significantly improve the sample image dataset and leverage the role of unlabeled sample image data in the training process without consuming excessive processing resources. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0036] Figure 1 A flowchart illustrating the specific implementation of a sample image data enhancement method provided in this application embodiment;

[0037] Figure 2 This is a schematic diagram showing the positional relationship between the detection box and the object, as well as the coordinates of the four corners of the detection box in the image coordinate system.

[0038] Figure 3 This is a schematic diagram illustrating the principle of sample augmentation on a dataset of sample images used to train a deep neural network performing a semi-supervised object detection task.

[0039] Figure 4a A schematic diagram of data augmentation for difficult sample images;

[0040] Figure 4b A schematic diagram of data augmentation for difficult sample images;

[0041] Figure 5 A schematic diagram illustrating the specific process of training an image classification model according to an embodiment of this application;

[0042] Figure 6 A schematic diagram of the specific structure of a sample image data enhancement device provided in an embodiment of this application;

[0043] Figure 7 A schematic diagram of the specific structure of a training device for an image classification model provided in an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the specific structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0047] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0048] Deep learning (DL) is a research area within machine learning (ML) that has been introduced to bring machine learning closer to its original goal—artificial intelligence (AI). In recent years, deep learning has achieved great success in several areas of computer vision, such as image classification and object detection.

[0049] In fields such as image classification and object detection, the training of deep learning models often relies on large-scale labeled image datasets. However, in practical applications, obtaining large-scale, high-quality image datasets and manually labeling them is difficult and costly. In contrast, obtaining unlabeled image datasets is usually easy and inexpensive. Therefore, researchers hope to alleviate the over-reliance of deep learning models on labeled image datasets to some extent by utilizing a large amount of unlabeled image datasets.

[0050] Semi-supervised learning is a training method that effectively improves the training performance of deep learning models on limited labeled image data by utilizing a large amount of unlabeled sample image data.

[0051] In semi-supervised object detection tasks, the number of labeled image samples in the dataset is often relatively small, leading to an imbalance in the proportion of labeled and unlabeled image samples. This imbalance can easily cause deep learning models to exhibit class imbalance when making class predictions—that is, the model is more likely to identify common or easily detected classes, while missing rare or difficult-to-detect classes. Furthermore, this imbalance can also lead to "confirmation bias" in semi-supervised deep learning models performing tasks such as image classification or object detection—that is, the model tends to misclassify unfamiliar targets as familiar ones.

[0052] One embodiment of this application provides a method for enhancing sample image data to address how to strengthen the role of unlabeled sample image data in the model training process, thereby avoiding the problem of low accuracy in class prediction by the model.

[0053] The subject executing this method can be any computing device capable of implementing the method, such as a server, mobile phone, personal computer, smart wearable device, smart robot, etc.

[0054] Furthermore, the embodiments of this application do not limit the execution order of different steps. When using the method provided in the embodiments of this application, the execution order of different steps can be adjusted according to actual needs.

[0055] For ease of description, the following uses a sample image data enhancement device as the execution subject of this method to provide a detailed description of the method provided in this application embodiment.

[0056] like Figure 1 The diagram shown is a flowchart illustrating a specific implementation of a sample image data enhancement method provided in this application, comprising the following steps:

[0057] Step 11: Select unlabeled sample image data that meet the preset classification difficulty conditions from the sample image dataset, and use them as difficult sample image data;

[0058] The unlabeled sample image data that meets the preset classification difficulty condition is at least one unlabeled sample image data in the sample image dataset. Furthermore, the at least one unlabeled sample image data satisfies the following condition: if the model to be trained (for ease of description, it can be referred to as the first image classification model to be trained) is used to predict the category of the at least one unlabeled sample image data, the difference between the predicted category and the true category of the at least one unlabeled sample image data is greater than a preset difference. In other words, the at least one unlabeled sample image data presents a "classification difficulty" problem for the first image classification model.

[0059] In summary, the "unlabeled sample image data that meets the preset classification difficulty conditions" mentioned in the embodiments of this application refers to unlabeled sample image data in which the target contained in the image belongs to a rare category and / or is difficult to detect for the first image classification model to be trained.

[0060] The degree of difference between the predicted category and the true category, as mentioned here, can be measured by the total loss generated by the first image classification model to be trained when predicting the category of the corresponding unlabeled sample. The larger the total loss, the greater the difference between the predicted category and the true category; conversely, the smaller the difference between the predicted category and the true category.

[0061] For a specific implementation of step 11, for example, the unlabeled sample image data in the sample image dataset can be input into the first image classification model to be trained, and the unlabeled sample image data corresponding to the unsupervised loss that satisfies the preset classification difficulty condition can be selected as the difficult sample image data.

[0062] The first image classification model to be trained mentioned here could be a deep learning model used to perform image data classification tasks, or a deep learning model used to perform object detection tasks, and so on.

[0063] This application does not limit the specific type of the model or the task to be performed.

[0064] The sample image dataset refers to the training set used to train the first image classification model to be trained. For ease of distinction, the sample image dataset mentioned in step 11 can be referred to as the first sample image dataset.

[0065] The first image classification model to be trained described in this application embodiment may employ a training method that includes semi-supervised learning. Corresponding to the semi-supervised learning training method, the first sample image dataset described in this application embodiment includes at least: unlabeled sample image data and labeled sample image data.

[0066] The unsupervised loss that satisfies the preset classification difficulty condition in step 11 may include, for example, at least one of the following:

[0067] 1) Unsupervised loss within a predetermined loss range—for example, greater than a predetermined loss threshold;

[0068] 2) When the unlabeled sample image data in the first sample image dataset is input into the first image classification model to be trained, the relatively larger unsupervised loss among the various unsupervised losses is selected. For example, it can be the unsupervised loss that is relatively earlier in the order of the loss value from largest to smallest among the various unsupervised losses of the first image classification model to be trained for all the unlabeled sample image data in the first sample image dataset.

[0069] The unsupervised losses ranked relatively high mentioned here can be a preset number of such losses. Since these preset number of unsupervised losses are ranked relatively high, they are greater than the other unsupervised losses among the obtained unsupervised losses excluding these preset number. Therefore, hard sample image data, which belongs to unlabeled sample image data with a large prediction loss, can significantly improve the deficiencies of the sample image dataset by performing data augmentation on hard sample image data compared to other unlabeled sample image data with relatively small prediction losses.

[0070] The following example illustrates the specific implementation of step 11 using a semi-supervised object detection task for image data.

[0071] Object detection, or object detection in English, refers to determining the location and category of objects from input image data (which often contains many objects). It is a core and fundamental task in computer vision. Image data segmentation, object tracking, and keypoint detection typically rely on object detection.

[0072] Generally, the objective of object detection is for the detector to output the object's class and the location of its corresponding bounding box. The bounding box, also called the object detection box or boundary box, is typically rectangular in shape. Figure 2 The diagram shows the positional relationship between the detection box and the object, as well as the coordinates of the four corners of the detection box in the image data coordinate system.

[0073] Typically, there are three formats for representing the position of the detection box:

[0074] XYXY, which is (X1,Y1,X2,Y2), where (X1,Y1) are the coordinates of the top left corner of the detection box, and (X2,Y2) are the coordinates of the bottom right corner of the detection box;

[0075] XYwh, or (X,Y,w,h), where (X,Y) are the coordinates of the top-left corner of the detection box, w is the width of the detection box, and h is the height of the detection box; w and h can be referenced... Figure 2 ;

[0076] cXcYwh, which is (Xc,Yc,w,h), where (Xc,Yc) are the coordinates of the center point of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0077] All three data formats mentioned above can be referred to as the four-dimensional data of the detection box. In particular, (X1, Y1, X2, Y2) are called the four-dimensional coordinates of the detection box.

[0078] like Figure 3 The diagram illustrates the principle of sample augmentation using the sample image data augmentation method provided in this application embodiment, specifically for training a first image classification model to perform the semi-supervised object detection task on image data. Based on this schematic diagram, the process related to determining difficult sample image data in step 11 mainly includes the following steps:

[0079] Step 1 ( Figure 3 (Not shown in the image): Training is performed on the pre-trained model based on labeled sample image data (the image data mentioned here can also be called image data);

[0080] The specific training method includes: dividing all labeled sample images in the sample image dataset into multiple batches; sequentially inputting the labeled sample images from different batches into the "pre-trained model" in the semi-supervised object detection framework to train the pre-trained model, thereby obtaining a trained pre-trained model. For ease of comparison and description, the trained pre-trained model can also be referred to as a pre-trained second image classification model.

[0081] The pre-trained model is then used as a teacher model to train the first image classification model, which serves as the student model, using knowledge distillation.

[0082] In one alternative implementation, the classifier of the teacher model can employ Faster RCNN and Region Proposal Networks (RPN) heads, while the backbone network can be a ResNet50 pre-trained on ImageNet.

[0083] When training a pre-trained model based on labeled sample image data, the formula for calculating the supervised loss of the pre-trained model can be shown in the following formula [1]:

[0084]

[0085] The relevant explanations for the above formula [1] are as follows:

[0086] L s (x,p * ,t * ) represents the supervised loss of the pre-trained model.

[0087] L s (x,p * ,t * The loss can include two parts: classification loss L. cls and regression loss L regAmong them, the classification loss L cls The regression loss L is used to measure the predictive loss of a pre-trained model for the category label of a detection box. reg Used to measure the degree of loss of a pre-trained model in predicting the location labels of detection boxes.

[0088] x represents the labeled sample image data.

[0089] p * The category label is a reference box that indicates the category of the object / target object enclosed by the reference box. The reference box is equivalent to the true / correct detection box of the target object in the labeled sample image data, and can be obtained by labeling the labeled sample image data.

[0090] This is the category label for the reference box at index i. Correspondingly, p i The category label of the detection box with index i predicted by the pre-trained model. To show that the category label mentioned here is predicted, it can also be called the predicted category label obtained by the pre-trained model.

[0091] t * The values ​​of the four-dimensional coordinates of the reference box can be called the position labels. This is the position label of the reference box at index i. Correspondingly, t i The location label of the detection box with index i predicted by the pre-trained model is also referred to as the predicted location label obtained by the pre-trained model, to show that the location label is predicted.

[0092] i is the index of a single reference box in all the labeled sample image data, and its value ranges from [1, I], where I is the maximum number of all reference boxes in all the labeled sample image data.

[0093] The purpose of training the pre-trained model using labeled sample image data with reference boxes is to ensure that when the pre-trained model performs object detection on image data, the predicted location of the detection box can be as close as possible to the actual location of the corresponding reference box, and the category label of the predicted detection box can be as similar as possible to the category label of the corresponding reference box.

[0094] For classification loss, it is used to measure the degree of loss of the pre-trained model in predicting the class label of the detection box at index i; For regression loss, the value of this function is used to measure the degree of loss of the pre-trained model in predicting the location label of the detection box at index i.

[0095] N clsN represents the number of labeled sample image data points input into a batch of the pre-trained model; reg This represents the number of reference box positions in the batch of labeled sample image data; λ represents the hyperparameter for balancing the weights.

[0096] Step 2: As Figure 3 As shown, unlabeled sample image data from the sample image dataset is input into a pre-trained model to generate pseudo-labels (called the first pseudo-labels) for the unlabeled sample image data based on the prediction results of the pre-trained model.

[0097] The specific implementation method of step 2 is as follows:

[0098] Suppose that for unlabeled sample image data x input to a pre-trained model, the predicted category label of the detection box obtained by the pre-trained model for x is θ. Then, if max(p(x,θ))≥γ, p(x,θ) is transformed into a pseudo-label q. * =ONE_HOT(argmax(p(x,θ))), which can be called the first pseudo-label of the category for easy distinction and description.

[0099] For the predicted location labels, no conversion is needed; the predicted location labels can be directly used as location pseudo-labels. For ease of distinction and description, these can be called the first location pseudo-labels.

[0100] Category pseudo-tags and position pseudo-tags are collectively called pseudo-tags. For ease of description, they can be referred to as the first pseudo-tag.

[0101] in:

[0102] γ is the confidence threshold for pseudo-labels, and its specific value can be set to, for example, 0.9.

[0103] θ represents one of the possible class labels of the detection box of the target object in x predicted by the pre-trained model for x.

[0104] p(x,θ) represents the probability that the bounding box in x has the class label θ. max(p(x,θ))≥γ means that the probability of the bounding box in x having the class label θ is greater than the false label confidence threshold, indicating that the true class label of the bounding box in x is very likely to be θ. Therefore, we can follow q... * =ONE_HOT(argmax(p(x,θ))) converts p(x,θ) into the first pseudo-label q of the category. * .

[0105] For ease of description, the unlabeled sample image data with the first pseudo-label obtained by performing step 2 is referred to as the first pseudo-label sample image data.

[0106] Step 3: After obtaining the first pseudo-label sample image data, perform strong data augmentation on the first pseudo-label sample image data to obtain the first pseudo-label sample image data after strong data augmentation.

[0107] The strong data augmentation mentioned here may include, but is not limited to, color transformation, geometric transformation, box-level transformation, or cutout on the first pseudo-label sample image data.

[0108] It should be noted that if strong data augmentation is performed on the first pseudo-label sample image data, including methods such as geometric transformations that may change the position of the detection box, then the position of the detection box in the first pseudo-label may change. Geometric transformations specifically map the coordinate position in one image data to a new coordinate position in another image data, thus establishing a mapping relationship between the pixel coordinates of the two image data.

[0109] In this case, the corresponding four-dimensional coordinates of the detection box after geometric transformation can be calculated based on, for example, the four-dimensional coordinates corresponding to the first pseudo-label of the detection box before geometric transformation, and the aforementioned mapping relationship, and used as the first pseudo-label of the position after strong data augmentation.

[0110] Alternatively, the first pseudo-label sample image data after strong augmentation can be input into a pre-trained model to predict a new location pseudo-label, which can then be used as the first location pseudo-label after strong data augmentation.

[0111] The first pseudo-label of location after strong data augmentation can also be called the second pseudo-label of location or the target pseudo-label of location. Sample image data with a second pseudo-label of location and a first pseudo-label of category after the above processing can also be called second pseudo-label sample image data, or strong data augmentation first pseudo-label sample image data. The second pseudo-label of location and the first pseudo-label of category possessed by the strong data augmentation first pseudo-label sample image data can be collectively referred to as the target pseudo-label.

[0112] Of course, in one implementation, step 3 can be skipped after step 2, and step 4 can be executed directly. This application does not limit whether the first pseudo-label sample image data needs to undergo strong data augmentation.

[0113] Step 4: Using the first pseudo-label sample image data, start the first round of training for the student model (the first image classification model to be trained) to obtain the unsupervised loss of the first image classification model to be trained for the first pseudo-label sample image data.

[0114] According to step 3 above, if strong data augmentation is performed on the first pseudo-label sample image data, then the first pseudo-label sample image data mentioned in step 4 is the first pseudo-label sample image data after strong data augmentation.

[0115] The following describes the specific implementation of step 4, using the first pseudo-label sample image data described in step 4 as an example, after strong data augmentation:

[0116] like Figure 3 As shown, all labeled sample image data and all strongly augmented first pseudo-label sample image data in the sample image dataset are input into the first image classification model to be trained (input can be in batches); the first image classification model to be trained makes predictions on all labeled sample image data and all strongly augmented first pseudo-label sample image data in the sample image dataset respectively, so that the category prediction loss value can be calculated based on the prediction results.

[0117] The category prediction loss value mentioned here is specifically the unsupervised loss of the first image classification model to be trained for the first pseudo-label sample image data after strong data augmentation, which can be calculated according to the following formula [2]:

[0118]

[0119] The relevant explanations for formula [2] are as follows:

[0120] L u (A(x u ,s * ),q * L represents the unsupervised loss of the first image classification model to be trained, which can be abbreviated as L. u L u It also includes two parts of loss: classification loss L cls and regression loss L reg .

[0121] x u This represents the image data of the first pseudo-label sample.

[0122] s * The first pseudo-label indicates the position of the detection box; q * This represents the first pseudo-label of the category of the detection box. For the detection box with index j, The first pseudo-label indicates the position of the detection box. This indicates the first pseudo-label of the category of the detection box. Correspondingly, s j q represents the predicted location label of the detection box by the first image classification model to be trained. j This represents the predicted category label of the first image classification model to be trained for the detection box.

[0123] A(x u ,s * In A(x), A represents strong data augmentation.u ,s * ) represents the first pseudo-labeled sample image data x u and its position pseudo-labels s * Data augmentation was performed before the first image classification model was input for training. x represents the first pseudo-label sample image data u The first pseudo-label was strongly augmented. As mentioned above, if the first pseudo-label sample image data is not strongly augmented and is directly input into the first image classification model to be trained, then the symbol “A” in the formula [2] used to calculate the unsupervised loss of the first image classification model to be trained can be removed.

[0124] N cls In formula [2], N represents the number of first pseudo-label sample image data in a batch input to the first image classification model to be trained; reg In formula [2], λ represents the number of detection boxes in the first pseudo-label sample image data of the batch; λ is the hyperparameter of the balancing weight.

[0125] Step 5: After the first round of training, process the L values ​​output by the detector. u The values ​​are sorted in descending order, and the top K L values ​​are counted. u The value is used as the top K losses; the first pseudo-label sample image data (or the first pseudo-label sample image data after strong enhancement) where the detection box corresponding to the top K losses is located is determined as the hard sample image data, that is, the hard sample image data mentioned in step 11.

[0126] This completes the identification of the difficult sample image data.

[0127] Step 12: Use data augmentation methods that match the classification task of the first image classification model to be trained to augment the difficult sample image data and obtain augmented sample image data.

[0128] The collection of augmented sample image data is called the augmented sample image dataset.

[0129] The augmented sample image dataset is used to add to the first sample image dataset to update the first sample image dataset to the second sample image dataset.

[0130] The second sample image dataset can be used to further input the first image classification model to be trained after the first round of training, so that the first image classification model to be trained can continue to undergo semi-supervised learning. Further semi-supervised learning after the first round of training can be found in Example 2 below.

[0131] The following explains some of the statements in step 12:

[0132] The data augmentation method that matches the classification task of the first image classification model to be trained refers to the method of augmenting the features that the first image classification model to be trained is interested in for the classification task.

[0133] Specifically, the classification task of the first image classification model to be trained may include, but is not limited to, image data classification and object detection. Therefore, data augmentation methods matching such a classification task could include enhancing the features of the target object in the sample image data.

[0134] Generally, different classification tasks may focus on different characteristics of the target objects, and thus the data augmentation methods that match different classification tasks are often different.

[0135] A specific example is: if the classification task of the first image classification model to be trained includes object detection of small objects in image data, then the data augmentation method that matches such a classification task can include: enhancing the features of small objects in image data.

[0136] Another concrete example is: if the classification task of the first image classification model to be trained includes object detection of red target objects in the image data, then the data augmentation methods that match such classification tasks can include: enhancing the overall color contrast of the image data, etc.

[0137] The following example illustrates the specific process of data augmentation for difficult sample image data.

[0138] Since the data augmentation of the difficult sample image data in the following examples involves scale transformation of the target objects in the difficult sample image data, it is different from the strong data augmentation described above. The data augmentation methods in the following examples can be called multi-scale data augmentation.

[0139] Example 1: Increase the area ratio of the target object in the difficult sample image data to obtain augmented sample image data.

[0140] Combination Figure 4a The data augmentation diagram for the difficult sample image data shown in Example 1 can be implemented by the following steps:

[0141] Step a: Retain the image data area enclosed by the detection box, and while keeping the aspect ratio of the difficult sample image data unchanged, crop out part of the background image data area of ​​the difficult sample image data;

[0142] Step b: For the difficult sample image data processed by step a, perform proportional enlargement processing with a fixed aspect ratio until it is enlarged to the original size before cropping.

[0143] The hard sample image data processed by steps a and b is the augmented sample image data.

[0144] The multi-scale data augmentation operation performed on the difficult sample image data using the method in Example 1 is equivalent to enlarging the target object in the difficult sample image data, which increases the probability that the features of the target object are accurately acquired by the first image classification model to be trained, thereby enabling the first image classification model to be trained to learn the difficult sample image data better.

[0145] Example 1 restores the difficult sample image data processed in step a to its original size before cropping. This is mainly because some semi-supervised object detection frameworks have requirements for the size consistency of sample image data.

[0146] Example 2: Based on a preset reduction factor, reduce the size of different difficult sample image data, stitch the different difficult sample image data after the size reduction, and use the stitched sample image data as augmented sample image data.

[0147] Combination Figure 4b The data augmentation diagram for the difficult sample image data shown in Example 2 can be implemented by the following steps:

[0148] Step 1: While keeping the aspect ratio of the difficult sample image data unchanged, reduce the difficult sample image data according to the preset reduction factor;

[0149] The preset reduction factor can be, for example, 1 / 2 or 1 / 4. The difficult sample image data is reduced according to such a preset reduction factor, that is, the difficult sample image data is reduced to 1 / 2 or 1 / 4 of the original size based on the original size of the difficult sample image data.

[0150] Step II: The multiple difficult sample image data processed by step a are stitched together to obtain stitched image data with the same size as the original data, which is used as augmented sample image data.

[0151] like Figure 4b The diagram shows how augmented sample image data is obtained by stitching together four difficult sample image data after each of them has been reduced to 1 / 4 of their original size according to step I above.

[0152] The multi-scale data augmentation operation performed on difficult sample image data using the method in Example 2 can be applied to the target detection task of the first image classification model to be trained, which includes the detection of small target objects. Following the method in Example 2, although the area ratio of the target object in the image data decreases, it increases the number of sample images containing small target objects in the sample image dataset. This allows the first image classification model to be trained to fully learn about small target objects, improving its ability to detect small target objects.

[0153] Using the method provided in the aforementioned embodiments, unlabeled sample image data corresponding to the unsupervised loss that meets the preset classification difficulty condition is selected for data augmentation, and the data-augmented difficult sample image dataset is used for model training of the first image classification model to be trained. This allows the training of the first image classification model to be trained based on the difficult sample image data to be more sufficient, thereby enabling the finally trained model to have high classification accuracy for "classification difficulties" such as rare categories and hard-to-detect targets.

[0154] Meanwhile, the method provided in the aforementioned embodiments focuses on difficult sample image data that requires data augmentation, rather than performing data augmentation on all unlabeled sample image data in the sample image dataset indiscriminately. Augmenting the difficult sample image data and then training the first image classification model based on the augmented sample image data significantly improves the accuracy of the first image classification model in recognizing "difficult-to-classify" targets. Therefore, this method is a preferred solution that can significantly improve the sample image dataset and leverage the role of unlabeled sample image data in the training process without consuming excessive processing resources.

[0155] One embodiment of this application provides a training method for an image classification model, based on the sample image data enhancement method provided in the above embodiments.

[0156] The subject executing this method can be any computing device capable of implementing the method, such as a server, mobile phone, personal computer, smart wearable device, smart robot, etc.

[0157] Different steps of this method can be implemented by the same execution entity or by different execution entities. This application does not limit which execution entity is used to implement the method.

[0158] Furthermore, the embodiments of this application do not limit the execution order of different steps. When using the method provided in the embodiments of this application, the execution order of different steps can be adjusted according to actual needs.

[0159] For ease of description, the following uses a model training device as the execution subject of this method to provide a detailed description of the method provided in the embodiments of this application.

[0160] The training method for this image classification model may specifically include, for example: Figure 5 The following steps are shown:

[0161] Step 51: The model training device acquires the sample image dataset;

[0162] The sample image data constituting the sample image dataset includes, in addition to labeled sample image data, unlabeled sample image data (i.e., first pseudo-labeled sample image data, or first pseudo-labeled sample image data with strong data augmentation), and first augmented sample image data obtained based on the unlabeled sample image data.

[0163] The first augmented sample image data can be obtained by using the sample image data enhancement method described in Example 1. How to obtain the first augmented sample image data will not be described in detail here.

[0164] In this embodiment of the application, the sample image data in the obtained sample image dataset is used for iterative training of the first image classification model to be trained.

[0165] Any image classification model (such as an object detection model) that can be augmented using the sample image data enhancement methods described in the foregoing embodiments and can be trained using the image classification model training methods described in this embodiment falls within the scope of the first image classification model to be trained described in this application. This application does not limit the type of the image classification model or the task it performs.

[0166] It should be noted that, in the foregoing embodiments, the data augmentation method for difficult sample images may include multi-scale data augmentation, as described in Examples 1 and 2 above.

[0167] Because multi-scale data augmentation generally changes the position of the detection box in the image. Therefore, after obtaining the first augmented sample image data (specifically the first augmented sample image) according to the multi-scale data augmentation described in Example 1, the position pseudo-label of the detection box in the first augmented sample image (called the second position pseudo-label) can be determined based on the first pseudo-label of the detection box position in the hard sample image and the mapping relationship between the pixel coordinates of the hard sample image and the first augmented sample image.

[0168] Alternatively, the first augmented sample image can be input into the pre-trained model to generate a second pseudo-label for the location of the detection box in the first augmented sample image. For details on how the pre-trained model generates location pseudo-labels for unlabeled sample images in Example 1, please refer to that example; it will not be repeated here.

[0169] It should be noted that the pseudo-label of the category of the detection box in the first augmented sample image is still the first pseudo-label of the category of the detection box.

[0170] In summary, for difficult sample image data with multi-scale data augmentation, the pseudo-labels of the detection boxes in the first augmented sample image after the above processing include: a second pseudo-label for location and a first pseudo-label for category.

[0171] Step 52: The model training device inputs the acquired sample image dataset into the first image classification model to be trained;

[0172] Step 53: The model training device iteratively trains the first image classification model to be trained. Based on the supervised loss and unsupervised loss of the first image classification model to be trained for the sample image dataset, the parameters of the first image classification model to be trained are adjusted until the iterative conditions of the first image classification model are met, and the trained image classification model is obtained.

[0173] The specific implementation methods of steps 52 and 53 are illustrated below:

[0174] Continuing with the examples from the previous embodiments, please refer to the details below. Figure 3 The first image classification model to be trained is used as the deep neural network to perform the semi-supervised object detection task, and the following is employed: Figure 3 Taking the pre-trained model shown as the teacher model as an example, after inputting the sample image data into the first image classification model to be trained, the total loss of the first image classification model to be trained can be calculated according to the following formula [3]:

[0175]

[0176] The relevant explanations for formula [3] are as follows:

[0177] L s (x,p * ,t * The physical meaning and specific calculation method of L are similar to those of formula [1] in the aforementioned embodiments, and will not be repeated here. In formula [3], L s (x,p * ,t *Specifically, it represents the supervised loss of the first image classification model to be trained for labeled (real labels, not pseudo labels) sample images.

[0178] It should be noted that when using formula [1] to calculate L in formula [3] s (x,p * ,t * When ), the meanings of some parameters in equation [1] change as follows:

[0179] Change 1: p i The category label of the detection box with index i predicted by the first image classification model to be trained.

[0180] Change 2: t i The location label of the detection box with index i predicted by the first image classification model to be trained.

[0181] Change 3: N cls N represents the number of labeled sample images in a batch of sample images input to the first image classification model to be trained; reg This indicates the number of reference boxes in these labeled sample images.

[0182] In formula [3], L u (A(x u ,s * ),q * ) represents the unsupervised loss of the first image classification model to be trained for unlabeled sample images (including the first pseudo-labeled sample image and the first augmented sample image). The specific calculation formula can be found in formula [2], which will not be repeated here.

[0183] λ u This represents a hyperparameter used to adjust the ratio between supervised learning loss and unsupervised learning loss, and its value can be, for example, a value within the range of [1, 2].

[0184] Minimizing the total loss is taken as the optimization objective of the first image classification model to be trained. The Stochastic Gradient Descent (SGD) algorithm is used to train the first image classification model to be trained with an initial learning rate of 0.01. A total of 180k training rounds can be performed. After 180k training rounds, it is considered that the iteration condition of the first image classification model is met. The first image classification model to be trained at this time is taken as the final model used to perform the object detection task.

[0185] Alternatively, the iteration condition for the first image classification model can also be "the first image classification model is trained and converged".

[0186] In one alternative implementation, the learning rate can be decayed to 0.1x at 120k and 160k iterations, respectively. Regarding the number of sample images input to the first image classification model to be trained, the number of images in a single batch can be 64.

[0187] The image classification model training method provided in this application uses augmented sample image data obtained by data augmentation based on difficult sample image data to train the first image classification model to be trained. This improves the imbalance between the number of labeled and unlabeled sample image data in the sample image dataset, allowing the first image classification model to be trained to be trained more fully on difficult sample image data, thereby improving the classification accuracy of the finally trained model. This solves the problem in the prior art where the imbalance between the number of labeled and unlabeled sample image data in the dataset easily leads to low classification accuracy of the model trained by semi-supervised learning.

[0188] In one alternative implementation, considering that the difficult sample image data has undergone data augmentation and thus been fully learned by the first image classification model to be trained, the so-called "difficulty" (i.e., the difficulty in accurately predicting by the first image classification model to be trained) of such difficult sample image data, along with the corresponding preset classification difficulty conditions, has become a historical situation. At this point, the sample image dataset may also contain unlabeled sample image data that is not currently identified as difficult sample image data but still possesses a certain degree of prediction difficulty. Such unlabeled sample image data may also require data augmentation.

[0189] Therefore, after one round of training of the first image classification model to be trained, the corresponding difficult sample image data can be adjusted so that the first image classification model to be trained can fully learn the unlabeled sample image data in the sample image dataset.

[0190] An alternative method for conditioning difficult sample image data is as follows:

[0191] If iterative training of the first image classification model to be trained includes: training the first image classification model to be trained for at least two rounds; then step 51 may specifically include the following sub-steps:

[0192] Sub-step 511: Discard the augmented sample image data (referred to as the first augmented sample image data for ease of description) obtained in the training of the first image classification model to be trained in the most recent round.

[0193] Sub-step 512: According to the unlabeled sample image data in the sample image dataset of the training of the first image classification model to be trained in the most recent round, when the unsupervised loss obtained satisfies the preset classification difficulty condition, the unlabeled sample image data is used as the target difficult sample image data.

[0194] The specific implementation of sub-step 512 can be found in the method for determining difficult sample image data in the aforementioned embodiments, and will not be repeated here.

[0195] The preset classification difficulty condition mentioned in sub-step 512 may be the same as or different from the preset classification difficulty condition mentioned in the previous embodiments.

[0196] Sub-step 513: Use a data augmentation method that matches the classification task of the first image classification model to be trained to perform data augmentation on the target hard sample image data, so as to obtain the data-augmented target hard sample image data as augmented sample image data (referred to as the second augmented sample image data).

[0197] The specific implementation of sub-step 513 can be referred to the data augmentation method for difficult sample image data in the aforementioned embodiments, and will not be repeated here.

[0198] Sub-step 514: Based on the second augmented sample image data and the sample image dataset, construct a dataset as the sample image dataset for the current training round, which can be called the augmented sample image dataset for the current training round. This augmented sample image dataset includes: the labeled sample image data, the unlabeled sample image data, and the second augmented sample image data.

[0199] By adopting the above adjustment method, the first image classification model to be trained can fully learn the unlabeled sample image data in the sample image dataset as the training rounds increase, thereby enabling the first image classification model to be trained to have better classification performance.

[0200] This application also provides an image classification method. The image classification method specifically includes the following steps.

[0201] Obtain the target image data to be classified;

[0202] The target image data is input into a trained image classification model (such as the first image classification model described above) to obtain the classification result of the image classification model for the target image data.

[0203] The image classification model is trained using the image classification model training method described above.

[0204] The image classification method provided in this application uses an image classification model trained on augmented sample image data obtained by data augmentation of difficult sample image data. This allows for more thorough training of the image classification model based on difficult sample image data, enabling the image classification model to achieve more accurate classification of difficult sample image data. Consequently, the overall classification results obtained by the image classification model are more accurate.

[0205] To address the issue of how to enhance the role of unlabeled sample image data in the training process and avoid imbalance in class prediction, based on the same inventive concept as the above embodiments of this application, Embodiment 3 of this application provides a sample image data enhancement device.

[0206] A schematic diagram of the specific structure of the device is shown below. Figure 6 As shown, it includes the following functional units:

[0207] The difficult sample selection unit 61 is used to select difficult sample image data that meets the preset classification difficulty conditions from the sample image dataset; the difficult sample image data is the unlabeled sample image data corresponding to the unsupervised loss that meets the preset classification difficulty conditions when the unlabeled sample image data in the sample image dataset is input into the first image classification model to be trained.

[0208] The augmentation unit 62 is used to perform data augmentation on the difficult sample image data by adopting a data augmentation method that matches the classification task of the first image classification model to be trained, so as to obtain an augmented sample image dataset.

[0209] Data augmentation of difficult sample image data can yield augmented sample image data. The collection of augmented sample image data is called the augmented sample image dataset.

[0210] The augmented sample image dataset is used to add to the sample image dataset to update the sample image dataset, and the updated sample image dataset is used for training the first image classification model.

[0211] Optionally, the unsupervised loss that satisfies the preset classification difficulty condition may include, but is not limited to, at least one of the following:

[0212] Unsupervised losses within the predetermined loss range; and / or,

[0213] The unlabeled sample image data is input into the first image classification model to be trained, and a preset number of unsupervised losses are obtained from the unsupervised losses; the preset number of unsupervised losses are greater than the other unsupervised losses obtained from the unsupervised losses excluding the preset number of unsupervised losses.

[0214] Optionally, the classification task of the first image classification model to be trained may include an object detection task; then, the difficult sample selection unit 61 may specifically be used for:

[0215] The target object in the difficult sample image data is scaled to obtain the augmented sample image dataset.

[0216] Optionally, the target object in the difficult sample image data may undergo scale transformation, including but not limited to:

[0217] The target objects in the difficult sample image data are enlarged to obtain an augmented sample image dataset, wherein the size of each augmented sample image in the augmented sample image dataset is the same as the size of the difficult sample image data.

[0218] And / or,

[0219] The size of the difficult sample image data is reduced according to a preset reduction factor. Multiple reduced-size difficult sample image data are then stitched together to obtain stitched augmented sample image data. The size of each augmented sample image data in the augmented sample image data set is consistent with that of the difficult sample image data.

[0220] Optionally, the augmented sample image dataset may be obtained by enlarging the target objects in the difficult sample image data.

[0221] The background region in the difficult sample image data is cropped, and the target object in the difficult sample image data is enlarged according to the aspect ratio of the difficult sample image data to obtain augmented sample image data.

[0222] Optionally, the unlabeled sample image data includes: unlabeled sample image data with target pseudo-labels. Therefore, the apparatus provided in this embodiment may further include: a pseudo-label generation unit. This pseudo-label generation unit is used for:

[0223] The unlabeled sample image data is input into a pre-trained second image classification model to obtain the first pseudo-label of the unlabeled sample image data.

[0224] Strong data augmentation is performed on the unlabeled sample image data and the corresponding first pseudo-label to obtain strongly data-augmented pseudo-labeled sample image data.

[0225] The pseudo-labeled sample image data after strong data augmentation is input into the pre-trained second image classification model to obtain the target pseudo-label of the unlabeled sample image data.

[0226] The sample image data enhancement device provided in Embodiment 3 of this application enhances the unlabeled sample image data corresponding to the preset classification difficulty condition from the sample image dataset. The enhanced difficult sample image data is then used for model training of the first image classification model to be trained. This allows for more thorough training of the first image classification model (such as a deep learning model) based on the difficult sample image data. Consequently, the model trained in the end can achieve higher classification accuracy for "classification difficulties" such as rare categories and hard-to-detect targets.

[0227] Meanwhile, the solution provided in this application focuses on difficult sample image data that requires data augmentation, rather than treating all unlabeled sample image data in the sample image dataset equally. Augmenting the difficult sample image data and then training the first image classification model based on the augmented sample image data significantly improves the accuracy of the first image classification model in recognizing "difficult-to-classify" targets. Therefore, the solution provided in this application is a preferred solution that can significantly improve the sample image dataset and leverage the role of unlabeled sample image data in the training process without consuming excessive processing resources.

[0228] Embodiment 3 of this application also provides a training device for an image classification model, which is used to improve the classification accuracy of the model obtained by training the model using a semi-supervised learning method.

[0229] The training apparatus for this image classification model includes, for example: Figure 7 The following functional units are shown:

[0230] The acquisition unit 71 acquires a sample image dataset; the sample image dataset includes labeled sample image data, unlabeled sample image data, and first augmented sample image data obtained based on the unlabeled sample image data.

[0231] The first augmented sample image data is obtained using the sample image data enhancement method described in the foregoing embodiments.

[0232] The data input unit 72 is used to input the sample image data into the first image classification model to be trained.

[0233] The parameter adjustment unit 73 is used to perform iterative training of the first image classification model to be trained. Based on the supervised loss and unsupervised loss of the first image classification model to be trained for the sample image dataset, the parameters of the first image classification model to be trained are adjusted until the iterative conditions of the first image classification model are met, and the trained image classification model is obtained.

[0234] Optionally, iterative training of the first image classification model to be trained includes: training the first image classification model to be trained for at least two rounds; then, the acquisition unit 71 can specifically be used for:

[0235] Discard the first augmented sample image data in the sample image dataset obtained in the most recent training round;

[0236] When the unlabeled sample image data from the sample image dataset in the most recent training round is input into the first image classification model to be trained, the unsupervised loss obtained satisfies the preset classification difficulty condition and the corresponding unlabeled sample image data is used as the target difficult sample image data.

[0237] The target difficult sample image data is augmented using a data augmentation method that matches the classification task of the first image classification model to be trained, to obtain the second augmented sample image data.

[0238] Based on the second augmented sample image data and the sample image dataset, a dataset is constructed as the sample image dataset for the current round of training; the constructed dataset includes: the labeled sample image data, the unlabeled sample image data, and the second augmented sample image data.

[0239] The training apparatus for the image classification model provided in this application uses augmented sample image data obtained by data augmentation based on difficult sample image data to train the first image classification model to be trained. This allows for more thorough training of the first image classification model (such as a deep learning model) based on difficult sample image data, thereby enabling the finally trained model to have high classification accuracy for "difficult to classify" targets such as rare categories and hard-to-detect targets.

[0240] This application also provides an image classification device to improve the classification accuracy of an image classification model obtained by training a model using a semi-supervised learning method.

[0241] The image classification device may specifically include:

[0242] The acquisition unit acquires the target image data to be classified.

[0243] The classification unit is used to input the target image data acquired by the acquisition unit into a trained image classification model (such as the aforementioned first image classification model) to obtain the classification result of the image classification model for the target image data.

[0244] The image classification model is trained using the image classification model training method provided in Embodiment 2 of this application.

[0245] The image classification device provided in this application embodiment uses an image classification model trained on augmented sample image data obtained by data augmentation of difficult sample image data. This allows for more thorough training of the image classification model based on difficult sample image data, enabling the image classification model to achieve more accurate classification of difficult sample image data. Consequently, the overall classification results obtained by the image classification model are more accurate.

[0246] Based on the same inventive concept as the foregoing embodiments of this application, this application also provides a computing device.

[0247] like Figure 8 As shown, the computing device includes a memory 81 and a processor 82. The memory 81 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. The memory 81 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0248] The processor 82, coupled to the memory 81, is used to execute the computer program stored in the memory 81 to perform a sample image data enhancement method as described in the foregoing embodiments, or a training method for an image classification model or an image classification method as described in Embodiment 2 of this application.

[0249] When the processor 82 executes the computer program to perform the sample image data augmentation method, it selects unlabeled sample image data corresponding to the unsupervised loss satisfying the preset classification difficulty condition from the sample image dataset for data augmentation, and uses the data-augmented difficult sample image data for model training of the first image classification model to be trained. This allows the training of the first image classification model (such as a deep learning model) based on the difficult sample image data to be more thorough, thereby enabling the finally trained model to have high classification accuracy for "classification difficulties" such as rare categories and hard-to-detect targets.

[0250] Furthermore, because the data augmentation is focused on difficult image samples rather than applying data augmentation to all unlabeled image samples in the dataset, augmenting the difficult image samples and then training the first image classification model based on the augmented image samples significantly improves the accuracy of the first image classification model in recognizing "difficult-to-classify" targets. Therefore, without consuming excessive processing resources, the sample image dataset can be significantly improved, and the role of unlabeled image samples in the training process can be fully utilized.

[0251] When the processor 82 executes the computer program to perform the training method of the image classification model, it uses augmented sample image data obtained by data augmentation based on difficult sample image data to train the first image classification model to be trained. This allows the training of the first image classification model (such as a deep learning model) based on difficult sample image data to be more thorough, thereby enabling the finally trained model to have high classification accuracy for "classification difficulties" such as rare categories and hard-to-detect targets.

[0252] When the processor 82 executes the computer program to perform the classification method, the image classification model used is trained on augmented sample image data obtained by data augmentation of difficult sample image data. Therefore, the training of the image classification model on the difficult sample image data is more sufficient, enabling the image classification model to achieve more accurate classification for difficult sample image data. This can improve the overall accuracy of the classification results obtained by the image classification model.

[0253] When the processor 82 executes the computer program in the memory 81, in addition to the functions described above, it can also perform other functions, as detailed in the descriptions of the preceding embodiments.

[0254] Furthermore, such as Figure 8 As shown, the computing device also includes other components such as a display 84, a communication component 83, a power supply component 85, and an audio component 86. Figure 8 The diagram only shows some components and does not mean that the computing device includes only these components. Figure 8 The components shown.

[0255] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the methods provided in the above embodiments.

[0256] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0257] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0258] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for enhancing sample image data, characterized in that, include: Select difficult sample images from the sample image dataset that meet the preset classification difficulty conditions; The difficult sample image data is the unlabeled sample image data from the sample image dataset input into the first image classification model to be trained, and the resulting unsupervised loss satisfies the unlabeled sample image data corresponding to the preset classification difficulty condition; wherein, the unsupervised loss satisfying the preset classification difficulty condition includes: unsupervised loss within a predetermined loss range; and / or, the unlabeled sample image data input into the first image classification model to be trained, and a preset number of unsupervised losses among the obtained unsupervised losses; the preset number of unsupervised losses are greater than other unsupervised losses among the obtained unsupervised losses other than the preset number of unsupervised losses; The difficult sample image data is augmented using a data augmentation method that matches the classification task of the first image classification model to be trained, resulting in an augmented sample image dataset. The classification task of the first image classification model to be trained includes object detection. Augmenting the difficult sample image data using a data augmentation method that matches the classification task of the first image classification model to be trained, resulting in an augmented sample image dataset, includes: scaling the target objects in the difficult sample image data to obtain the augmented sample image dataset. The augmented sample image dataset is used for model training of the first image classification model to be trained.

2. The method as described in claim 1, characterized in that, The target object in the difficult sample image data is scaled to obtain the augmented sample image dataset, which includes: The target objects in the difficult sample image data are enlarged to obtain an augmented sample image dataset, wherein the size of each augmented sample image in the augmented sample image dataset is the same as the size of the difficult sample image data. And / or, The size of the difficult sample image data is reduced according to a preset reduction factor. Multiple reduced-size difficult sample image data are then stitched together to obtain stitched augmented sample image data. The size of each augmented sample image data in the augmented sample image data set is consistent with that of the difficult sample image data.

3. The method as described in claim 2, characterized in that, The process of enlarging the target object in the difficult sample image data to obtain the augmented sample image dataset includes: The background region in the difficult sample image data is cropped, and the target object in the difficult sample image data is enlarged according to the aspect ratio of the difficult sample image data to obtain augmented sample image data.

4. The method according to any one of claims 1-3, characterized in that, The unlabeled sample image data includes: unlabeled sample image data with target pseudo-labels; The method for generating pseudo-labels from unlabeled sample image data with target pseudo-labels includes: The unlabeled sample image data is input into a pre-trained second image classification model to obtain the first pseudo-label of the unlabeled sample image data. Strong data augmentation is performed on the unlabeled sample image data and the corresponding first pseudo-label to obtain strongly data-augmented pseudo-labeled sample image data. The pseudo-labeled sample image data after strong data augmentation is input into the pre-trained second image classification model to obtain the target pseudo-label of the unlabeled sample image data.

5. A training method for an image classification model, characterized in that, include: Obtain a sample image dataset; the sample image dataset includes labeled sample image data, unlabeled sample image data, and first augmented sample image data obtained based on the unlabeled sample image data; Input the sample image dataset into the first image classification model to be trained; Iterative training is performed, and the parameters of the first image classification model to be trained are adjusted according to the supervised loss and unsupervised loss of the first image classification model to be trained for the sample image dataset; until the iteration conditions of the first image classification model are met, the trained first image classification model is obtained. The first augmented sample image data is obtained by the method described in any one of claims 1 to 4.

6. An image classification method, characterized in that, include: Obtain the target image data to be classified; The target image data is input into the trained first image classification model to obtain the image classification result of the target image data; The first image classification model is trained using the method described in claim 5.

7. An enhancement device for sample image data, characterized in that, include: The difficult sample selection unit selects difficult sample image data that meet the preset classification difficulty conditions from the sample image dataset; The difficult sample image data is the unlabeled sample image data from the sample image dataset input into the first image classification model to be trained, and the resulting unsupervised loss satisfies the unlabeled sample image data corresponding to the preset classification difficulty condition; wherein, the unsupervised loss satisfying the preset classification difficulty condition includes: unsupervised loss within a predetermined loss range; and / or, the unlabeled sample image data input into the first image classification model to be trained, and a preset number of unsupervised losses among the obtained unsupervised losses; the preset number of unsupervised losses are greater than other unsupervised losses among the obtained unsupervised losses other than the preset number of unsupervised losses; An augmentation unit is configured to perform data augmentation on the difficult sample image data using a data augmentation method that matches the classification task of the first image classification model to be trained, thereby obtaining an augmented sample image dataset; wherein, the classification task of the first image classification model to be trained includes an object detection task; performing data augmentation on the difficult sample image data using a data augmentation method that matches the classification task of the first image classification model to be trained, thereby obtaining an augmented sample image dataset, includes: performing scale transformation on the target objects in the difficult sample image data to obtain the augmented sample image dataset; The augmented sample image dataset is used for model training of the first image classification model to be trained.

8. An image classification device, characterized in that, include: The acquisition unit acquires the target image data to be classified. A classification unit is used to input the target image data into a trained first image classification model to obtain the classification result of the first image classification model for the target image data; The first image classification model is trained using the method described in claim 5.

9. A computing device, characterized in that, include: Memory and processor, among which, The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program stored in the memory for performing the method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, which, when executed by a computer, enables the implementation of the method described in any one of claims 1 to 6.

Images