Classifier training method, device, storage medium and electronic device

Abstract

The present application discloses a classifier training method, device, storage medium, and electronic device, wherein the method includes: obtaining at least one bounding box annotated sample and at least one image-level annotated sample, determining the label category corresponding to each bounding box annotated sample in the at least one bounding box annotated sample using the minimum class principle, generating counterfactual samples corresponding to each bounding box annotated sample based on the at least one bounding box annotated sample and the label category corresponding to each bounding box annotated sample, and training an image classifier using the at least one bounding box annotated sample, the counterfactual samples corresponding to each bounding box annotated sample, and at least one image-level annotated sample to obtain a target classifier. Using the present application, by using the minimum class principle to classify samples, and using a small number of bounding box annotated samples to generate counterfactual samples and then training the image classifier, the balance of the training process is ensured, and the accuracy of the target classifier is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a classifier training method, device, storage medium, and electronic device. Background Art

[0002] Using labeled image samples to train image classifiers allows the trained image classifier to label unlabeled images. Image classifiers trained based on image-level labeled samples have the problem of model understanding deviation caused by the co-occurrence of foreground and background. In particular, when generating a class activation map (CAM), the CAM cannot fall on the location of the target corresponding to the label, resulting in low accuracy of image classifiers in existing technologies. Summary of the Invention

[0003] The present invention provides a classifier training method, apparatus, storage medium, and electronic device. This method can classify samples using the minimum class principle and generate counterfactual samples using a small number of bounding box-labeled samples before training the image classifier. This ensures a balanced training process and improves the accuracy of the target classifier. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a classifier training method, the method comprising:

[0005] Obtain at least one bounding box annotation sample and at least one image-level annotation sample;

[0006] Determine the label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample using the minimum class principle;

[0007] Based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples, generating counterfactual samples corresponding to the bounding box labeled samples;

[0008] An image classifier is trained using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain a target classifier.

[0009] In a second aspect, an embodiment of the present application provides a classifier training device, the device comprising:

[0010] A sample acquisition module, configured to acquire at least one bounding box annotation sample and at least one image-level annotation sample;

[0011] a category determination module, configured to determine a label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample by adopting a minimum class principle;

[0012] A sample generation module, configured to generate counterfactual samples corresponding to the bounding box labeled samples based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples;

[0013] A training module is configured to train an image classifier using the at least one bounding box labeled sample, the counterfactual samples corresponding to the bounding box labeled samples, and the at least one image-level labeled sample to obtain a target classifier.

[0014] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0016] The beneficial effects of the technical solutions provided by some embodiments of the present application include at least:

[0017] In one or more embodiments of the present application, at least one bounding box annotated sample and at least one image-level annotated sample are obtained, and the minimum class principle is used to determine the label category corresponding to each bounding box annotated sample in the at least one bounding box annotated sample. Based on the at least one bounding box annotated sample and the label category corresponding to each bounding box annotated sample, counterfactual samples corresponding to each bounding box annotated sample are generated. The at least one bounding box annotated sample, the counterfactual samples corresponding to each bounding box annotated sample, and the at least one image-level annotated sample are used to train an image classifier to obtain a target classifier. By classifying the samples using the minimum class principle and using a small number of bounding box annotated samples to generate counterfactual samples and then training the image classifier, the balance of the training process is ensured and the accuracy of the target classifier is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is an example diagram of an image classifier training provided in an embodiment of the present application;

[0020] Figure 2 This is a flow chart of a classifier training method provided in an embodiment of the present application;

[0021] Figure 3 This is a flow chart of a classifier training method provided in an embodiment of the present application;

[0022] Figure 4 This is an example diagram of determining a tag category provided in an embodiment of the present application;

[0023] Figure 5 This is an example schematic diagram of generating a first counterfactual sample provided by an embodiment of the present application;

[0024] Figure 6 This is an example schematic diagram of generating a second counterfactual sample provided by an embodiment of the present application;

[0025] Figure 7 Schematic diagram of the structure of a classifier training device provided in an embodiment of the present application;

[0026] Figure 8 Schematic diagram of the structure of a classifier training device provided in an embodiment of the present application;

[0027] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The classifier training method provided in the embodiment of the present application can be implemented by a computer program and can be run on a classifier training device based on the von Neumann system. The computer program can be integrated into the application or run as an independent tool application. Figure 1 , an example diagram of image classifier training is provided for an embodiment of the present application. The classifier training device can use labeled image samples to train the image classifier to obtain a target classifier, wherein the image samples include bounding box labeled samples and image-level labeled samples. The image-level labeled samples are image samples that have been labeled, but the location of the target corresponding to the label in the image is unknown, for example Figure 1 The image level annotation sample in the image level annotation sample has a basketball and a basket in it. Labeling this image with the labels "basketball" and "basketball" generates an image level annotation sample. The bounding box annotation sample is an image sample that has been labeled and knows the position of the target corresponding to the label in the image. For example Figure 1 There is a basketball and a basket in the image corresponding to the bounding box labeled sample in , and the basketball is surrounded by a bounding box and labeled with the label "basketball", and the basket is surrounded by a bounding box and labeled with the label "basketball", and a bounding box labeled sample is generated. The classifier training device uses the acquired bounding box labeled samples to train the image classifier using the fully supervised learning (FSL) method, and uses the image-level labeled samples to train the image classifier using the weakly supervised learning (WSL) method, thereby obtaining a target classifier, and inputting unlabeled pictures into the target classifier, so that the target classifier can label the pictures. Among them, when the image classifier is trained using bounding box labeled samples based on strong supervised learning, the classifier training device can generate counterfactual samples based on the bounding box labeled samples. The counterfactual samples are pseudo samples generated by changing the bounding box labeled samples. The image classifier is trained using the counterfactual samples and the bounding box labeled samples using the strongly supervised learning method.

[0030] The classifier training method provided in this application is described in detail below with reference to specific embodiments.

[0031] See Figure 2 , provides a flow chart of a classifier training method according to an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps S101-S104.

[0032] S101, obtaining at least one bounding box annotation sample and at least one image-level annotation sample.

[0033] In one embodiment, the classifier training device can obtain at least one bounding box labeled sample and at least one image-level labeled sample. These bounding box labeled samples and image-level labeled samples can be manually labeled by relevant staff. It is understandable that the label granularity of the bounding box labeled samples is coarser than that of the image-level labeled samples. Therefore, it takes longer and costs more to obtain bounding box labeled samples than to obtain image-level labeled samples. Therefore, the bounding box labeled samples obtained by the classifier training device can be fewer than the image-level labeled samples. The classifier training device can obtain bounding box labeled samples and image-level labeled samples according to a preset ratio. For example, the number of image-level labeled samples can be four times that of the bounding box labeled samples.

[0034] S102: Determine the label category corresponding to each bounding box labeled sample in at least one bounding box labeled sample by adopting the minimum class principle.

[0035] In one embodiment, the minimum class principle is used to determine the label category corresponding to each bounding box labeled sample. For example, if the label "basketball" exists in the bounding box labeled sample, the label category corresponding to the bounding box labeled sample may be the label "basketball" exists. It is understandable that there may be multiple labels in a bounding box labeled sample. In order to make the number of samples in all label categories relatively balanced, the classifier training device may adopt the minimum class principle to obtain all labels in a bounding box labeled sample, and confirm the label with the least number of samples in the label category corresponding to the label as the main label of the bounding box labeled sample. Then the label category of the bounding box labeled sample is the main label exists. Since the number of samples in each label category is relatively balanced, the balance of the training process is guaranteed, and the accuracy of the target classifier is improved.

[0036] S103 : Based on at least one bounding box labeled sample and the label category corresponding to each bounding box labeled sample, generate a counterfactual sample corresponding to each bounding box labeled sample.

[0037] In one embodiment, the classifier training device can generate counterfactual samples corresponding to each bounding box labeled sample based on at least one acquired bounding box labeled sample and the label category corresponding to each bounding box labeled sample. For example, if sample a in at least one bounding box labeled sample has the label "basketball", then the label category of sample a is that the label "basketball" exists. The classifier training device can erase the bounding box portion corresponding to the label "basketball" in sample a, thereby obtaining a counterfactual sample for sample a, where the label category of the counterfactual sample is that the label "basketball" does not exist. It can be understood that using sample a and the counterfactual samples corresponding to sample a to train the image classifier can achieve class-limited biased correlation decoupling (C-BCD), so that the trained target classifier can not only label the image, but also locate the position of the target corresponding to the label in the image, so that the CAM of the image labeled by the target classifier can fall within the bounding box range corresponding to the label, thereby improving the accuracy of the target classifier.

[0038] S104 , training an image classifier using at least one bounding box labeled sample, counterfactual samples corresponding to each bounding box labeled sample, and at least one image-level labeled sample to obtain a target classifier.

[0039] In one embodiment, the classifier training device can use at least one bounding box labeled sample, the counterfactual sample corresponding to each bounding box labeled sample, and at least one image-level labeled sample to train the image classifier, thereby obtaining a target classifier. The image classifier can sequentially extract a training sample from at least one bounding box labeled sample or at least one image-level labeled sample according to a preset probability to train the image classifier. If the training sample is a bounding box labeled sample, the image classifier is trained using the training sample and the counterfactual sample corresponding to the training sample; if the training sample is an image-level labeled sample, the image classifier is directly trained using the training sample until the number of training times reaches the preset number to obtain the target classifier. The preset probability and the preset number of times can be the initial settings of the classifier training device, or can be set by relevant staff and saved in the classifier training device. For example, the preset probability can be set to 50%, and the preset number of times can be set to 5000 times.

[0040] In an embodiment of the present application, at least one bounding box annotated sample and at least one image-level annotated sample are obtained, and the minimum class principle is used to determine the label category corresponding to each bounding box annotated sample in the at least one bounding box annotated sample. Based on the at least one bounding box annotated sample and the label category corresponding to each bounding box annotated sample, counterfactual samples corresponding to each bounding box annotated sample are generated. The at least one bounding box annotated sample, the counterfactual samples corresponding to each bounding box annotated sample, and the at least one image-level annotated sample are used to train an image classifier to obtain a target classifier. By classifying the samples using the minimum class principle and using a small number of bounding box annotated samples to generate counterfactual samples and then training the image classifier, the balance of the training process is ensured, and the accuracy of the target classifier is improved.

[0041] See Figure 3 , provides a flow chart of a classifier training method according to an embodiment of the present application. Figure 3 As shown, the method of the embodiment of the present application may include the following steps S201-S208.

[0042] S201: Obtain at least one bounding box annotation sample and at least one image-level annotation sample.

[0043] In one embodiment, the classifier training device can obtain at least one bounding box labeled sample and at least one image-level labeled sample. These bounding box labeled samples and image-level labeled samples can be manually labeled by relevant staff. It is understandable that the label granularity of the bounding box labeled samples is coarser than that of the image-level labeled samples. Therefore, it takes longer and costs more to obtain bounding box labeled samples than to obtain image-level labeled samples. Therefore, the bounding box labeled samples obtained by the classifier training device can be fewer than the image-level labeled samples. The classifier training device can obtain bounding box labeled samples and image-level labeled samples according to a preset ratio. For example, the number of image-level labeled samples can be four times that of the bounding box labeled samples.

[0044] S202: Generate a strong label dataset corresponding to each strong label based on all strong labels in at least one bounding box annotation sample.

[0045] In one embodiment, it is understood that a bounding box annotated sample may contain multiple labels. To ensure a relatively balanced number of samples across all label categories, the classifier training device may employ the minimum class principle to determine the label category corresponding to each bounding box annotated sample and group them by label category. The classifier training device may generate a strong label dataset corresponding to each strong label based on all strong labels in at least one bounding box annotated sample, i.e., each strong label corresponds to a strong label dataset.

[0046] S203 , using the minimum class principle to save at least one bounding box labeled sample in the strong label data set corresponding to each strong label, and determining the label category corresponding to each bounding box labeled sample.

[0047] In one embodiment, the classifier training device can adopt the minimum class principle to obtain all strong labels in a bounding box labeled sample, confirm the strong label with the least number of samples in the strong label data set corresponding to the strong label as the main label of this bounding box labeled sample, and save this bounding box labeled sample in the strong label data set corresponding to the main label, and determine that the label category of this bounding box labeled sample is that the main label exists. It can be understood that the label category of all samples in the strong label data set corresponding to the main label is that the main label exists.

[0048] Optionally, the classifier training device can sequentially obtain samples from at least one bounding box labeled sample. For example, the classifier training device can obtain the first sample from at least one bounding box labeled sample, obtain all target labels in the first sample, and then obtain the number of samples of the strong label data set corresponding to each label in all target labels, and confirm the strong label data set with the minimum number of samples as the target strong label data set. The classifier training device can save the first sample in the target strong label data set, obtain the first label corresponding to the target strong label data set, and then the classifier training device can confirm that the label category of the first sample is that the first label exists.

[0049] Please also see Figure 4 , an example schematic diagram of label category determination is provided for an embodiment of the present application. In a first sample of at least one bounding box annotation sample, there are four target labels, namely "cat", "person", "balloon" and "traffic light". The classifier training device can obtain the number of samples of the strong label data set corresponding to each label in all target labels. The number of samples of the strong label data set corresponding to the labels "cat", "person", "balloon" and "traffic light" are 3, 15, 5 and 10 respectively. Since there is a minimum number of samples in the strong label data set corresponding to the label "cat", the classifier training device can confirm the strong label data set corresponding to the label "cat" as the target strong label data set. The label "cat" is the first label corresponding to the target strong label data set. The classifier training device can save the first sample in the target strong label data set, and the label category of the first sample is that the label "cat" exists.

[0050] S204 : Based on at least one bounding box labeled sample and the label category corresponding to each bounding box labeled sample, generate a counterfactual sample corresponding to each bounding box labeled sample.

[0051] In one embodiment, the classifier training device may generate counterfactual samples corresponding to each bounding box labeled sample based on the acquired at least one bounding box labeled sample and the label category corresponding to each bounding box labeled sample.

[0052] Optionally, the classifier training device can obtain a second sample from at least one bounding box labeled sample, and obtain the second label corresponding to the strong label data set of the second sample. The classifier training device can erase the bounding box portion corresponding to the second label in the second sample to obtain a first counterfactual sample. Since the target corresponding to the second label no longer exists in the first counterfactual sample, the second label does not exist in the first counterfactual sample, and the label category of the first counterfactual sample is that the second label does not exist. It can be understood that using the second sample and the first counterfactual sample to train the image classifier can achieve class-limited error association decoupling, so that the image classifier not only learns whether there is a second label in the image, but also learns to determine the position of the target corresponding to the second label in the image, so that the CAM of the image labeled by the target classifier can fall within the bounding box range corresponding to the label, thereby improving the accuracy of the target classifier.

[0053] Please also see Figure 5 , an example schematic diagram of the generation of a first counterfactual sample is provided for an embodiment of the present application. The second label corresponding to the strongly labeled data set to which the second sample belongs is "cat", and the label category of the second sample is the existence of the label "cat". It can be understood that using the second sample to train the image classifier can enable the image classifier to learn to label images with cats with the label "cat". The bounding box portion corresponding to the label "cat" in the second sample is erased to obtain the first counterfactual sample. There is no cat in the first counterfactual sample, so the label category of the first counterfactual sample is the absence of the label "cat". It can be understood that using the second sample and the first counterfactual sample to train the image classifier can not only enable the image classifier to learn to label images with cats with the label "cat", but also enable the image classifier to learn to locate the position of the cat in the image.

[0054] Optionally, a label may correspond to multiple bounding boxes in the bounding box annotation sample. For example, there may be multiple bounding boxes corresponding to the second label in the second sample. If the image classifier is trained directly with the second sample or the second sample and the first counterfactual sample, when the image classifier labels an image with multiple targets corresponding to the second label, it may only identify that there is only one target corresponding to the second label in the image, but cannot identify all targets corresponding to the second label. For example, there are two cats in the image. Although the image classifier labels the image with the label "cat", the image classifier will only confirm the position of one of the cats in the image. Therefore, if the number of bounding boxes corresponding to the second label in the second sample is greater than 1, the classifier training device can obtain a target bounding box in the bounding box corresponding to the second label, and then restore the target bounding box part in the first counterfactual sample to obtain the second counterfactual sample. Since the target corresponding to the second label appears again in the second counterfactual sample, the label category of the second counterfactual sample is that the second label exists. The target bounding box can be a random bounding box among the bounding boxes corresponding to the second label. It is understood that the image classifier needs to be trained multiple times, and the second sample may also train the image classifier multiple times. During each training, a random bounding box can be restored to obtain a second counterfactual sample, and the second counterfactual sample and the second sample are trained together with the image classifier. Training the image classifier with the second counterfactual sample can achieve class-limited instance separation enhancement (C-ISE), allowing the image classifier to further understand the meaning of the label corresponding to the target and simultaneously locate the targets corresponding to multiple labels in the image.

[0055] It is understandable that although the second sample contains multiple labels, the classifier training device only targets the second label and uses the second sample to generate a second counterfactual sample, limiting the label category range to the second label, thereby avoiding the occurrence of the long-tail coexistence effect during the training process of the image classifier.

[0056] Please also see Figure 6 , an example schematic diagram of generating a second counterfactual sample is provided for an embodiment of the present application. Since there are two bounding boxes corresponding to the second label "cat" in the second sample, the classifier training device can randomly obtain a target bounding box and restore the target bounding box part in the first counterfactual sample to obtain the second counterfactual sample. Since there is a cat again in the second counterfactual sample, the label category of the second counterfactual sample is that the label "cat" exists.

[0057] S205: Based on all weak labels in at least one image-level annotated sample, generate a weak label dataset corresponding to each weak label, use the minimum class principle to save at least one image-level annotated sample in the weak label dataset corresponding to each weak label, and determine the label category corresponding to each image-level annotated sample.

[0058] In one embodiment, in addition to at least one bounding box annotated sample being classified using the minimum class principle, the classifier training device may also classify at least one image-level annotated sample using the minimum class principle. Similar to steps S202-S203, the classifier training device may generate a weak label dataset corresponding to each weak label based on all weak labels of the at least one image-level annotated sample, and then, using the minimum class principle, store the at least one image-level annotated sample in the weak label dataset corresponding to each weak label, and determine the label category corresponding to each image-level annotated sample.

[0059] S206 , obtaining training samples from all strong-label data sets and all weak-label data sets according to preset probabilities.

[0060] In one embodiment, the classifier training device can obtain a training sample from all strong label data sets and all weak label data sets according to a preset probability. The preset probability can be an initial setting of the classifier training device, or it can be set by relevant staff and saved in the classifier training device. For example, the preset probability can be 50%.

[0061] Optionally, if the preset probability is 50%, the classifier training device can obtain a random number in the range of 0-1. If the random number is greater than 0.5, the classifier training device can obtain a training sample from all bounding box labeled samples contained in all strong label data sets; if the random number is less than or equal to 0.5, the classifier training device can obtain a training sample from all image-level labeled samples contained in all weak label data sets.

[0062] S207: Determine whether the training sample belongs to a strong label dataset.

[0063] In one embodiment, the classifier training device trains the image classifier using bounding box-labeled samples based on a strongly supervised learning approach, and using image-level labeled samples based on a weakly supervised learning approach. Therefore, after the classifier training device obtains a training sample, it can determine whether the training sample belongs to a strongly labeled dataset. If so, it executes S208; if not, it executes S209.

[0064] S208 , based on strongly supervised learning, the image classifier is trained using the training samples and the counterfactual samples corresponding to the training samples.

[0065] In one embodiment, if the training samples belong to a strongly labeled dataset, that is, the training samples are bounding box labeled samples, the classifier training device can train the image classifier based on strongly supervised learning using the training samples and counterfactual samples corresponding to the training samples.

[0066] S209, based on weakly supervised learning, the image classifier is trained using the training samples.

[0067] In one embodiment, if the training samples belong to a weakly labeled dataset, that is, the training samples are image-level labeled samples, the classifier training device can use the training samples to train the image classifier based on weakly supervised learning.

[0068] S210: Determine whether the number of training times for the image classifier reaches a preset number.

[0069] In one embodiment, after the classifier training device completes a training session, it can determine whether the number of training cycles for the image classifier has reached a preset test. If so, S211 is executed; if not, S206 is continued. The preset number of cycles can be an initial setting of the classifier training device, or can be set by relevant personnel and stored in the classifier training device. For example, the preset number of cycles can be 10,000.

[0070] S211, obtain the target classifier.

[0071] In one embodiment, the classifier training device may obtain a target classifier after training the image classifier for a preset number of times.

[0072] In an embodiment of the present application, at least one bounding box labeled sample and at least one image-level labeled sample are obtained, and the bounding box labeled samples and image-level labeled samples are classified using the minimum class principle, and label categories are determined so that the number of samples corresponding to all label categories is relatively balanced, thereby ensuring balance during the image classifier training process. Based on the at least one bounding box labeled sample and the label category corresponding to each bounding box labeled sample, counterfactual samples corresponding to each bounding box labeled sample are generated, achieving class-defined error association decoupling and class-defined instance independence enhancement, so that the image classifier understands the meaning of the label corresponding to the target and simultaneously locates multiple label corresponding targets in the image. In addition, the label category is limited in the process of generating counterfactual samples, so that the long-tail coexistence effect is avoided during the image classifier training process. Training samples are obtained from a strong label dataset and a weak label dataset using a preset probability to train the image classifier. If the training sample belongs to the strong label dataset, the image classifier is trained based on strong supervised learning using the training sample and the counterfactual sample corresponding to the training sample. If the training sample belongs to the weak label dataset, the image classifier is trained based on weak supervised learning using the training sample. When the number of training times reaches a preset number, the target classifier is obtained. By adopting the minimum class principle to classify samples and using a small number of bounding box annotated samples to generate counterfactual samples and then training the image classifier, the balance of the training process is ensured and the accuracy of the target classifier is improved.

[0073] The following will be combined with the Figure 7 -Attached Figure 8 , the classifier training device provided in the embodiment of the present application is introduced in detail. Figure 7 -Attached Figure 8 The classifier training device in this application is used to perform Figure 2 and Figure 3 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2 and Figure 3 The embodiment shown.

[0074] See Figure 7 , which shows a schematic diagram of the structure of a classifier training device provided by an exemplary embodiment of the present application. The classifier training device can be implemented as all or part of the device through software, hardware, or a combination of both. The device 1 includes a sample acquisition module 11, a category determination module 12, a sample generation module 13, and a training module 14.

[0075] A sample acquisition module 11 is configured to acquire at least one bounding box labeled sample and at least one image-level labeled sample;

[0076] A category determination module 12 is configured to determine a label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample by adopting a minimum category principle;

[0077] A sample generation module 13 is configured to generate counterfactual samples corresponding to the bounding box labeled samples based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples;

[0078] The training module 14 is configured to train an image classifier using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain a target classifier.

[0079] In this embodiment, at least one bounding box annotated sample and at least one image-level annotated sample are obtained. The minimum class principle is used to determine the label category corresponding to each bounding box annotated sample in the at least one bounding box annotated sample. Based on the at least one bounding box annotated sample and the label category corresponding to each bounding box annotated sample, counterfactual samples corresponding to each bounding box annotated sample are generated. The at least one bounding box annotated sample, the counterfactual samples corresponding to each bounding box annotated sample, and the at least one image-level annotated sample are used to train an image classifier to obtain a target classifier. By classifying samples using the minimum class principle and using a small number of bounding box annotated samples to generate counterfactual samples for training the image classifier, a balanced training process is ensured, thereby improving the accuracy of the target classifier.

[0080] See Figure 8 , which shows a schematic diagram of the structure of a classifier training device provided by an exemplary embodiment of the present application. The classifier training device can be implemented as all or part of the device through software, hardware, or a combination of both. The device 1 includes a sample acquisition module 11, a category determination module 12, a sample generation module 13, an image-level sample classification module 15, and a training module 14.

[0081] A sample acquisition module 11 is configured to acquire at least one bounding box labeled sample and at least one image-level labeled sample;

[0082] A category determination module 12 is configured to determine a label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample by adopting a minimum category principle;

[0083] Optionally, the category determination module 12 is specifically configured to generate a strong label dataset corresponding to each strong label based on all strong labels in the at least one bounding box annotation sample;

[0084] The at least one bounding box labeled sample is respectively stored in the strong label data set corresponding to each strong label by adopting the minimum class principle, and the label category corresponding to each bounding box labeled sample is determined.

[0085] Optionally, the category determination module 12 is specifically configured to obtain a first sample from the at least one bounding box labeled sample, and obtain all target labels from the first sample;

[0086] Obtain the number of samples of the strong label data set corresponding to each label among all the target labels, and identify the strong label data set with the smallest number of samples as the target strong label data set;

[0087] A first label corresponding to the target strong label data set is obtained, and the first sample is saved in the target strong label data set, where the label category of the first sample is that the first label exists.

[0088] A sample generation module 13 is configured to generate counterfactual samples corresponding to the bounding box labeled samples based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples;

[0089] Optionally, the sample generation module 13 is specifically configured to obtain a second sample from the at least one bounding box labeled sample, and obtain a second label corresponding to the strong label dataset to which the second sample belongs;

[0090] The bounding box portion corresponding to the second label in the second sample is erased to obtain a first counterfactual sample, where the label category of the first counterfactual sample is that the second label does not exist.

[0091] Optionally, the sample generating module 13 is specifically configured to obtain a target bounding box among the bounding boxes corresponding to the second label if the number of bounding boxes corresponding to the second label in the second sample is greater than 1;

[0092] The target bounding box portion in the first counterfactual sample is restored to obtain a second counterfactual sample, where the label category of the second counterfactual sample is that the second label exists.

[0093] An image-level sample classification module 15 is configured to generate a weak label dataset corresponding to each weak label based on all weak labels in the at least one image-level annotated sample;

[0094] The at least one image-level annotated sample is respectively stored in a weak label data set corresponding to each weak label using the minimum class principle, and a label category corresponding to each image-level annotated sample is determined.

[0095] The training module 14 is configured to train an image classifier using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain a target classifier.

[0096] Optionally, the training module 14 is specifically configured to obtain training samples from all strong label data sets and all weak label data sets according to a preset probability;

[0097] If the training sample belongs to all the strongly labeled data sets, training the image classifier using the training sample and the counterfactual sample corresponding to the training sample;

[0098] If the training sample belongs to all the weakly labeled data sets, the image classifier is trained using the training sample;

[0099] If the number of training times for the image classifier reaches a preset number, a target classifier is obtained.

[0100] Optionally, the training module 14 is specifically configured to obtain a random number in the range of 0 to 1;

[0101] If the random number is greater than 0.5, a training sample is obtained from all the strong label data sets;

[0102] If the random number is less than or equal to 0.5, a training sample is obtained from all the weakly labeled data sets.

[0103] Optionally, the training module 14 is specifically configured to train an image classifier based on strong supervised learning using the training sample and a counterfactual sample corresponding to the training sample if the training sample belongs to all the strongly labeled data sets.

[0104] Optionally, the training module 14 is specifically configured to train the image classifier using the training sample based on weakly supervised learning if the training sample belongs to all the weakly labeled data sets.

[0105] In this embodiment, at least one bounding box annotated sample and at least one image-level annotated sample are obtained. The bounding box annotated sample and the image-level annotated sample are classified using the minimum class principle, and label categories are determined. This ensures a relatively balanced number of samples corresponding to all label categories, thus ensuring balance during image classifier training. Based on the at least one bounding box annotated sample and the label category corresponding to each bounding box annotated sample, counterfactual samples corresponding to each bounding box annotated sample are generated. This achieves class-defined error association decoupling and class-defined instance independence enhancement, enabling the image classifier to understand the meaning of label-correlated objects and simultaneously locate multiple label-correlated objects in the image. Furthermore, label categories are limited during the generation of counterfactual samples, thus avoiding the long-tail coexistence effect during image classifier training. Training samples are obtained from a strongly labeled dataset and a weakly labeled dataset with preset probabilities to train the image classifier. If the training sample belongs to the strongly labeled dataset, the image classifier is trained using the training sample and the corresponding counterfactual samples based on strong supervised learning. If the training sample belongs to the weakly labeled dataset, the image classifier is trained using the training sample based on weak supervised learning. When the training cycle reaches a preset number of times, the target classifier is obtained. By adopting the minimum class principle to classify samples and using a small number of bounding box annotated samples to generate counterfactual samples and then training the image classifier, the balance of the training process is ensured and the accuracy of the target classifier is improved.

[0106] It should be noted that the classifier training device provided in the above embodiment only uses the division of the above functional modules as an example when executing the classifier training method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the classifier training device provided in the above embodiment and the classifier training method embodiment are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0107] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0108] The present application also provides a computer storage medium that can store multiple instructions, which are suitable for being loaded and executed by a processor as described above. Figure 2-Figure 6 The classifier training method of the embodiment shown in the figure can be found in the specific execution process. Figure 2-Figure 6 The detailed description of the illustrated embodiment will not be repeated here.

[0109] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 2-Figure 6The classifier training method of the embodiment shown in the figure can be found in the specific execution process. Figure 2-Figure 6 The detailed description of the illustrated embodiment will not be repeated here.

[0110] Please refer to Figure 9 , which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0111] The processor 110 may include one or more processing cores. The processor 110 utilizes various interfaces and circuits to connect various components within the electronic device. It executes instructions, programs, code sets, or instruction sets stored in the memory 120, as well as accesses data stored in the memory 120, to perform various functions and process data for the terminal 100. Optionally, the processor 110 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interfaces, and applications; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 110 and may be implemented separately via a communications chip.

[0112] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an iOS system developed by Apple, including a system deeply developed based on the iOS system or other systems.

[0113] The memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve better operating results, the operating system allocates corresponding system resources to different third-party applications. However, the requirements for system resources in different application scenarios in the same third-party application are also different. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and the third-party application are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0114] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0115] The input device 130 is used to receive input commands or data and includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch-sensitive device. The output device 140 is used to output commands or data and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 may be combined, and the input device 130 and the output device 140 may be a touch-sensitive display.

[0116] The touch display screen can be designed as a full screen, a curved screen or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the present embodiment.

[0117] In addition, those skilled in the art will understand that the structures of the electronic devices shown in the above figures do not limit the electronic devices. The electronic devices may include more or fewer components than shown, or may combine certain components, or arrange the components differently. For example, the electronic devices may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, and other components, which will not be described in detail here.

[0118] exist Figure 9 In the electronic device shown, the processor 110 may be configured to call a classifier training application stored in the memory 120 and specifically perform the following operations:

[0119] Obtain at least one bounding box annotation sample and at least one image-level annotation sample;

[0120] Determine the label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample using the minimum class principle;

[0121] Based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples, generating counterfactual samples corresponding to the bounding box labeled samples;

[0122] An image classifier is trained using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain a target classifier.

[0123] In one embodiment, when the processor 110 determines the label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample using the minimum class principle, the processor 110 specifically performs the following operations:

[0124] Based on all strong labels in the at least one bounding box annotation sample, generate a strong label dataset corresponding to each strong label;

[0125] The at least one bounding box labeled sample is respectively stored in the strong label data set corresponding to each strong label by adopting the minimum class principle, and the label category corresponding to each bounding box labeled sample is determined.

[0126] In one embodiment, when the processor 110 executes the minimum class principle to store the at least one bounding box labeled sample in the strong label dataset corresponding to each strong label and determines the label category corresponding to each bounding box labeled sample, the processor 110 specifically performs the following operations:

[0127] Obtain a first sample from the at least one bounding box labeled sample, and obtain all target labels in the first sample;

[0128] Obtain the number of samples of the strong label data set corresponding to each label among all the target labels, and identify the strong label data set with the smallest number of samples as the target strong label data set;

[0129] A first label corresponding to the target strong label data set is obtained, and the first sample is saved in the target strong label data set, where the label category of the first sample is that the first label exists.

[0130] In one embodiment, when the processor 110 generates counterfactual samples corresponding to the bounding box labeled samples based on the at least one bounding box labeled sample and the label category corresponding to the bounding box labeled samples, the processor 110 specifically performs the following operations:

[0131] Obtaining a second sample from the at least one bounding box labeled sample, and obtaining a second label corresponding to the strong label dataset to which the second sample belongs;

[0132] The bounding box portion corresponding to the second label in the second sample is erased to obtain a first counterfactual sample, where the label category of the first counterfactual sample is that the second label does not exist.

[0133] In one embodiment, after erasing the bounding box portion corresponding to the second label in the second sample to obtain the first counterfactual sample, the processor 110 further performs the following operations:

[0134] If the number of bounding boxes corresponding to the second label in the second sample is greater than 1, obtaining a target bounding box among the bounding boxes corresponding to the second label;

[0135] The target bounding box portion in the first counterfactual sample is restored to obtain a second counterfactual sample, where the label category of the second counterfactual sample is that the second label exists.

[0136] In one embodiment, before training the image classifier using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain the target classifier, the processor 110 further performs the following operations:

[0137] Based on all weak labels in the at least one image-level annotated sample, generate a weak label dataset corresponding to each weak label;

[0138] The at least one image-level annotated sample is respectively stored in a weak label data set corresponding to each weak label using the minimum class principle, and a label category corresponding to each image-level annotated sample is determined.

[0139] In one embodiment, when the processor 110 trains an image classifier using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain a target classifier, the processor 110 specifically performs the following operations:

[0140] Obtain training samples from all strong label datasets and all weak label datasets according to preset probabilities;

[0141] If the training sample belongs to all the strongly labeled data sets, training the image classifier using the training sample and the counterfactual sample corresponding to the training sample;

[0142] If the training sample belongs to all the weakly labeled data sets, the image classifier is trained using the training sample;

[0143] If the number of training times for the image classifier reaches a preset number, a target classifier is obtained.

[0144] In one embodiment, when the processor 110 acquires training samples from all strong-label data sets and all weak-label data sets according to a preset probability, the processor 110 specifically performs the following operations:

[0145] Get a random number between 0 and 1;

[0146] If the random number is greater than 0.5, a training sample is obtained from all the strong label data sets;

[0147] If the random number is less than or equal to 0.5, a training sample is obtained from all the weakly labeled data sets.

[0148] In one embodiment, when the processor 110 executes the following operations when training an image classifier using the training sample and a counterfactual sample corresponding to the training sample if the training sample belongs to all the strongly labeled data sets:

[0149] If the training sample belongs to all the strongly labeled data sets, the image classifier is trained based on strongly supervised learning using the training sample and the counterfactual sample corresponding to the training sample.

[0150] In one embodiment, when executing the step of training the image classifier using the training sample if the training sample belongs to all weakly labeled data sets, the processor 110 specifically performs the following operations:

[0151] If the training sample belongs to all the weakly labeled data sets, the image classifier is trained using the training sample based on weakly supervised learning.

[0152] In this embodiment, at least one bounding box annotated sample and at least one image-level annotated sample are obtained. The bounding box annotated sample and the image-level annotated sample are classified using the minimum class principle, and label categories are determined. This ensures a relatively balanced number of samples corresponding to all label categories, thus ensuring balance during image classifier training. Based on the at least one bounding box annotated sample and the label category corresponding to each bounding box annotated sample, counterfactual samples corresponding to each bounding box annotated sample are generated. This achieves class-defined error association decoupling and class-defined instance independence enhancement, enabling the image classifier to understand the meaning of label-correlated objects and simultaneously locate multiple label-correlated objects in the image. Furthermore, label categories are limited during the generation of counterfactual samples, thus avoiding the long-tail coexistence effect during image classifier training. Training samples are obtained from a strongly labeled dataset and a weakly labeled dataset with preset probabilities to train the image classifier. If the training sample belongs to the strongly labeled dataset, the image classifier is trained using the training sample and the corresponding counterfactual samples based on strong supervised learning. If the training sample belongs to the weakly labeled dataset, the image classifier is trained using the training sample based on weak supervised learning. When the training cycle reaches a preset number of times, the target classifier is obtained. By adopting the minimum class principle to classify samples and using a small number of bounding box annotated samples to generate counterfactual samples and then training the image classifier, the balance of the training process is ensured and the accuracy of the target classifier is improved.

[0153] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0154] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A classifier training method, characterized in that: The method comprises: Obtain at least one bounding box annotation sample and at least one image-level annotation sample; Determine the label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample using the minimum class principle, wherein the minimum class principle is to determine the label category with the smallest number of samples in the bounding box labeled samples as the label category corresponding to the bounding box labeled sample; Based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples, generating counterfactual samples corresponding to the bounding box labeled samples; The at least one bounding box labeled sample and the counterfactual samples corresponding to each bounding box labeled sample are used to train an image classifier using a strong supervised learning method, and the at least one image-level labeled sample is used to train the image classifier using a weak supervised learning method to obtain a target classifier; The generating, based on the at least one bounding box labeled sample and the label category corresponding to each bounding box labeled sample, counterfactual samples corresponding to each bounding box labeled sample includes: Obtaining a second sample from the at least one bounding box labeled sample, and obtaining a second label corresponding to the strong label dataset to which the second sample belongs; The bounding box portion corresponding to the second label in the second sample is erased to obtain a first counterfactual sample, where the label category of the first counterfactual sample is that the second label does not exist.

2. The method according to claim 1, characterized in that The determining the label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample by using the minimum class principle includes: Based on all strong labels in the at least one bounding box annotation sample, generate a strong label dataset corresponding to each strong label; The at least one bounding box labeled sample is respectively stored in the strong label data set corresponding to each strong label by adopting the minimum class principle, and the label category corresponding to each bounding box labeled sample is determined.

3. The method according to claim 2, characterized in that The adopting the minimum class principle to store the at least one bounding box labeled sample in the strong label data set corresponding to each strong label, and determining the label category corresponding to each bounding box labeled sample, includes: Obtain a first sample from the at least one bounding box labeled sample, and obtain all target labels in the first sample; Obtain the number of samples of the strong label data set corresponding to each label among all the target labels, and identify the strong label data set with the smallest number of samples as the target strong label data set; A first label corresponding to the target strong label data set is obtained, and the first sample is saved in the target strong label data set, where the label category of the first sample is that the first label exists.

4. The method according to claim 1, wherein After erasing the bounding box portion corresponding to the second label in the second sample to obtain the first counterfactual sample, the method further includes: If the number of bounding boxes corresponding to the second label in the second sample is greater than 1, obtaining a target bounding box among the bounding boxes corresponding to the second label; The target bounding box portion in the first counterfactual sample is restored to obtain a second counterfactual sample, where the label category of the second counterfactual sample is that the second label exists.

5. The method according to claim 2, characterized in that Before the image classifier is trained using the at least one bounding box labeled sample, the counterfactual samples corresponding to the bounding box labeled samples, and the at least one image-level labeled sample to obtain a target classifier, the method further includes: Based on all weak labels in the at least one image-level annotated sample, generate a weak label dataset corresponding to each weak label; The at least one image-level annotated sample is respectively stored in a weak label data set corresponding to each weak label using the minimum class principle, and the label category corresponding to each image-level annotated sample is determined.

6. The method according to claim 5, characterized in that The method of training an image classifier using the at least one bounding box labeled sample, the counterfactual samples corresponding to each bounding box labeled sample, and the at least one image-level labeled sample to obtain a target classifier includes: Obtain training samples from all strong label datasets and all weak label datasets according to preset probabilities; If the training sample belongs to all the strongly labeled data sets, training the image classifier using the training sample and the counterfactual sample corresponding to the training sample; If the training sample belongs to all the weakly labeled data sets, the image classifier is trained using the training sample; If the number of training times for the image classifier reaches a preset number, a target classifier is obtained.

7. The method according to claim 6, characterized in that The step of obtaining training samples from all strong-label datasets and all weak-label datasets according to a preset probability includes: Get a random number between 0 and 1; If the random number is greater than 0.5, a training sample is obtained from all the strong label data sets; If the random number is less than or equal to 0.5, a training sample is obtained from all the weakly labeled data sets.

8. The method according to claim 6, characterized in that If the training sample belongs to all the strongly labeled data sets, training the image classifier using the training sample and the counterfactual sample corresponding to the training sample includes: If the training sample belongs to all the strongly labeled data sets, the image classifier is trained based on strongly supervised learning using the training sample and the counterfactual sample corresponding to the training sample.

9. The method according to claim 6, characterized in that If the training sample belongs to all the weakly labeled data sets, training the image classifier using the training sample includes: If the training sample belongs to all the weakly labeled data sets, the image classifier is trained using the training sample based on weakly supervised learning.

10. A classifier training device, characterized in that: The device comprises: A sample acquisition module, configured to acquire at least one bounding box annotation sample and at least one image-level annotation sample; a category determination module, configured to determine a label category corresponding to each bounding box labeled sample in the at least one bounding box labeled sample using a minimum category principle, wherein the minimum category principle is to determine the label category with the smallest number of samples in the bounding box labeled sample as the label category corresponding to the bounding box labeled sample; A sample generation module, configured to generate counterfactual samples corresponding to the bounding box labeled samples based on the at least one bounding box labeled sample and the label categories corresponding to the bounding box labeled samples; a training module configured to train an image classifier using the at least one bounding box annotated sample and the counterfactual samples corresponding to each bounding box annotated sample using a strong supervised learning method, and to train the image classifier using the at least one image-level annotated sample using a weak supervised learning method, to obtain a target classifier; The sample generation module is specifically configured to obtain a second sample from the at least one bounding box labeled sample, and obtain a second label corresponding to the strong label dataset to which the second sample belongs; The bounding box portion corresponding to the second label in the second sample is erased to obtain a first counterfactual sample, where the label category of the first counterfactual sample is that the second label does not exist.

11. A computer storage medium, characterized in that The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 9.

Images