An automatic labeling method and device, electronic equipment and medium

Abstract

The application provides an automatic labeling method, device, electronic equipment and medium. The automatic labeling method can predict a first classification result of each sample data through a target classification model. A labeling judgment module installed after a main network of the target classification model obtains a confidence degree of the target classification model predicting that the sample data is a first target category label and a second classification result of at least part of the sample data. Whether the target classification model meets a stop training condition is judged based on the second classification result. If not, the target classification model is updated based on the confidence degree of the first target category label of each sample data. If yes, the classification labeling is stopped, and a trained target classification model and labeled sample data set are obtained. Thus, the problem of insufficient data during training of the classification model is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to an automatic labeling method and device, an electronic device and a medium. BACKGROUND

[0002] With the rise of artificial intelligence, deep learning has made great achievements in image processing, speech processing and other aspects, and is widely used in many fields. In image processing, speech processing and other processing processes, one of the most important ideas is classification, that is, to identify the category of images, speech, footprints, etc. The classification method of deep learning is based on supervised learning method, and model training requires a large amount of labeled data. Then use a large amount of labeled data to train the model. However, data labeling often requires a huge amount of manpower, material resources and time, thereby greatly affecting the cost of project development. SUMMARY

[0003] Therefore, the purpose of the present application is to provide an automatic labeling method, device, electronic device and medium, which can realize model training while automatically labeling sample data by the model, and solve the problem of insufficient sample data in the model training process in a specific scenario.

[0004] An automatic labeling method provided by an embodiment of the present application comprises:

[0005] The data features of each sample data in the sample data set are extracted by the backbone network of the target classification model, and the data features are processed by the target classification model to obtain the first classification result of each sample data. The first classification result includes a first target category label and a first classification probability corresponding to the first target category label. The target classification model is obtained by preliminary training based on a first data set manually labeled;

[0006] The data features of each sample data are processed by the labeling judgment module installed after the backbone network of the target classification model to obtain the confidence of the target classification model predicting that the sample data is the first target category label;

[0007] The data features of at least part of the sample data are processed by the labeling judgment module to obtain a second classification result corresponding to the at least part of the sample data. The second classification result includes a second target category label and a second classification probability corresponding to the second target category label;

[0008] Whether the target classification model meets the stop training condition is determined based on the second classification result of the at least part of the sample data;

[0009] If not, the target classification model is updated based on the confidence of the first target category label of each sample data, and the next round of classification labeling is performed on the sample data set using the updated target classification model until the target classification model meets the stopping training condition.

[0010] If yes, the classification labeling is stopped, the trained target classification model is obtained, and the labeled sample data set is obtained using the first classification result of the target classification model.

[0011] In some embodiments, in the automatic labeling method, the labeling judgment module added after the backbone network of the target classification model processes the data features of each sample data to obtain the confidence of the target classification model predicting the sample data as the first target category label, including:

[0012] The data features of each sample data are input into the confidence calculation function of the labeling judgment module added after the backbone network of the target classification model.

[0013] The confidence calculation function of the labeling judgment module respectively transforms the data features of each sample into a confidence between 0 and 1.

[0014] In some embodiments, in the automatic labeling method, before the labeling judgment module processes the data features of at least part of the sample data again, the method further includes:

[0015] The negative sample data meeting the preset negative sample condition in confidence is filtered out from all sample data, and the data features of the negative sample data are obtained.

[0016] Correspondingly, the labeling judgment module processes the data features of at least part of the sample data again to obtain a second classification result corresponding to the at least part of the sample data, including:

[0017] The labeling judgment module processes the data features of the negative sample data again to obtain the second classification result of each negative sample data.

[0018] In some embodiments, in the automatic labeling method, the determination of whether the target classification model meets the stopping training condition based on the second classification result of at least part of the sample data includes:

[0019] The second category label of each negative sample data and the existing category label of the negative sample data are determined to be consistent to obtain the judgment result of each negative sample data; wherein the existing category label of the negative sample data in the first round of classification labeling is the first target category label of this round; the existing category label of the negative sample data in other rounds of classification labeling is the first target category label of the previous round.

[0020] If the determination results of each negative sample data are consistent, the classification result of the target classification model meets the preset stopping training condition.

[0021] If the determination results of at least one negative sample data are inconsistent, the classification result of the target classification model does not meet the preset stopping training condition.

[0022] In some embodiments, when the determination results of at least one negative sample data are inconsistent, the automatic labeling method further comprises:

[0023] comparing the first classification probability of the existing category label of the negative sample data and the second classification probability corresponding to the second target category label;

[0024] If the first classification probability is less than the second classification probability, and the difference between the first classification probability and the second classification probability exceeds a preset threshold, the second classification result is taken as the first classification result of the negative sample data in the current training, and the next round of classification labeling is performed based on the updated first classification result.

[0025] Otherwise, the existing category label of the negative sample data and the first classification probability are taken as the first classification result of the negative sample data in the current training, and the next round of classification labeling is performed based on the updated first classification result.

[0026] In some embodiments, the automatic labeling method further comprises:

[0027] Based on the confidence of the first target category label of each sample data input into the target classification model and a preset positive and negative sample screening condition, positive sample data and negative sample data in the sample data set are screened out.

[0028] Based on the confidence of the positive sample data and the confidence of the negative sample data, the target classification model is updated.

[0029] In some embodiments, the automatic labeling method further comprises:

[0030] The mean value of the confidence of the positive sample data and the mean value of the confidence of the negative sample data are calculated.

[0031] Based on the mean value of the confidence of the positive sample data, the mean value of the confidence of the negative sample data, a preset negative sample suppression constant, and a preset loss function calculation rule, a loss function calculation result is determined.

[0032] Based on the loss function calculation result, the target classification model is updated.

[0033] In some embodiments, an automatic labeling device is also provided, and the device comprises:

[0034] an extraction module configured to extract data features of each sample data in a sample data set by a backbone network of a target classification model, and obtain a first classification result of each sample data by processing the data features by the target classification model; wherein the first classification result comprises a first target class label and a first classification probability corresponding to the first target class label; and the target classification model is obtained by preliminary training based on a first data set labeled by human beings;

[0035] a processing module configured to process the data features of each sample data by a labeling judgment module installed after the backbone network of the target classification model, and obtain a confidence degree of the target classification model predicting the sample data as the first target class label;

[0036] a secondary processing module configured to process the data features of at least part of the sample data by the labeling judgment module again, and obtain a second classification result corresponding to the at least part of the sample data; wherein the second classification result comprises a second target class label and a second classification probability corresponding to the second target class label;

[0037] a judgment module configured to judge whether the target classification model meets a stop training condition based on the second classification result of the at least part of the sample data;

[0038] an updating module configured to update the target classification model based on the confidence degree of the first target class label of each sample data when the target classification model does not meet the stop training condition, and re-perform a next round of classification labeling on the sample data set by using the updated target classification model until the target classification model meets the stop training condition; and when the target classification model meets the stop training condition, stop the classification labeling, obtain a trained target classification model, and obtain a labeled sample data set by using the first classification result of the target classification model.

[0039] In some embodiments, an electronic device is also provided, and the electronic device comprises a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, the processor and the memory communicate through the bus when the electronic device is running, and the machine readable instructions are executed by the processor to perform the steps of the automatic labeling method.

[0040] In some embodiments, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the automatic labeling method.

[0041] The embodiment of the application provides an automatic labeling method, device, electronic equipment and medium. The automatic labeling method can extract data features of each sample data in a sample data set through a backbone network of a target classification model, and obtain a first classification result of each sample data by processing the data features through the target classification model. The first classification result includes a first target category label and a first classification probability corresponding to the first target category label. The target classification model is obtained by preliminary training based on a first data set labeled by a person. The data features of each sample data are processed through a labeling judgment module installed behind the backbone network of the target classification model to obtain a confidence degree of the target classification model predicting that the sample data is the first target category label. The data features of at least part of the sample data are processed by the labeling judgment module to obtain a second classification result corresponding to the at least part of the sample data. The second classification result includes a second target category label and a second classification probability corresponding to the second target category label. Whether the target classification model meets a stop training condition is determined based on the second classification result of the at least part of the sample data. If not, the target classification model is updated based on the confidence degree of the first target category label of each sample data, and the sample data set is re-classified and labeled by using the updated target classification model in the next round until the target classification model meets the stop training condition. If yes, the classification and labeling is stopped, the trained target classification model is obtained, and the sample data set labeled by the target classification model is obtained. The method is applicable to all CNN classification models and is not limited by the original CNN classification model. The re-training of the CNN classification model can be realized based on the original CNN classification model, the CNN classification model outputs the automatic labeling result of a large amount of data, and the automatic labeling of a large amount of classification data set can be realized by using a small amount of manually labeled data. The training of the CNN classification model can be realized at the same time, and the problem of insufficient data in the training of the CNN classification model in a specific scene is solved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0043] Figure 1 The method flowchart of the automatic labeling method described in the embodiment of the application is shown;

[0044] Figure 2 The method flowchart of obtaining the confidence degree of the target classification model predicting that the sample data is the first target category label is shown.

[0045] Figure 3 A flow chart of a method for determining whether a target classification model meets a stop training condition based on a second classification result of at least part of sample data according to an embodiment of the present application is shown;

[0046] Figure 4 A flow chart of a method for updating a target classification model based on a confidence of a first target class label of each sample data according to an embodiment of the present application is shown;

[0047] Figure 5 A structural schematic diagram of an automatic labeling device according to an embodiment of the present application is shown;

[0048] Figure 6 A structural schematic diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and superiorities of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application are only intended to illustrate and describe the present application, and are not intended to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flow charts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow charts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flow charts or removed from the flow charts by those skilled in the art under the guidance of the content of the present application.

[0050] In addition, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] It should be noted that the term “comprising” will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0052] With the rise of artificial intelligence, deep learning has made a big difference in image processing, speech processing and other aspects, and is widely used in many fields. In image processing, speech processing and other processing processes, one of the most important ideas is classification, that is, to identify the categories of images, speech, footprints, etc. The classification method of deep learning is based on supervised learning method, and model training needs a large amount of labeled data. Then use a large amount of labeled data to train the model. However, data labeling often requires huge manpower, material resources and time, thereby greatly affecting the cost of project development.

[0053] And relying on manual annotation may result in annotation errors, which may cause the model not to converge at the beginning of training and affect the accuracy of model training.

[0054] Therefore, in the embodiments of the present application, an automatic labeling method and device, an electronic device and a medium are provided. The automatic labeling method can extract data features of each sample data in a sample data set through the backbone network of a target classification model, and obtain a first classification result of each sample data by processing the data features through the target classification model. The first classification result includes a first target class label and a first classification probability corresponding to the first target class label. The target classification model is obtained by preliminary training based on a first data set labeled by manual annotation. The data features of each sample data are processed by a labeling judgment module installed after the backbone network of the target classification model to obtain a confidence degree of the target classification model predicting that the sample data is the first target class label. The data features of at least part of the sample data are processed by the labeling judgment module to obtain a second classification result corresponding to the at least part of the sample data. The second classification result includes a second target class label and a second classification probability corresponding to the second target class label. Whether the target classification model meets a stop training condition is determined based on the second classification result of the at least part of the sample data. If not, the target classification model is updated based on the confidence degree of the first target class label of each sample data, and the sample data set is reclassified and labeled by using the updated target classification model in the next round until the target classification model meets the stop training condition. If yes, the classification labeling is stopped, and a trained target classification model is obtained. Meanwhile, a labeled sample data set is obtained by using the first classification result of the target classification model. The method is applicable to all CNN classification models and is not limited by the original CNN classification model. The retraining of the CNN classification model can be realized based on the original CNN classification model, and the CNN classification model can output the automatic labeling result of a large amount of data. In this way, a small amount of manually labeled data can realize the automatic labeling of a large amount of classification data set, and the training of the CNN classification model can be realized at the same time, solving the problem of insufficient data in the training of the CNN classification model in a specific scenario.

[0055] Please refer to Figure 1, Figure 1 A method flowchart of the automatic labeling method described in the embodiments of the present application is shown; specifically, the method includes the following steps S101-S106:

[0056] S101, extracting data features of each sample data in a sample data set through a backbone network of a target classification model, and processing the data features through the target classification model to obtain a first classification result of each sample data; wherein the first classification result includes a first target class label and a first classification probability corresponding to the first target class label; the target classification model is obtained by preliminary training based on a first data set manually labeled;

[0057] S102, processing the data features of each sample data through a labeling judgment module installed after the backbone network of the target classification model to obtain a confidence degree of the target classification model predicting that the sample data is the first target class label;

[0058] S103, the labeling judgment module processes the data features of at least part of the sample data twice to obtain a second classification result corresponding to the at least part of the sample data; wherein the second classification result includes a second target class label and a second classification probability corresponding to the second target class label;

[0059] S104, judging whether the target classification model meets a stop training condition based on the second classification result of the at least part of the sample data;

[0060] S105, if not, updating the target classification model based on the confidence degree of the first target class label of each sample data, and using the updated target classification model to re-classify and label the sample data set in the next round until the target classification model meets the stop training condition;

[0061] S106, if yes, stopping the classification and labeling, obtaining a trained target classification model, and using the first classification result of the target classification model to obtain a labeled sample data set.

[0062] Based on the above steps S101-S106, the core of the automatic labeling method is the labeling judgment module installed after the target classification model backbone network. The labeling judgment module processes the data features of each sample data extracted by the backbone network of the target classification model to obtain the confidence of the target classification model predicting the sample data as the first target class label. The labeling judgment module processes the data features of the sample data with low confidence again to obtain the second classification result corresponding to the part of the sample data with low confidence. Whether the target classification model meets the stopping training condition is judged based on the second classification result of the sample data with low confidence. If not, the target classification model is updated based on the confidence of the first target class label of each sample data, and the updated target classification model is used to re-classify and label the sample data set in the next round until the target classification model meets the stopping training condition. If it meets, the classification and labeling is stopped, the trained target classification model is obtained, and the labeled sample data set is obtained using the first classification result of the target classification model. The labeling judgment module in this method can be installed after any type of CNN classification model, so it is suitable for all CNN classification models and is not limited by the original CNN classification model. It can retrain the CNN classification model based on the original CNN classification model and make the CNN classification model output the automatic labeling result of the massive data. In this way, a small amount of manually labeled data can realize the automatic labeling of massive classification data sets, and the training of the CNN classification model can be realized at the same time, solving the problem of insufficient data in the training of the CNN classification model in a specific scenario.

[0063] In the step S101, the data features of each sample data in the sample data set are extracted by the backbone network of the target classification model, and the first classification result of each sample data is obtained by processing the data features by the target classification model. The first classification result includes a first target class label and a first classification probability corresponding to the first target class label. The target classification model is obtained by preliminary training based on the first data set labeled by manual labeling.

[0064] The target classification model can be any type of classification model, such as an image classification model, a speech classification model, etc. More specifically, the image classification model can be an image classification model that realizes various specific functions based on images in various application scenarios, such as a brain image classification model for predicting brain age based on brain images, an electrocardiogram classification model for predicting heart disease based on electrocardiogram, a plant recognition model for identifying plant varieties based on plant images, a face recognition model for identifying characters based on face images, etc.

[0065] Take various image classification models as examples, a training set required by a face recognition model is labeled, but now the development of CNN classification models is getting faster and faster, and these models will be used in various professional fields, which often do not have labeled sample data sets, such as brain images, electrocardiograms, and welding images.

[0066] Therefore, an automatic labeling method is needed to realize automatic labeling of a sample data set of a target classification model in a target application scenario. The sample data set includes a large amount of sample data of the target application scenario, such as images and voices.

[0067] In the embodiment of the application, a first classification result of each sample data is predicted by a target classification model. Here, the target classification model is obtained by preliminarily training a first data set based on artificial labeling. The first data set has a small amount of labeled data. In this way, the target classification model preliminarily trained by training the target classification model based on the first data set can predict the category of input data, for example, an animal species prediction model can predict the animal species according to an animal image. However, since the target classification model is trained based on a small amount of data, the target classification model does not learn the image features sufficiently, and therefore the prediction accuracy is often not enough.

[0068] The first classification result includes a first target category label and a first classification probability corresponding to the first target category label. The CNN classification model essentially predicts the probability of data belonging to each category. The first target category is the category with the maximum prediction probability, and the first classification probability corresponding to the first target category label is the maximum prediction probability. For example, the animal species prediction model outputs the prediction result of image A as follows: cat, probability 0.8; dog, probability 0.6; pig, probability 0.2; and the maximum prediction probability is 0.8. Therefore, the first classification result of image A is: cat, probability 0.8.

[0069] It should be noted that the vast amount of sample data in the sample data set is not input to the target classification model at one time, but is input in batches, for example, 10,000 sample data, 100 sample data for one batch, 100 batches, and each batch of sample data is executed in steps S101-S106 until the target classification model meets the stopping training condition, and the automatic labeling of the vast amount of sample data in the sample data set is completed.

[0070] In the step S102, the data features of each sample data are processed by the labeling judgment module installed after the main network of the target classification model to obtain the confidence of the target classification model predicting that the sample data is the first target category label.

[0071] Please refer to Figure 2The label judgment module added after the target classification model backbone network processes the data features of each sample data to obtain a confidence of the target classification model predicting the sample data as a first target category label, including the following steps S201-S202:

[0072] S201, input the data features of each sample data into the confidence calculation function of the label judgment module added after the target classification model backbone network;

[0073] S202, the confidence calculation function of the label judgment module respectively transforms the data features of each sample into a confidence between 0 and 1.

[0074] Specifically, the confidence calculation function of the label judgment module is a Sigmoid activation function. The Sigmoid activation function can map all real numbers to the interval (0, 1).

[0075] After obtaining the confidence of each sample data, before the label judgment module processes the data features of at least part of the sample data again, the automatic labeling method further includes:

[0076] Filtering negative sample data with a confidence meeting a preset negative sample condition from all sample data, and obtaining data features of the negative sample data.

[0077] The preset negative sample condition can be a preset confidence threshold, a preset proportion of sample data with a confidence in the back, a preset number of sample data with a confidence in the back, etc.

[0078] For example, in the embodiment of the present application, the number of sample data in this batch is set to N. Then the sample data features of each batch are F1, F2...Fn, and the corresponding confidence is W1, W2...Wn.

[0079] Sort the weights from high to low, define the first N*c sample data in the order as positive sample data (0<c<1), and the rest as negative sample data; wherein, c is the positive sample proportion, and 1-c is the negative sample proportion.

[0080] Correspondingly, the label judgment module processes the data features of at least part of the sample data again to obtain a second classification result corresponding to the at least part of the sample data one by one, including:

[0081] The label judgment module processes the data features of the negative sample data again to obtain a second classification result of each negative sample data.

[0082] That is, based on the confidence, the negative sample data with low confidence predicted by the target prediction model is judged again to determine the classification result of the negative sample data.

[0083] Here, the labeling and judgment module performs secondary processing on the data features of the negative sample data to obtain a second classification result for each negative sample data. Specifically, the data features of the negative sample data are fed into the fully connected layer of the labeling and judgment module, and after softmax calculation, the classification probabilities p1, p2...pn of n categories for each image can be obtained. Here, the maximum classification probability is set to p. max p max The corresponding category is the second target category label.

[0084] In other words, the second classification result includes: the second target category label, p max .

[0085] Here, the softmax function, also known as the normalization exponential function, aims to present the results of multi-class classification in the form of probabilities.

[0086] In step S103, the target classification model is judged to meet the stop training condition based on the second classification results of at least a portion of the sample data.

[0087] For details, please refer to Figure 3 In this embodiment of the application, the step of determining whether the target classification model meets the stopping training condition based on the second classification result of at least a portion of the sample data includes the following steps S301-S303:

[0088] S301. Determine whether the second category label of each negative sample data is consistent with the existing category label of the negative sample data, and obtain the judgment result of each negative sample data; wherein, the existing category label of the negative sample data in the first round of classification labeling is the first target category label of this round; the existing category label of the negative sample data in other rounds of classification labeling is the first target category label of the previous round;

[0089] S302. If the judgment result of each negative sample data is consistent, then the classification result of the target classification model meets the preset stopping training condition.

[0090] S303. If there is at least one negative sample data whose judgment result is inconsistent, then the classification result of the target classification model does not meet the preset stop training condition.

[0091] The target classification model performs multiple rounds of labeling on the sample data set. In this process, the sample data features learned by the target classification model become more and more, and the model parameters are continuously updated and become more and more optimal. Therefore, the sample features extracted by the target classification model become more and more accurate, and the first target category label determined by the target classification model based on the sample features and the second target category label determined by the labeling judgment module based on the sample features also become more and more accurate. Until the second target category label obtained in the latest round of labeling and the first target category label obtained in the last round of labeling are the same, it is indicated that the model accuracy has reached the requirement, and the labeling of the sample data is no longer wrong, and the first target category label predicted by the target classification model is the category label of the sample data.

[0092] In the embodiment of the application, there are two cases for the existing category label of the negative sample data. In the first round of classification labeling, since there is no first target category label of the last round of labeling, the existing category label of the negative sample data is the first target category label of the current round, and the first target category label and the second target category label of the current round are directly compared. In other rounds of classification labeling, the existing category label of the negative sample data is the first target category label of the last round. The target category labels of the same sample data in different rounds are compared to determine whether they are consistent.

[0093] In the embodiment of the application, when the judgment result of at least one negative sample data is inconsistent, the method further comprises:

[0094] The first classification probability of the existing category label of the negative sample data and the second classification probability corresponding to the second target category label are compared.

[0095] If the first classification probability is less than the second classification probability, and the difference between the first classification probability and the second classification probability exceeds a preset threshold, the second classification result is taken as the first classification result of the current round of training of the negative sample data, and the next round of classification labeling is performed based on the updated first classification result.

[0096] Conversely, the existing category label and the first classification probability of the negative sample data are taken as the first classification result of the current round of training of the negative sample data, and the next round of classification labeling is performed based on the updated first classification result.

[0097] That is, except for the first round, the first classification results of the latest round and the last round of the negative sample data are compared, and the classification probability of which round is higher, which indicates that the prediction result of this round is more accurate. The higher classification probability and category label are updated as the first target classification result of the current round, and are taken as the existing category label of the sample data in the next round.

[0098] Specifically, the first classification probability of the existing category label is set as p last, the first classification probability of the second target class label in the second classification result is p max , the preset threshold is δ, which is a constant

[0099] The first classification probability is less than the second classification probability, and the difference between the first classification probability and the second classification probability exceeds the preset threshold, that is, p max -p last＞ W.

[0100] In the step S105, if the condition is not met, the target classification model is updated based on the confidence of the first target class label of each sample data, and the next round of classification labeling is performed on the sample data set using the updated target classification model until the target classification model meets the stopping training condition.

[0101] That is, if the predicted class labels for the same sample data are inconsistent twice, the predicted class label needs to be modified, and the target classification model also needs to be continuously trained.

[0102] Here, please refer to Figure 4 The target classification model is updated based on the confidence of the first target class label of each sample data, including the following steps S401-S402:

[0103] S401, based on the confidence of the first target class label of each sample data input into the target classification model and the preset positive and negative sample screening condition, the positive sample data and the negative sample data in the sample data set are screened out.

[0104] S402, updating the target classification model based on the confidence of the positive sample data and the confidence of the negative sample data.

[0105] Updating the target classification model based on the confidence of the positive sample data and the confidence of the negative sample data includes:

[0106] Calculating the confidence mean of the positive sample data and the confidence mean of the negative sample data.

[0107] Based on the confidence mean of the positive sample data, the confidence mean of the negative sample data, the preset negative sample suppression constant and the preset loss function calculation rule, the loss function calculation result is determined.

[0108] Updating the target classification model based on the loss function calculation result.

[0109] Based on the confidence of the first target class label of each sample data and the preset positive and negative sample screening condition, the positive sample data and the negative sample data in the sample data set are screened out. In the embodiment of the application, the preset positive and negative sample screening condition is the same as the preset negative sample condition, and the specific condition is as follows:

[0110] Set the number of sample data in this batch to N. Then the characteristics of the sample data for each batch are: F1, F2... Fn, and the corresponding confidence levels are W1, W2... Wn.

[0111] Sort the weights from high to low, and define the first N*c sample data with higher ranking as positive sample data (0 < c < 1), and the rest as negative sample data; where c is the proportion of positive samples, and 1 - c is the proportion of negative samples.

[0112] Similarly, the model parameters of the target classification model are also updated based on the sample data of this batch, that is, based on each sample data input into the target classification model in this batch.

[0113] In this way, based on the mean confidence level of positive sample data, the mean confidence level of negative sample data, the preset negative sample suppression constant, and the preset loss function calculation rule, determine the calculation result of the loss function, where the specific preset loss function calculation rule is as follows:

[0114] The mean confidence level W of positive sample data postive is:

[0115]

[0116] The mean confidence level W of negative sample data negtive is:

[0117]

[0118] Here, N is the number of sample data in this batch, N*c is the number of positive sample data, N*(1 - c) is the number of negative sample data, i is the i-th sample data, and W i is the confidence level of the i-th sample data among the confidence levels W1, W2... Wn.

[0119] To strengthen positive samples and suppress negative samples, the embodiment of this application introduces a preset negative sample suppression constant. Design the preset loss function calculation rule as:

[0120] Loss = max(β - (W postive - W negtive ), 0); [[ID=4​​​​​​​​​​

[0123] Based on the same inventive concept, the application also provides an automatic labeling device corresponding to the automatic labeling method. Since the device solves problems in the same principle as the automatic labeling method described above, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0124] Please refer to Figure 5 , Figure 5 The structure of the automatic labeling device is shown in the schematic diagram; specifically, the automatic labeling device comprises:

[0125] The extraction module 501 is configured to extract the data features of each sample data in the sample data set through the backbone network of the target classification model, and obtain the first classification result of each sample data by processing the data features through the target classification model; wherein the first classification result comprises a first target class label and a first classification probability corresponding to the first target class label; the target classification model is obtained by preliminary training based on the first data set manually labeled;

[0126] The processing module 502 is configured to process the data features of each sample data through the labeling judgment module installed after the backbone network of the target classification model, and obtain the confidence of the target classification model predicting the sample data as the first target class label;

[0127] The secondary processing module 503 is configured to process the data features of at least part of the sample data through the labeling judgment module to obtain the second classification result corresponding to the at least part of the sample data; wherein the second classification result comprises a second target class label and a second classification probability corresponding to the second target class label;

[0128] The judgment module 504 is configured to judge whether the target classification model meets the stopping training condition based on the second classification result of at least part of the sample data;

[0129] The update module 505 is configured to update the target classification model based on the confidence of the first target class label of each sample data when the target classification model does not meet the stopping training condition, and to re-classify and label the sample data set using the updated target classification model until the target classification model meets the stopping training condition; when the target classification model meets the stopping training condition, the classification and labeling is stopped, and the trained target classification model is obtained, and the labeled sample data set is obtained using the first classification result of the target classification model.

[0130] The automatic labeling device can extract data features of each sample data in the sample data set through the backbone network of the target classification model, and obtain a first classification result of each sample data by processing the data features through the target classification model; wherein the first classification result comprises a first target category label and a first classification probability corresponding to the first target category label; the target classification model is obtained by preliminary training based on a first data set manually labeled; the data features of each sample data are processed through the labeling judgment module installed after the backbone network of the target classification model to obtain a confidence degree that the target classification model predicts the sample data as the first target category label; the data features of at least part of the sample data are processed by the labeling judgment module to obtain a second classification result corresponding to the at least part of the sample data; wherein the second classification result comprises a second target category label and a second classification probability corresponding to the second target category label; whether the target classification model meets the stop training condition is determined based on the second classification result of the at least part of the sample data; if not, the target classification model is updated based on the confidence degree of the first target category label of each sample data, and the sample data set is re-classified and labeled by using the updated target classification model in the next round until the target classification model meets the stop training condition; if yes, the classification and labeling is stopped, the trained target classification model is obtained, and the labeled sample data set is obtained by using the first classification result of the target classification model; the method is applicable to all CNN classification models and is not limited by the original CNN classification model; the CNN classification model can be retrained based on the original CNN classification model, and the CNN classification model can output the automatic labeling result of a large amount of data; in this way, a small amount of manually labeled data can realize automatic labeling of a large amount of classification data set, and the training of the CNN classification model can be realized at the same time, solving the problem of insufficient data in the training of the CNN classification model in a specific scenario.

[0131] In some embodiments, the automatic labeling device, when processing the data features of each sample data through the labeling judgment module installed after the backbone network of the target classification model to obtain the confidence degree that the target classification model predicts the sample data as the first target category label, is specifically used for:

[0132] inputting the data features of each sample data into a confidence degree calculation function of the labeling judgment module installed after the backbone network of the target classification model;

[0133] The confidence degree calculation function of the labeling judgment module respectively transforms the data features of each sample into a confidence degree between 0 and 1.

[0134] In some embodiments, the secondary processing module of the automatic labeling device is further configured to, before the labeling judging module processes the data features of at least part of the sample data, filter out negative sample data whose confidence meets a preset negative sample condition from all the sample data, and obtain data features of the negative sample data.

[0135] Correspondingly, when the secondary processing module processes the data features of at least part of the sample data through the labeling judging module to obtain second classification results corresponding to the at least part of the sample data, the secondary processing module is specifically configured to:

[0136] The labeling judging module processes the data features of the negative sample data to obtain a second classification result of each negative sample data.

[0137] In some embodiments, when the judging module judges whether the target classification model meets a stop training condition based on the second classification results of at least part of the sample data, the judging module is specifically configured to:

[0138] judge whether the second classification label of each negative sample data is consistent with the existing classification label of the negative sample data, to obtain a judgment result of each negative sample data; wherein the existing classification label of the negative sample data in the first round of classification labeling is the first target classification label in the current round; and the existing classification label of the negative sample data in other rounds of classification labeling is the first target classification label in the previous round.

[0139] If the judgment result of each negative sample data is consistent, the classification result of the target classification model meets the preset stop training condition.

[0140] If the judgment result of at least one negative sample data is inconsistent, the classification result of the target classification model does not meet the preset stop training condition.

[0141] In some embodiments, when the judgment result of at least one negative sample data is inconsistent, the updating module of the automatic labeling device is further configured to:

[0142] compare the first classification probability of the existing classification label of the negative sample data with the second classification probability corresponding to the second target classification label.

[0143] If the first classification probability is less than the second classification probability, and the difference between the first classification probability and the second classification probability exceeds a preset threshold, the second classification result is taken as the first classification result of the negative sample data in the current round of training, and the next round of classification labeling is performed based on the updated first classification result.

[0144] Otherwise, the existing class label of the negative sample data and the first classification probability are taken as the first classification result of the negative sample data in the current training, and the next round of classification labeling is performed based on the updated first classification result.

[0145] In some embodiments, the updating module in the automatic labeling apparatus is specifically configured to:

[0146] based on the confidence of the first target class label of each sample data input into the target classification model and a preset positive and negative sample screening condition, screening out positive sample data and negative sample data in the sample data set;

[0147] updating the target classification model based on the confidence of the positive sample data and the confidence of the negative sample data.

[0148] In some embodiments, the updating module in the automatic labeling apparatus is specifically configured to:

[0149] calculating the mean of the confidence of the positive sample data and the mean of the confidence of the negative sample data;

[0150] based on the mean of the confidence of the positive sample data, the mean of the confidence of the negative sample data, a preset negative sample suppression constant and a preset loss function calculation rule, determining a loss function calculation result;

[0151] updating the target classification model based on the loss function calculation result.

[0152] Based on the same inventive concept, the embodiments of the present application also provide an electronic device corresponding to the automatic labeling method. Since the principle of solving problems in the electronic device of the embodiments of the present application is similar to that of the above-mentioned electronic device of the embodiments of the present application, the implementation of the electronic device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0153] Please refer to Figure 6 , Figure 6 The structure schematic diagram of the electronic device of the embodiments of the present application is shown, specifically, the electronic device 600 comprises a processor 602, a memory 601 and a bus, the memory 601 stores machine readable instructions executable by the processor 602, when the electronic device 600 runs, the processor 602 and the memory 601 communicate through the bus, and the machine readable instructions are executed by the processor 602 to execute the steps of the automatic labeling method.

[0154] Based on the same inventive concept, the application further provides a computer readable storage medium corresponding to the automatic labeling method. Since the computer readable storage medium in the application solves the problem in the same principle as the computer readable storage medium described above, the implementation of the computer readable storage medium can be referred to the implementation of the method, and the repeated parts will not be described here.

[0155] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is run by a processor to execute the steps of the automatic labeling method.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system and device can refer to the corresponding process in the method embodiment, and the application will not be described here. In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. The above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some communication interface, indirect coupling or communication connection between the devices or modules, which can be electrical, mechanical or other forms.

[0157] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to the actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0158] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0159] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0160] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An automatic labeling method, characterized by: The method comprises: extracting data features of each sample data in a sample data set through a backbone network of a target classification model, and obtaining a first classification result of each sample data by processing the data features through the target classification model; wherein the first classification result comprises a first target category label and a first classification probability corresponding to the first target category label; the target classification model is obtained by preliminary training based on a first data set with artificial labeling; the sample data set is a sample image data set or a sample voice data set; processing the data features of each sample data through a labeling judgment module added behind the backbone network of the target classification model to obtain a confidence degree of the target classification model predicting that the sample data is the first target category label; the labeling judgment module processes the data features of at least part of the sample data for a second time to obtain a second classification result corresponding to the at least part of the sample data; wherein the second classification result comprises a second target category label and a second classification probability corresponding to the second target category label; judging whether the target classification model meets a stop training condition based on the second classification result of the at least part of the sample data; if not, updating the target classification model based on the confidence degree of the first target category label of each sample data, and re-performing a next round of classification labeling on the sample data set by using the updated target classification model until the target classification model meets the stop training condition; if yes, stopping the classification labeling to obtain a trained target classification model, and obtaining a labeled sample data set by using the first classification result of the target classification model; before the labeling judgment module processes the data features of at least part of the sample data for a second time, the method further comprises: selecting negative sample data meeting a preset negative sample condition from all sample data, and obtaining data features of the negative sample data; correspondingly, the labeling judgment module processes the data features of at least part of the sample data for a second time to obtain a second classification result corresponding to the at least part of the sample data, comprising: the labeling judgment module processes the data features of the negative sample data for a second time to obtain a second classification result of each negative sample data; the judging whether the target classification model meets the stop training condition based on the second classification result of the at least part of the sample data comprises: judging whether the second category label of each negative sample data is consistent with the category label already existing in the negative sample data to obtain a judgment result of each negative sample data; wherein the category label already existing in the negative sample data in a first round of classification labeling is the first target category label in the round; the category label already existing in the negative sample data in other rounds of classification labeling is the first target category label in a previous round; if the judgment result of each negative sample data is consistent, the classification result of the target classification model meets a preset stop training condition; if the judgment result of at least one negative sample data is inconsistent, the classification result of the target classification model does not meet the preset stop training condition.

2. The automatic labeling method of claim 1, wherein, the labeling judgment module added behind the backbone network of the target classification model processes the data features of each sample data to obtain a confidence degree of the target classification model predicting that the sample data is the first target category label, comprising: The data features of each sample data are input into a confidence calculation function of a labeling judgment module added behind a main network of a target classification model; The confidence calculation function of the labeling judgment module respectively transforms the data features of each sample into a confidence between 0 and 1.

3. The automatic labeling method of claim 1, wherein, When the judgment result of at least one negative sample data is inconsistent, the method further comprises: comparing a first classification probability of the existing category label of the negative sample data with a second classification probability corresponding to the second target category label; if the first classification probability is less than the second classification probability, and the difference between the first classification probability and the second classification probability exceeds a preset threshold, the second classification result is taken as the first classification result of the negative sample data in the current round of training, and the next round of classification labeling is performed based on the updated first classification result; otherwise, the existing category label and the first classification probability of the negative sample data are taken as the first classification result of the negative sample data in the current round of training, and the next round of classification labeling is performed based on the updated first classification result.

4. The automatic labeling method of claim 1, wherein, The method of updating the target classification model based on the confidence of the first target category label of each sample data comprises: screening positive sample data and negative sample data in the sample data set based on the confidence of the first target category label of each sample data input into the target classification model and a preset positive and negative sample screening condition; updating the target classification model based on the confidence of the positive sample data and the confidence of the negative sample data.

5. The automatic labeling method of claim 4, wherein, The method of updating the target classification model based on the confidence of the positive sample data and the confidence of the negative sample data comprises: calculating the mean of the confidence of the positive sample data and the mean of the confidence of the negative sample data; determining a loss function calculation result based on the mean of the confidence of the positive sample data, the mean of the confidence of the negative sample data, a preset negative sample suppression constant and a preset loss function calculation rule; updating the target classification model based on the loss function calculation result.

6. An automatic labeling apparatus characterized by comprising: The device comprises: an extraction module configured to extract data features of each sample data in a sample data set through a main network of a target classification model, and obtain a first classification result of each sample data by processing the data features through the target classification model; wherein the first classification result comprises a first target category label and a first classification probability corresponding to the first target category label; the target classification model is obtained by preliminary training based on a first data set labeled by a human; and the sample data set is a sample image data set or a sample speech data set; a processing module configured to process the data features of each sample data through a labeling judgment module added behind the main network of the target classification model, and obtain a confidence of the target classification model in predicting that the sample data is the first target category label; a secondary processing module configured to process the data features of at least part of the sample data through the labeling judgment module, and obtain a second classification result corresponding to the at least part of the sample data; wherein the second classification result comprises a second target category label and a second classification probability corresponding to the second target category label; a judgment module configured to judge whether the target classification model meets a stop training condition based on the second classification result of the at least part of the sample data. The updating module is configured to update the target classification model based on the confidence of the first target category label of each sample data when the target classification model does not meet the stop training condition, and to perform the next round of classification labeling on the sample data set again by using the updated target classification model until the target classification model meets the stop training condition; when the target classification model meets the stop training condition, the classification labeling is stopped, the trained target classification model is obtained, and the sample data set with the labeling is obtained by using the first classification result of the target classification model; Before the annotation judgment module processes the data features of at least part of the sample data again, the method further comprises: Filtering negative sample data with a confidence meeting a preset negative sample condition from all sample data, and obtaining data features of the negative sample data; Correspondingly, the annotation judgment module processes the data features of at least part of the sample data again to obtain a second classification result corresponding to the at least part of the sample data, comprising: The annotation judgment module processes the data features of the negative sample data again to obtain a second classification result of each negative sample data; The method of judging whether the target classification model meets the stop training condition based on the second classification result of at least part of the sample data comprises: Judging whether the second category label of each negative sample data is consistent with the existing category label of the negative sample data to obtain a judgment result of each negative sample data; wherein the existing category label of the negative sample data in the first round of classification labeling is the first target category label in the round; and the existing category label of the negative sample data in other rounds of classification labeling is the first target category label in the previous round; If the judgment result of each negative sample data is consistent, the classification result of the target classification model meets the preset stop training condition; If the judgment result of at least one negative sample data is inconsistent, the classification result of the target classification model does not meet the preset stop training condition.

7. An electronic device, comprising: The electronic device comprises a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, the processor and the memory communicate through the bus when the electronic device is running, and the machine readable instructions are executed by the processor to perform the steps of the automatic labeling method according to any one of claims 1 to 5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the automatic labeling method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, ​

Images