Training data acquisition method and apparatus, and electronic device

Abstract

The present application provides a training data acquisition method and apparatus, and an electronic device, which are applied to the technical field of data processing. The training data acquisition method comprises: inputting labeled data into an initial model to obtain first reasoning data; for a label, calculating an optimal threshold value corresponding to the label on the basis of the label and first reasoning data corresponding to labeled data labeled with the label, the optimal threshold value being used for distinguishing positive samples and negative samples input into the initial model; and, on the basis of the optimal threshold value corresponding to the label, screening out from amongst data to be labeled training data corresponding to the label. In the described solution, compared with directly labeling massive volume of data to be labeled in the prior art, the training data acquisition method provided in the embodiments of the present application can only label effective training data amongst data to be labeled, thus improving the efficiency of acquiring effective data.

Description

Method, device and electronic device for acquiring training data

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application CN202410374236.5, filed on March 29, 2024, entitled “A method, device and electronic device for acquiring training data”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of data processing technology, and in particular relates to a method, device and electronic device for acquiring training data. Background Art

[0004] Although user-facing enterprise applications often have a wealth of internal business data (for example, social networking and sharing apps), filtering this massive amount of data to identify data that is beneficial for model training is a time-consuming task. Especially in content security scenarios, user-posted content can challenge models and change daily. Therefore, it's necessary to continuously collect user content to create training sets and retrain models to maintain stable business results.

[0005] However, labeling every piece of user-generated image and text content would be a significant cost investment. Furthermore, as training data continues to grow, over 50%-80% of the labeled data will share similar characteristics with historical data, providing little benefit to model learning. Labeling this data would also be ineffective. Furthermore, this data would increase model training time and training costs. Therefore, using existing techniques to collect training data is inefficient in acquiring valid data.

[0006] Summary of the Invention

[0007] The present application provides a method, device and electronic device for obtaining training data, which are used to solve the technical problem of low efficiency in obtaining effective data when collecting training data in the prior art.

[0008] In a first aspect, an embodiment of the present application provides a method for obtaining training data, comprising: inputting labeled data into an initial model to obtain first inference data; for a label, calculating an optimal threshold corresponding to the label based on the label and the first inference data corresponding to the labeled data labeled with the label; wherein the optimal threshold is used to distinguish between positive samples and negative samples input into the initial model; and filtering out training data corresponding to the label from the data to be labeled based on the optimal threshold corresponding to the label.

[0009] In the above scheme, before labeling the data to be labeled, valid training data can be filtered from the data to be labeled by calculating the optimal threshold corresponding to the label, so that only the above-screened training data can be labeled. Therefore, compared with the prior art method of directly labeling the huge amount of data to be labeled, the training data acquisition method provided in the embodiment of the application can only label the valid training data in the data to be labeled, thereby improving the efficiency of obtaining valid data.

[0010] In an optional embodiment, the step of calculating the optimal threshold corresponding to the label based on the label and the first inference data corresponding to the labeled data annotated with the label includes: calculating the optimal threshold for optimizing the model performance evaluation index based on the first inference data and the label. In the above scheme, in the process of determining the optimal threshold, the optimal threshold for optimizing the model performance evaluation index can be calculated, thereby accurately screening out valid training data from the data to be labeled. Since valid training data can be screened out from the data to be labeled, the amount of training data that needs to be labeled can be reduced, thereby further improving the efficiency of obtaining valid data.

[0011] In an optional embodiment, the method of calculating the optimal threshold value for the model performance evaluation indicator based on the first inference data and the label includes: traversing multiple initial thresholds based on a preset interval, calculating the F1 scores under the multiple initial thresholds; and determining the initial threshold corresponding to the maximum F1 score as the optimal threshold corresponding to the label. In the above scheme, a brute force algorithm can be used to traverse multiple initial thresholds in turn and calculate the F1 scores corresponding to the multiple initial thresholds, so that the initial threshold corresponding to the maximum F1 score can be determined as the optimal threshold corresponding to the label. Among them, since the maximum F1 score can maximize the overall performance of the model, the optimal threshold corresponding to the maximum F1 score can accurately filter out valid training data from the data to be labeled, thereby further improving the efficiency of obtaining valid data.

[0012] In an optional embodiment, the method of calculating the optimal threshold value for achieving the best model performance evaluation index based on the first inference data and the label includes: calculating the optimal threshold value for maximizing the F1 score using a gradient descent algorithm. In the above scheme, since the F1 score is a convex function, the gradient descent algorithm can be used to calculate the optimal threshold value for maximizing the F1 score, thereby quickly calculating the optimal threshold value corresponding to the label. Among them, the maximum F1 score can maximize the overall performance of the model. Therefore, the optimal threshold value corresponding to the maximum F1 score can accurately filter out valid training data from the data to be labeled, thereby further improving the efficiency of obtaining valid data.

[0013] In an optional embodiment, the method of filtering out the training data corresponding to the label from the data to be labeled according to the optimal threshold corresponding to the label includes: determining the filtering range corresponding to the label according to the optimal threshold corresponding to the label and a pre-set floating value; inputting the data to be labeled into the initial model to obtain the second inference data output by the initial model; and determining whether the data to be labeled is the training data according to the second inference data and the filtering range. In the above scheme, the filtering range corresponding to the label can be determined according to the optimal threshold and the floating value, so that effective training data can be filtered out from the data to be labeled based on the above filtering range. Among them, since the optimal threshold can make the model performance evaluation index reach the best, a batch of training data with high accuracy can be filtered out by setting the floating value.

[0014] In an optional embodiment, the second inference data includes: an inference label and an inference probability value corresponding to the inference label; determining whether the data to be labeled is the training data based on the second inference data and the screening range includes: judging whether the inference probability value is within the screening range corresponding to the inference label; if the inference probability value is within the screening range corresponding to the inference label, determining the data to be labeled as the training data. In the above scheme, if the inference probability value corresponding to the data to be labeled is within the screening range corresponding to the inference label, it means that the data to be labeled has a high improvement on the initial model, and therefore can be used as effective training data, thereby improving the efficiency of training the initial model.

[0015] In an optional embodiment, the floating value includes a first value and a second value, and the first value is not equal to the second value; the determining of the screening range based on the optimal threshold and the pre-set floating value includes: determining the sum of the optimal threshold and the first value as the upper limit of the screening range, and determining the difference between the optimal threshold and the second value as the lower limit of the screening range. In the above scheme, the pre-set floating value can include two different values, so that the resulting screening range can be more suitable for the current business scenario, thereby more accurately filtering out valid training data from the data to be labeled.

[0016] In a second aspect, an embodiment of the present application provides a device for acquiring training data, comprising: an input module for inputting labeled data into an initial model to obtain first inference data; a calculation module for calculating, for a label, an optimal threshold corresponding to the label based on the label and the first inference data corresponding to the labeled data labeled with the label; wherein the optimal threshold is used to distinguish between positive samples and negative samples input into the initial model; and a screening module for screening out the training data corresponding to the label from the data to be labeled based on the optimal threshold corresponding to the label.

[0017] In the above scheme, before labeling the data to be labeled, valid training data can be filtered from the data to be labeled by calculating the optimal threshold corresponding to the label, so that only the above-screened training data can be labeled. Therefore, compared with the prior art method of directly labeling the huge amount of data to be labeled, the training data acquisition method provided in the embodiment of the application can only label the valid training data in the data to be labeled, thereby improving the efficiency of obtaining valid data.

[0018] In an optional embodiment, the calculation module is specifically configured to calculate, based on the first inference data and the label, an optimal threshold for achieving the best model performance evaluation indicator. In the above scheme, during the process of determining the optimal threshold, the optimal threshold for achieving the best model performance evaluation indicator can be calculated, thereby accurately screening out valid training data from the data to be labeled. Since valid training data can be screened out from the data to be labeled, the amount of training data that requires labeling can be reduced, thereby further improving the efficiency of obtaining valid data.

[0019] In an optional embodiment, the calculation module is further used to: traverse multiple initial thresholds based on preset intervals, calculate the F1 scores under the multiple initial thresholds; and determine the initial threshold corresponding to the maximum F1 score as the optimal threshold corresponding to the label. In the above scheme, a brute force algorithm can be used to traverse multiple initial thresholds in sequence and calculate the F1 scores corresponding to the multiple initial thresholds, so that the initial threshold corresponding to the maximum F1 score can be determined as the optimal threshold corresponding to the label. Among them, since the maximum F1 score can maximize the overall performance of the model, the optimal threshold corresponding to the maximum F1 score can accurately filter out valid training data from the data to be labeled, thereby further improving the efficiency of obtaining valid data.

[0020] In an optional embodiment, the calculation module is further configured to calculate an optimal threshold that maximizes the F1 score using a gradient descent algorithm. In the above scheme, since the F1 score is a convex function, the gradient descent algorithm can be used to calculate the optimal threshold that maximizes the F1 score, thereby quickly calculating the optimal threshold corresponding to the label. The maximum F1 score maximizes the overall performance of the model. Therefore, the optimal threshold corresponding to the maximum F1 score can accurately filter valid training data from the data to be labeled, thereby further improving the efficiency of obtaining valid data.

[0021] In an optional embodiment, the screening module is specifically used to: determine the screening range corresponding to the label based on the optimal threshold corresponding to the label and a pre-set floating value; input the data to be labeled into the initial model to obtain the second inference data output by the initial model; and determine whether the data to be labeled is the training data based on the second inference data and the screening range. In the above scheme, the screening range corresponding to the label can be determined based on the optimal threshold and the floating value, so that effective training data can be screened from the data to be labeled based on the above screening range. Among them, since the optimal threshold can make the model performance evaluation index reach the best, a batch of high-accuracy training data can be screened out by setting the floating value.

[0022] In an optional embodiment, the second inference data includes: an inference label and an inference probability value corresponding to the inference label; the screening module is further configured to: determine whether the inference probability value is within a screening range corresponding to the inference label; if the inference probability value is within the screening range corresponding to the inference label, then the data to be labeled is determined as the training data. In the above scheme, if the inference probability value corresponding to the data to be labeled is within the screening range corresponding to the inference label, it indicates that the data to be labeled has a high improvement on the initial model and can therefore be used as valid training data, thereby improving the efficiency of training the initial model.

[0023] In an optional embodiment, the floating value includes a first value and a second value, and the first value is not equal to the second value; the screening module is further configured to: determine the sum of the optimal threshold value and the first value as the upper limit of the screening range, and determine the difference between the optimal threshold value and the second value as the lower limit of the screening range. In the above scheme, the pre-set floating value can include two different values, so that the resulting screening range can be more suitable for the current business scenario, thereby more accurately screening out valid training data from the data to be labeled.

[0024] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a bus; the processor and the memory communicate with each other via the bus; the memory stores computer program instructions that can be executed by the processor, and the processor calls the computer program instructions to execute the method for acquiring training data as described in the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the method for obtaining training data as described in the first aspect.

[0026] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0028] FIG1 is a flow chart of a method for acquiring training data provided in an embodiment of the present application;

[0029] FIG2 is a structural block diagram of a device for acquiring training data provided in an embodiment of the present application;

[0030] FIG3 is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] For image or text classification models, training data typically includes content and labels, and the output typically includes the label and the probability of the model matching that label. Continuously collecting and enriching training data can improve model training effectiveness and enable the model to learn capabilities that better meet business expectations. Common methods for enriching training data include data augmentation and iteratively collecting business data.

[0032] Data augmentation methods enhance data by inverting and sharpening images and adjusting the word order of text, thereby generating additional samples that were not collected in the production environment. Compared to training with the original dataset, this method can effectively improve the performance of the trained model. However, in content security scenarios where iterative model training is required, continuous collection of user-posted content is necessary to create a training set. This method cannot solve the problem of large amounts of invalid annotations during the training set creation process and can only be used as an additional method to improve training results.

[0033] The method of iteratively collecting business data is to randomly extract a part of the business data that has been inferred by the model for manual labeling, and then obtain the data that the model did not recall or mistakenly recalled; making this type of data into a training set can effectively improve the effect of model training, but the amount of data that can be collected is strongly correlated with the number of manual random inspections, and only a small part of the random inspection data is unrecalled and mistakenly recalled data. Therefore, there are still a large number of invalid annotations, the cost of inputting annotation is high, and the output of effective data is low, and the efficiency of obtaining effective data is insufficient.

[0034] Based on the above analysis, an embodiment of the present application provides a method for obtaining training data, which can filter the data to be labeled and only label the filtered data, thereby reducing invalid labeling and improving the efficiency of obtaining valid data when collecting training data.

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.

[0036] It should be noted that the term "plurality" in the embodiments of the present invention refers to two or more. Therefore, in the embodiments of the present invention, "plurality" can also be understood as "at least two." "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / ," unless otherwise specified, generally indicates an "or" relationship between the associated objects.

[0037] Please refer to Figure 1, which is a flow chart of a method for obtaining training data provided in an embodiment of the present application. The method for obtaining training data is used to filter out valid training data from a large amount of data to be labeled, so that only the filtered training data can be labeled, and the labeled training data can be used to train the initial model to obtain a trained model. The method for obtaining training data provided in an embodiment of the present application may specifically include the following steps:

[0038] Step S101: input the labeled data into the initial model to obtain first inference data.

[0039] Step S102: For a label, calculate the optimal threshold corresponding to the label based on the label and the first inference data corresponding to the labeled data labeled with the label.

[0040] Step S103: Filter out training data corresponding to the label from the data to be labeled according to the optimal threshold corresponding to the label.

[0041] Specifically, in step S101 above, labeled data refers to data that has been previously labeled with corresponding labels. The labels indicate the correct output value or category of each data item after processing by the initial model. It is understood that the initial model generally corresponds to multiple labels, and different initial models may correspond to different labels. This embodiment of the present application does not specifically limit this, and those skilled in the art may make appropriate adjustments based on actual circumstances.

[0042] It should be noted that the embodiments of this application do not specifically limit the specific implementation method for obtaining labeled data, and those skilled in the art may make appropriate adjustments based on actual circumstances. For example, labeled data may be received from an external device; or labeled data may be read from a local or cloud storage.

[0043] Furthermore, those skilled in the art are not particularly limited to the specific implementation of the initial model and may make appropriate adjustments based on actual circumstances. For example, the initial model may be an untrained original neural network model; or, the initial model may be a model pre-trained using the ImageNet dataset; or, the initial model may be a pre-trained model, etc.

[0044] It is understood that in different scenarios, the labeled data, label types, and corresponding initial models may be the same, partially the same, or completely different. For example, in an enterprise business scenario, the labeled data may include business data, the label type may include customer type, and the initial model may include a customer behavior analysis model; or in a path planning scenario, the labeled data may include obstacle data, the label type may include obstacle type, and the initial model may include a path planning model, etc.

[0045] Taking the enterprise business scenario as an example, business data refers to the online business data related to the initial model, while the business data under a certain label refers to the part of the business data that belongs to the label.

[0046] It should be noted that the embodiments of this application do not specifically limit the specific implementation methods for obtaining business data, and those skilled in the art may make appropriate adjustments based on actual circumstances. For example, business data sent by an external device may be received; business data stored locally or in the cloud may be read; or business data may be directly extracted from online business data.

[0047] Taking extraction directly from online business data as an example, as an implementation method, multiple tags can be traversed, and a small portion of data can be extracted from a large amount of online business data as the business data under the tag in the above step S101.

[0048] In step S102, for one of the multiple labels, an optimal threshold corresponding to the label can be calculated based on the first inference data corresponding to the labeled data labeled with the label obtained in step S101 and the label. The optimal threshold is used to distinguish between positive and negative samples input into the initial model. Specifically, if the result obtained after inputting a piece of data into the initial model is greater than the optimal threshold corresponding to a certain label, the data can be considered a positive sample of the label; if the result obtained after inputting a piece of data into the initial model is less than the optimal threshold corresponding to a certain label, the data can be considered a negative sample of the label.

[0049] It should be noted that the embodiments of this application do not impose specific limitations on the specific implementation of calculating the above-mentioned optimal threshold value, and those skilled in the art may make appropriate adjustments based on actual circumstances. For example, the accuracy of the first inference data may be determined based on the labels, and the optimal threshold value may be determined based on the accuracy; alternatively, the recall and precision of the initial model may be calculated based on the first inference data and the labels, and the optimal threshold value may be determined based on the recall and precision; alternatively, the F1 score of the initial model may be calculated based on the first inference data and the labels, and the optimal threshold value may be determined based on the F1 score, etc.

[0050] As an implementation manner, the above step S102 may be repeatedly performed for different tags, thereby obtaining optimal thresholds corresponding to multiple tags.

[0051] In step S103, based on the optimal threshold corresponding to the label calculated in step S102, training data corresponding to the label can be filtered from the data to be labeled. The data to be labeled refers to a large amount of data collected from the Internet, business, etc., which contains a large amount of invalid data. Therefore, the efficiency of training the initial model using the data to be labeled is low. The training data refers to a portion of the data to be labeled, which basically includes valid data. Therefore, the efficiency of training the initial model using the training data is high.

[0052] It is understandable that after obtaining the above training data, the above training data can be labeled, and after obtaining the corresponding labels, the initial model can be trained using the above training data and the labels corresponding to the training data.

[0053] It should be noted that the embodiments of this application do not impose any specific limitations on the specific implementation of the aforementioned screening training data. Persons skilled in the art may make appropriate adjustments based on actual circumstances. For example, only data in the unlabeled data whose inference results are equal to the optimal threshold may be determined as training data; data in the unlabeled data whose inference results are near the optimal threshold may be determined as training data, etc.

[0054] As an implementation method, after calculating the optimal threshold corresponding to a label, the training data corresponding to the label can be filtered based on the above optimal threshold; by repeatedly executing the above steps S102-S103, training data corresponding to multiple labels can be obtained.

[0055] As another implementation, the above step S102 may be repeatedly performed to obtain optimal thresholds corresponding to multiple labels, and then the training data corresponding to the multiple labels may be screened based on the multiple optimal thresholds.

[0056] In the above scheme, before labeling the data to be labeled, valid training data can be filtered from the data to be labeled by calculating the optimal threshold corresponding to the label, so that only the above-screened training data can be labeled. Therefore, compared with the prior art method of directly labeling the huge amount of data to be labeled, the training data acquisition method provided in the embodiment of the application can only label the valid training data in the data to be labeled, thereby improving the efficiency of obtaining valid data.

[0057] Furthermore, based on the above embodiment, the above step S102 may specifically include the following steps:

[0058] The optimal threshold for model performance evaluation indicators is calculated based on the inference data and labels.

[0059] Specifically, the embodiments of this application do not specifically limit the specific implementation of the above-mentioned model performance evaluation indicators, and those skilled in the art may make appropriate adjustments based on actual conditions. For example, the model performance evaluation indicators may include one or more parameters such as precision, recall, or F1 score.

[0060] In the above scheme, during the process of determining the optimal threshold, the optimal threshold that optimizes the model performance evaluation indicator can be calculated, thereby accurately selecting valid training data from the data to be labeled. Furthermore, since valid training data can be selected from the data to be labeled, the amount of training data that needs to be labeled can be reduced, thereby further improving the efficiency of obtaining valid data.

[0061] Furthermore, based on the above embodiment, as an implementation method, the step of calculating the optimal threshold value for achieving the best model performance evaluation index based on the inference data and the labeled data may specifically include the following steps:

[0062] Step 1) traverses multiple initial thresholds based on preset intervals and calculates the F1 scores under the multiple initial thresholds.

[0063] Step 2) The initial threshold corresponding to the maximum F1 score is determined as the optimal threshold corresponding to the label.

[0064] Specifically, in step 1) above, the F1 score is typically used to measure the accuracy of a classification model. It is the harmonic mean of precision and recall, aiming to find a balance that achieves optimal performance for both metrics. The maximum F1 score is 1, indicating a perfect classification performance; the minimum is 0, indicating a very poor classification performance.

[0065] The specific values ​​of precision and recall are both related to the initial threshold. Precision refers to the proportion of samples predicted by the model as positive that are actually positive. It focuses on the quality of the prediction results, that is, the proportion of samples predicted as positive by the model that are correct. For example, in an earthquake prediction model, if the model predicts an earthquake on a certain day, precision is the probability that an earthquake will actually occur on that day.

[0066] Recall, also known as recall, refers to the proportion of samples that are correctly predicted as positive by the model out of all samples that are actually positive. It focuses on the model's coverage of positive samples, that is, how many of all real earthquakes the model was able to predict.

[0067] As an implementation method, the F1 score will be high only when both precision and recall are high. If a model has high precision but low recall, or high recall but low precision, its F1 score will not be very high. The F1 score can be expressed using the following formula:

[0068] Among them, F1-score represents the F1 score, p represents the precision, and r represents the recall rate.

[0069] For example, suppose there are 200 black samples under the cat label. When the initial threshold is 0.5, it means that the model rejects 100 pictures (identifies them as cats with a score greater than 0.5) and correctly rejects (recalls) 50 pictures. The recall rate r is 0.25, the precision rate p is 0.5, and the F1 score is 33%.

[0070] In the embodiment of the present application, a brute force algorithm can be used to sequentially traverse multiple initial thresholds and calculate the F1 scores corresponding to the multiple initial thresholds. The embodiment of the present application does not specifically limit the specific implementation of the above-mentioned preset intervals, and those skilled in the art can make appropriate adjustments based on actual conditions.

[0071] In step 2), the initial threshold corresponding to the maximum F1 score can be determined as the optimal threshold for the label. For each sample inferred by the model, if the probability score of a label obtained by inference is higher than the optimal threshold and consistent with the annotation, the model inference result can be considered correct. If the annotation is inconsistent or the probability scores corresponding to the inferred labels are all lower than the optimal threshold, the model inference result can be considered incorrect.

[0072] In the above solution, a brute force algorithm can be used to sequentially traverse multiple initial thresholds and calculate the F1 scores corresponding to each of these initial thresholds. This allows the initial threshold corresponding to the largest F1 score to be determined as the optimal threshold for that label. Since the largest F1 score maximizes the overall performance of the model, the optimal threshold corresponding to the largest F1 score can accurately filter out valid training data from the unlabeled data, further improving the efficiency of obtaining valid data.

[0073] Furthermore, based on the above embodiment, as another implementation method, the above step of calculating the optimal threshold value for achieving the best model performance evaluation index based on the inference data and the labeled data may specifically include the following steps:

[0074] The gradient descent algorithm is used to calculate the optimal threshold that maximizes the F1 score.

[0075] Specifically, the F1 score is a convex function about the optimal threshold and has an extreme value. Therefore, the optimal threshold that maximizes the F1 score can be calculated using a gradient descent algorithm. The embodiments of this application do not specifically limit the specific implementation of the above-mentioned gradient descent algorithm, and those skilled in the art may make appropriate adjustments based on actual circumstances.

[0076] In the above scheme, since the F1 score is a convex function, the gradient descent algorithm can be used to calculate the optimal threshold that maximizes the F1 score, thereby quickly calculating the optimal threshold corresponding to the label. The maximum F1 score maximizes the overall performance of the model. Therefore, the optimal threshold corresponding to the maximum F1 score can accurately filter out valid training data from the data to be labeled, further improving the efficiency of obtaining valid data.

[0077] Furthermore, based on the above embodiment, the above step S103 may specifically include the following steps:

[0078] Step 1) Determine the screening range corresponding to the tag based on the optimal threshold corresponding to the tag and a pre-set floating value.

[0079] Step 2) Input the data to be labeled into the initial model to obtain the second inference data output by the initial model.

[0080] Step 3) Determine whether the data to be labeled is training data based on the second inference data and the screening range.

[0081] Specifically, in the above step 1), a floating value corresponding to the tag may be pre-set, and the screening range corresponding to the tag may be determined according to the floating value corresponding to the tag and the optimal threshold corresponding to the tag.

[0082] In the above step 2), the data to be labeled can be input into the initial model, so as to obtain the second inference data output by the initial model. The second inference data can include a label and a probability value corresponding to the label.

[0083] In the above step 3), whether the data to be labeled is training data can be determined based on the second inference data and the screening range.

[0084] Among them, after labeling, if the label obtained by labeling the data is consistent with the label after model inference, then its probability score should be close to 1, and the actual score of model inference is within the screening range. This data is used as a training sample, which is conducive to the model learning capabilities that meet business expectations; similarly, after labeling, if the label obtained by labeling the data is inconsistent with the label after model inference, its probability score should be close to 0, and the actual score of model inference is within the screening range, so it is also a valid training sample.

[0085] This method only labels data near the optimal threshold, because when the model inference score of a certain data is close to 1, the probability of correct inference will be greatly improved. When the labeling result is consistent with the inference result and the score is close to 1, then the improvement of the model by this sample is low and it is an invalid sample. Similarly, when the model inference score of a certain data is close to 0, the probability of this sample being an invalid sample is high.

[0086] In the above scheme, the optimal threshold and floating value can be used to determine the filter range corresponding to the label, thereby filtering out valid training data from the unlabeled data based on the above filtering range. Since the optimal threshold can optimize the model performance evaluation index, setting the floating value can filter out a batch of high-accuracy training data.

[0087] Furthermore, based on the above embodiment, the second inference data may include: an inference label and an inference probability value corresponding to the inference label. In this case, the step of determining whether the data to be labeled is training data based on the second inference data and the screening range may specifically include the following steps:

[0088] Step 1) determines whether the inference probability value is within the screening range corresponding to the inference label.

[0089] Step 2): If the inference probability value is within the screening range corresponding to the inference label, the data to be labeled is determined as training data.

[0090] In the above scheme, if the inference probability value corresponding to the data to be labeled is within the screening range corresponding to the inference label, it means that the data to be labeled has a high improvement on the initial model. Therefore, it can be used as effective training data, thereby improving the efficiency of training the initial model.

[0091] Furthermore, based on the above embodiment, as an implementation method, the floating value may include a first value and a second value, and the first value is not equal to the second value. In this case, the step of determining the screening range based on the optimal threshold and the pre-set floating value may specifically include the following steps:

[0092] The sum of the optimal threshold value and the first value is determined as the upper limit of the screening range, and the difference between the optimal threshold value and the second value is determined as the lower limit of the screening range.

[0093] In the above solution, the pre-set floating value may include two different values, so that the resulting screening range can be more suitable for the current business scenario, thereby more accurately screening out effective training data from the data to be labeled.

[0094] Furthermore, based on the above embodiment, as another implementation method, the floating value may include a third value. In this case, the step of determining the screening range based on the optimal threshold and the preset floating value may specifically include the following steps:

[0095] The sum of the optimal threshold value and the third value is determined as the upper limit of the screening range, and the difference between the optimal threshold value and the third value is determined as the lower limit of the screening range.

[0096] Please refer to Figure 2, which is a structural block diagram of a training data acquisition device provided in an embodiment of the present application. The training data acquisition device 200 includes: an input module 201, which is used to input labeled data into an initial model to obtain first inference data; a calculation module 202, which is used to calculate the optimal threshold corresponding to a label based on the label and the first inference data corresponding to the labeled data labeled with the label; wherein the optimal threshold is used to distinguish between positive samples and negative samples input into the initial model; and a screening module 203, which is used to screen out training data corresponding to the label from the data to be labeled based on the optimal threshold corresponding to the label.

[0097] In the above scheme, before labeling the data to be labeled, valid training data can be filtered from the data to be labeled by calculating the optimal threshold corresponding to the label, so that only the above-screened training data can be labeled. Therefore, compared with the prior art method of directly labeling the huge amount of data to be labeled, the training data acquisition method provided in the embodiment of the application can only label the valid training data in the data to be labeled, thereby improving the efficiency of obtaining valid data.

[0098] Furthermore, based on the above embodiment, the calculation module 202 is specifically used to: calculate the optimal threshold value for the model performance evaluation indicator to reach the best according to the first inference data and the label.

[0099] In the above scheme, during the process of determining the optimal threshold, the optimal threshold that optimizes the model performance evaluation indicator can be calculated, thereby accurately selecting valid training data from the data to be labeled. Furthermore, since valid training data can be selected from the data to be labeled, the amount of training data that needs to be labeled can be reduced, thereby further improving the efficiency of obtaining valid data.

[0100] Furthermore, based on the above embodiment, the calculation module 202 is further configured to: traverse multiple initial thresholds based on preset intervals, calculate the F1 scores under the multiple initial thresholds; and determine the initial threshold corresponding to the maximum F1 score as the optimal threshold corresponding to the label.

[0101] In the above solution, a brute force algorithm can be used to sequentially traverse multiple initial thresholds and calculate the F1 scores corresponding to each of these initial thresholds. This allows the initial threshold corresponding to the largest F1 score to be determined as the optimal threshold for that label. Since the largest F1 score maximizes the overall performance of the model, the optimal threshold corresponding to the largest F1 score can accurately filter out valid training data from the unlabeled data, further improving the efficiency of obtaining valid data.

[0102] Furthermore, based on the above embodiment, the calculation module 202 is further configured to calculate an optimal threshold for maximizing the F1 score using a gradient descent algorithm.

[0103] In the above scheme, since the F1 score is a convex function, the gradient descent algorithm can be used to calculate the optimal threshold that maximizes the F1 score, thereby quickly calculating the optimal threshold corresponding to the label. The maximum F1 score maximizes the overall performance of the model. Therefore, the optimal threshold corresponding to the maximum F1 score can accurately filter out valid training data from the data to be labeled, further improving the efficiency of obtaining valid data.

[0104] Furthermore, based on the above embodiment, the screening module 203 is specifically used to: determine the screening range corresponding to the label based on the optimal threshold corresponding to the label and a pre-set floating value; input the data to be labeled into the initial model to obtain the second inference data output by the initial model; determine whether the data to be labeled is the training data based on the second inference data and the screening range.

[0105] In the above scheme, the optimal threshold and floating value can be used to determine the filter range corresponding to the label, thereby filtering out valid training data from the unlabeled data based on the above filtering range. Since the optimal threshold can optimize the model performance evaluation index, setting the floating value can filter out a batch of high-accuracy training data.

[0106] Furthermore, based on the above embodiment, the second inference data includes: an inference label and an inference probability value corresponding to the inference label; the screening module 203 is also used to: determine whether the inference probability value is within the screening range corresponding to the inference label; if the inference probability value is within the screening range corresponding to the inference label, the data to be labeled is determined as the training data.

[0107] In the above scheme, if the inference probability value corresponding to the data to be labeled is within the screening range corresponding to the inference label, it means that the data to be labeled has a high improvement on the initial model. Therefore, it can be used as effective training data, thereby improving the efficiency of training the initial model.

[0108] Further, based on the above embodiment, the floating value includes a first value and a second value, and the first value is not equal to the second value; the screening module 203 is also used to: determine the sum of the optimal threshold and the first value as the upper limit of the screening range, and determine the difference between the optimal threshold and the second value as the lower limit of the screening range.

[0109] In the above solution, the pre-set floating value may include two different values, so that the resulting screening range can be more suitable for the current business scenario, thereby more accurately screening out effective training data from the data to be labeled.

[0110] Please refer to Figure 3, which is a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 300 includes: at least one processor 301, at least one communication interface 302, at least one memory 303 and at least one communication bus 304. Among them, the communication bus 304 is used to realize direct connection and communication between these components, the communication interface 302 is used to communicate signaling or data with other node devices, and the memory 303 stores machine-readable instructions executable by the processor 301. When the electronic device 300 is running, the processor 301 communicates with the memory 303 through the communication bus 304, and the machine-readable instructions are called by the processor 301 to execute the above-mentioned method for obtaining training data.

[0111] For example, the processor 301 of an embodiment of the present application reads a computer program from the memory 303 via the communication bus 304 and executes the computer program to implement the following method: Step S101: Inputting labeled data into an initial model to obtain first inference data. Step S102: For a label, calculating an optimal threshold corresponding to the label based on the label and the first inference data corresponding to the labeled data labeled with the label. Step S103: Filtering training data corresponding to the label from the data to be labeled based on the optimal threshold corresponding to the label.

[0112] Among them, the processor 301 includes one or more, which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 301 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a special-purpose processor, including a neural network processor (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Moreover, when there are multiple processors 301, some of them can be general-purpose processors and the other part can be special-purpose processors.

[0113] The memory 303 includes one or more, which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0114] It will be understood that the structure shown in FIG3 is for illustration only, and the electronic device 300 may also include more or fewer components than those shown in FIG3 , or have a configuration different from that shown in FIG3 . The components shown in FIG3 may be implemented using hardware, software, or a combination thereof. In the embodiments of the present application, the electronic device 300 may be, but is not limited to, a physical device such as a desktop computer, a laptop computer, a smart phone, a smart wearable device, an in-vehicle device, and may also be a virtual device such as a virtual machine. In addition, the electronic device 300 is not necessarily a single device, but may also be a combination of multiple devices, such as a server cluster, and the like.

[0115] An embodiment of the present application further provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed by a computer, the computer executes the method for acquiring training data described in the aforementioned method embodiment.

[0116] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] Furthermore, units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of this embodiment.

[0118] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0119] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0120] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0121] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for obtaining training data, characterized in that: include: Input the labeled data into the initial model to obtain the first inference data; For a label, calculating an optimal threshold corresponding to the label based on the label and first inference data corresponding to the labeled data labeled with the label; wherein the optimal threshold is used to distinguish positive samples and negative samples input into the initial model; The training data corresponding to the label is filtered out from the data to be labeled according to the optimal threshold corresponding to the label.

2. The method for obtaining training data according to claim 1, wherein: The calculating, based on the label and the first inference data corresponding to the labeled data labeled with the label, an optimal threshold corresponding to the label includes: An optimal threshold value is calculated based on the first inference data and the label so that the model performance evaluation indicator reaches the best.

3. The method for obtaining training data according to claim 2, wherein: The step of calculating, based on the first inference data and the label, an optimal threshold value for achieving the best model performance evaluation indicator includes: Traversing multiple initial thresholds based on preset intervals, and calculating F1 scores under the multiple initial thresholds; The initial threshold corresponding to the maximum F1 score is determined as the optimal threshold corresponding to the label.

4. The method for obtaining training data according to claim 2, wherein: The step of calculating, based on the first inference data and the label, an optimal threshold value for achieving the best model performance evaluation indicator includes: The gradient descent algorithm is used to calculate the optimal threshold that maximizes the F1 score.

5. The method for obtaining training data according to any one of claims 1 to 4, characterized in that: The step of selecting training data corresponding to the label from the data to be labeled according to the optimal threshold corresponding to the label includes: Determine the screening range corresponding to the tag according to the optimal threshold corresponding to the tag and a pre-set floating value; Inputting the data to be labeled into the initial model to obtain second inference data output by the initial model; Determine whether the data to be labeled is the training data according to the second inference data and the screening range.

6. The method for obtaining training data according to claim 5, characterized in that: The second inference data includes: an inference label and an inference probability value corresponding to the inference label; The determining whether the data to be labeled is the training data according to the second inference data and the screening range includes: Determining whether the inference probability value is within the screening range corresponding to the inference label; If the inference probability value is within the screening range corresponding to the inference label, the The labeled data is determined as the training data.

7. The method for obtaining training data according to claim 5, characterized in that: The floating value includes a first value and a second value, and the first value is not equal to the second value; Determining the screening range according to the optimal threshold and the pre-set floating value includes: The sum of the optimal threshold value and the first value is determined as the upper limit of the screening range, and the difference between the optimal threshold value and the second value is determined as the lower limit of the screening range.

8. A device for acquiring training data, characterized in that: include: An input module, configured to input the labeled data into the initial model to obtain first inference data; a calculation module configured to calculate, for a label, an optimal threshold corresponding to the label based on the label and first inference data corresponding to the labeled data annotated with the label; wherein the optimal threshold is used to distinguish between positive samples and negative samples input into the initial model; The screening module is used to screen out the training data corresponding to the label from the data to be labeled according to the optimal threshold value corresponding to the label.

9. An electronic device, characterized in that: include: processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores computer program instructions that can be executed by the processor, and the processor calls the computer program instructions to execute the method for obtaining training data according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a computer, the computer is caused to execute the method for obtaining training data according to any one of claims 1 to 7.

Images