Training sample acquisition method, device, computer equipment and readable storage medium

Abstract

The present application relates to a method, apparatus, computer device, and readable storage medium for obtaining training samples. The method includes obtaining an initial sample set, training a target model and a discrimination model based on the labeled samples in the initial sample set; inputting the unlabeled samples in the initial sample set into the target model to obtain labeled data; determining input parameters of the discrimination model according to the labeled data and the unlabeled samples, and inputting the input parameters into the discrimination model to obtain an evaluation score; and determining whether to update the labeled samples in the initial sample set according to the labeled data and the unlabeled samples. The method for obtaining training samples provided by the present application does not require staff to label a large number of training samples, which can reduce the consumption of manpower and thus improve the speed of obtaining training samples.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and particularly to a method, device, computer device, and readable storage medium for obtaining training samples. Background Art

[0002] Currently, the method of implementing corresponding functions through a pre-trained deep learning model has been widely used. For example, image segmentation of an image is achieved through a trained image segmentation model. To obtain an accurate deep learning model, a large number of high-quality labeled samples are required.

[0003] In traditional techniques, it is usually the staff who label the training samples to obtain labeled samples. However, labeling a large number of training samples requires a lot of manpower. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device, computer device, and readable storage medium for obtaining training samples to solve the above technical problems.

[0005] In a first aspect, an embodiment of this application provides a method for obtaining training samples, including

[0006] Obtaining an initial sample set, and training a target model and a discriminative model based on the labeled samples in the initial sample set; the discriminative model is used to evaluate the target model;

[0007] Inputting the unlabeled samples in the initial sample set into the target model to obtain labeled data;

[0008] Determining the input parameters of the discriminative model according to the labeled data and the unlabeled samples; and inputting the input parameters into the discriminative model to obtain an evaluation score;

[0009] Determining whether to update the labeled samples of the initial sample set according to the labeled data and the unlabeled samples based on the evaluation score.

[0010] In one embodiment, determining whether to update the labeled samples of the initial sample set according to the evaluation score includes:

[0011] If the evaluation score is greater than or equal to a preset threshold, then using the samples composed of the unlabeled samples and the labeled data as new labeled samples to be added to the initial sample set, and the preset threshold is used to characterize the accuracy of the output result of the target model.

[0012] In one embodiment, determining the input parameters of the discriminative model according to the labeled data and the unlabeled samples includes:

[0013] Using the unlabeled samples and the labeled data as input parameters;

[0014] And / or, perform feature extraction processing on unlabeled samples and labeled data, and use the extracted features as input parameters.

[0015] In one embodiment, the discrimination model includes a deep learning model and / or a machine learning model.

[0016] In one embodiment, when the discrimination model includes a deep learning model and a machine learning model, input the input parameters into the discrimination model to obtain an evaluation score, including:

[0017] Input the input parameters into the deep learning model and the machine learning model respectively;

[0018] Perform a mean value processing on the outputs of the deep learning model and the machine learning model to obtain the evaluation score.

[0019] In one embodiment, the training sample acquisition method further includes:

[0020] Perform sample evaluation on multiple initial labeled samples, and obtain labeled samples from the multiple initial labeled samples based on the evaluation results.

[0021] In one embodiment, performing sample evaluation on multiple initial labeled samples and obtaining the labeled samples from the multiple initial labeled samples based on the evaluation results includes:

[0022] Perform the following screening process on multiple initial labeled samples:

[0023] Determine L sets of sample sets according to multiple initial labeled samples, determine a training set and a validation set according to the L sets of sample sets, train an initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models; both L and N are integers greater than zero;

[0024] Input the multiple initial labeled samples into the N sets of sample screening models, determine the evaluation results according to the outputs of the N sets of sample screening models, and determine the abnormal samples among the multiple initial labeled samples according to the evaluation results;

[0025] Update the multiple initial labeled samples according to the abnormal samples, and perform the screening process on the updated multiple initial labeled samples until there are no abnormal samples among the multiple initial labeled samples.

[0026] In one embodiment, determining a training set and a validation set according to the L sets of sample sets, and training an initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models includes:

[0027] Traverse each set of sample sets in the L sets of sample sets, use one set of sample sets as the validation set, and use the remaining sample sets in the L sets of sample sets as the training set;

[0028] Train an initial model using a training set to obtain a training result, and validate the trained initial model using a validation set to obtain a validation result;

[0029] Determine a sample screening model corresponding to a set of sample sets according to the training result, the validation result, and a preset constraint condition;

[0030] Determine N sets of sample screening models according to the sample screening models corresponding to each set of sample sets in the L sets of sample sets.

[0031] In one embodiment, the training sample acquisition method further includes:

[0032] Receive a subjective evaluation score for the labeled data and a modification result for the labeled data; the subjective evaluation score is used to represent the subjective evaluation of the labeled data;

[0033] If the subjective evaluation score is less than the evaluation score, train the target model and the discriminative model according to the modification result, the unlabeled samples, and the subjective evaluation score.

[0034] In a second aspect, an embodiment of the present application provides a training sample acquisition device, including:

[0035] An acquisition module, configured to acquire an initial sample set, and train a target model and a discriminative model according to the labeled samples in the initial sample set; the discriminative model is used to evaluate the target model;

[0036] A first determination module, configured to input the unlabeled samples in the initial sample set into the target model to obtain labeled data;

[0037] A second determination module, configured to determine input parameters of the discriminative model according to the labeled data and the unlabeled samples; and input the input parameters into the discriminative model to obtain an evaluation score;

[0038] A third determination module, configured to determine whether to update the labeled samples of the initial sample set according to the labeled data and the unlabeled samples according to the evaluation score.

[0039] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method provided in the above embodiment are implemented.

[0041] The embodiments of the present application provide a method, apparatus, computer device, and readable storage medium for obtaining training samples. The method includes obtaining an initial sample set, training a target model and a discrimination model based on the labels in the initial sample set; inputting the unlabeled samples in the initial sample set into the target model to obtain labeled data; determining the input parameters of the discrimination model according to the labeled data and the unlabeled samples; inputting the input parameters into the discrimination model to obtain an evaluation score; and determining whether to update the labeled samples of the initial sample set according to the labeled data and the unlabeled samples. The method for obtaining training samples provided by the embodiments of the present application can label unlabeled samples by using the target model trained with a small number of labeled samples in the initial sample set, so as to obtain a large number of labeled samples, that is, training samples. In this way, it is not necessary to manually label a large number of unlabeled samples, which can reduce the consumption of manpower and improve the efficiency of obtaining training samples. In addition, the embodiments of the present application also use the discrimination model trained with labeled samples to discriminate the accuracy of labeling unlabeled samples by using the target model, so as to improve the accuracy of the finally determined labeled samples. By using the method for obtaining training samples provided by the embodiments of the present application, a large number of high-quality labeled samples can be obtained with a small number of labeled samples. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a schematic flowchart of the steps of the method for obtaining training samples provided by an embodiment of the present application;

[0044] Figure 2 It is a schematic flowchart of the steps of the method for obtaining training samples provided by an embodiment of the present application;

[0045] Figure 3 It is a schematic flowchart of the steps of the method for obtaining training samples provided by an embodiment of the present application;

[0046] Figure 4 It is a schematic flowchart of the steps of the method for obtaining training samples provided by an embodiment of the present application;

[0047] Figure 5 It is a schematic flowchart of the steps of the method for obtaining training samples provided by an embodiment of the present application;

[0048] Figure 6 It is a schematic flowchart of the steps of the method for obtaining training samples provided by an embodiment of the present application;

[0049] Figure 7 A structural schematic diagram of a training sample acquisition device provided in an embodiment of the present application;

[0050] Figure 8 A structural schematic diagram of a computer device provided in an embodiment of the present application. Detailed implementation manners

[0051] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will describe the detailed implementation manners of the present application with reference to the accompanying drawings. Many specific details are set forth in the following description to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0052] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the described objects and do not have any sequential or technical meanings.

[0053] Currently, methods for implementing corresponding functions through pre-trained deep learning models are increasingly widely used. To obtain an accurate deep learning model, a large number of high-quality labeled samples are required. In the traditional technology, it is usually necessary for staff to label training samples to obtain labeled samples. However, labeling a large number of training samples requires a lot of manpower. For example, although the technology of using a trained image segmentation model to implement medical image segmentation has made great progress, this technology still relies on high-quality annotation data (that is, the annotation data is very accurate). There are difficulties in obtaining medical image annotation data for model training (hospital clinical images and annotation data are required, which involves patient privacy information and requires signing a strict data use agreement with the hospital), difficulties in annotation (there are differences in the annotation habits of different hospitals and different doctors, and doctors will adjust the annotation methods according to the actual situation of patients, so that the data obtained from the hospital cannot be directly used for image segmentation training), and high annotation costs (highly senior doctors are required to modify and uniformly annotate medical images), which requires a lot of manpower. In response to this, the present application provides a training sample acquisition method.

[0054] The training sample acquisition method provided by the present application can be implemented through a computer device. The computer device includes, but is not limited to, a control chip, a personal computer, a laptop computer, a smart phone, a tablet computer, and a portable wearable device. The method provided by the present application can be implemented through JAVA software and can also be applied to other software.

[0055] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the technical problem. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0056] Please refer to Figure 1 , an embodiment of this application provides a method for obtaining training samples. This embodiment of the application takes a computer device as the execution subject to specifically describe the method for obtaining training samples, and the steps include:

[0057] Step 100: Obtain an initial sample set, and train a target model and a discrimination model based on the labeled samples in the initial sample set; the discrimination model is used to evaluate the target model.

[0058] The computer device obtains an initial sample set, which includes labeled samples and unlabeled samples. The labeled samples can be a small number of high-quality labeled samples annotated by staff. After the computer device obtains the initial sample set, it uses the labeled samples in the initial sample set to train the initial model of the target model to obtain the target model, and trains the initial model of the discrimination model to obtain the discrimination model. The initial model of the target model can be a neural network model, that is to say, training the neural network model can obtain the target model. The initial model of the discrimination model can be a deep learning model or a machine learning model, that is to say, training the deep learning model or the machine learning model can obtain the discrimination model. The target model refers to the model required by the staff. Assuming that the staff needs to segment medical images, the target model is an image segmentation model; assuming that the staff needs to recognize medical images, the target model is a recognition model. The discrimination model is used to evaluate the target model. In other words, the discrimination model is used to determine whether the trained target model can accurately implement its function. If the target model is an image segmentation model, the discrimination model is used to determine whether the image segmentation model can accurately segment medical images. The initial sample set can be pre-stored by the staff in the memory of the computer device, and the computer device can directly obtain it from the memory when needed. This embodiment does not limit the method for obtaining the initial sample set and the type of the target model, as long as its function can be realized.

[0059] Step 110: Input the unlabeled samples in the initial sample set into the target model to obtain labeled data.

[0060] After the computer device trains a target model using the labeled samples in the initial sample set, it inputs the unlabeled samples in the initial sample set into the target model to annotate the unlabeled samples and obtain annotation data for the unlabeled samples. Only after using the annotation data to annotate the unlabeled samples can the corresponding labeled samples after annotating the unlabeled samples be obtained.

[0061] Step 120: Determine the input parameters of the discriminative model according to the annotation data and the unlabeled samples; and input the input parameters into the discriminative model to obtain an evaluation score.

[0062] After the computer device obtains the annotation data of the unlabeled samples, it determines the input parameters of the discriminative model according to the annotation data and the unlabeled samples. After inputting the input parameters into the discriminative model, according to the output result of the discriminative model, it determines whether the labeled samples obtained by using the target model to annotate the unlabeled samples are accurate. In this embodiment, the output result of the discriminative model is an evaluation score, and different evaluation scores can represent the accuracy of the labeled samples obtained by using the target model to annotate the unlabeled samples. This embodiment does not limit the method for determining the input parameters of the discriminative model according to the annotation data and the unlabeled samples, as long as its function can be realized.

[0063] Step 130: Determine whether to update the labeled samples in the initial sample set according to the evaluation score and the annotation data and the unlabeled samples.

[0064] After the computer device obtains the evaluation score, it determines whether to update the labeled samples in the initial sample set according to the evaluation score and the annotation data and the unlabeled samples. In other words, the computer device determines whether the annotation data obtained according to the target model is accurate according to the evaluation score. If it is accurate, it obtains the labeled samples according to the annotation data and the unlabeled samples, adds the labeled samples to the initial sample set, and increases the number of labeled samples in the initial sample set. This embodiment does not limit the method for determining whether the annotation data obtained through the target model is accurate according to the evaluation score, as long as its function can be realized.

[0065] The training sample acquisition method provided by the embodiments of the present application obtains an initial sample set, and trains a target model and a discrimination model according to the labels in the initial sample set; inputs the unlabeled samples in the initial sample set into the target model to obtain labeled data; determines the input parameters of the discrimination model according to the labeled data and the unlabeled samples; inputs the input parameters into the discrimination model to obtain an evaluation score; and determines whether to update the labeled samples in the initial sample set according to the labeled data and the unlabeled samples. The training sample acquisition method provided by the embodiments of the present application can label unlabeled samples by using the target model trained with a small number of labeled samples in the initial sample set, so as to obtain a large number of labeled samples, that is, training samples. In this way, there is no need to manually label a large number of unlabeled samples, which can reduce the consumption of human resources and improve the efficiency of obtaining training samples. In addition, the embodiments of the present application also use the discrimination model trained with labeled samples to discriminate the accuracy of the labeled unlabeled samples by using the target model, so as to improve the accuracy of the finally determined labeled samples. By using the training sample acquisition method provided by the embodiments of the present application, a large number of high-quality labeled samples can be obtained with a small number of labeled samples.

[0066] In one embodiment, a possible implementation manner for determining whether to update the labeled samples in the initial sample set according to the evaluation score includes:

[0067] If the evaluation score is greater than or equal to a preset threshold, the samples composed of the unlabeled samples and the labeled data are used as new labeled samples and added to the initial sample set, and the preset threshold is used to characterize the accuracy of the output result of the target model.

[0068] After obtaining the output result of the discrimination model, that is, the evaluation score, the computer device compares the evaluation score with the preset threshold, and the preset threshold is used to characterize the accuracy of the output result of the target model. The preset threshold can be a value pre-stored in the computer device by the staff. If the computer device determines that the evaluation score is greater than or equal to the preset threshold, it means that the labeled data obtained by the target model is accurate, and new labeled samples (i.e., high-quality labeled samples) are formed according to the unlabeled samples and the labeled data, and the labeled samples are added to the initial sample set to increase the number of labeled samples in the initial sample set. If the computer device determines that the evaluation score is less than the preset threshold, it means that the labeled data obtained by the target model is inaccurate, and the labeled samples composed of the unlabeled samples and the labeled data at this time cannot be added to the initial sample set.

[0069] In this embodiment, the method for determining whether the labeled data obtained by the target model is accurate is simple and easy to understand and implement.

[0070] Please refer to Figure 2, in one embodiment, it relates to possible implementation manners of determining input parameters of a discrimination model based on labeled data and unlabeled samples, including:

[0071] Step 200: Use the unlabeled samples and the labeled data as input parameters.

[0072] After obtaining the labeled data, the computer device can directly use the labeled data and the unlabeled samples corresponding to the labeled data as the input parameters of the discrimination model. That is to say, after obtaining the labeled data, the computer device can, without performing relevant processing on the unlabeled samples and the labeled data, use them as the input parameters of the discrimination model and input them into the discrimination model.

[0073] And / or, Step 210: Perform feature extraction processing on the unlabeled samples and the labeled data, and use the extracted features as input parameters.

[0074] After obtaining the labeled data, the computer device can perform feature extraction processing on the unlabeled samples and the labeled data, and use the extracted features as input parameters. That is to say, after obtaining the labeled data, the computer device can first perform feature extraction on the labeled data and the unlabeled data, and use the extracted features as the input parameters of the discrimination model and input them into the discrimination model. The extracted features are related to the types of the labeled data and the unlabeled samples. If the unlabeled samples are medical images and the labeled data are segmentation data of the medical images, the extracted features may include features such as the size and shape of the medical images, as well as features such as the position, shape, and area of the labeled data. In this embodiment, no specific feature extraction method is limited as long as its function can be realized.

[0075] In this embodiment, two methods for determining the input parameters of the discrimination model based on the labeled data and the unlabeled samples are proposed. Users can select according to the actual situation, which can improve the applicability of the training sample acquisition method.

[0076] In one embodiment, the discrimination model includes a deep learning model and / or a machine learning model.

[0077] If there are two modes for the input parameters of the discrimination model provided in the above embodiment, the corresponding discrimination model also includes two models, namely a deep learning model and a machine learning model. The input parameters corresponding to the deep learning model are unlabeled samples and labeled data, and the input parameters corresponding to the machine learning model are the features after performing feature extraction processing on the unlabeled samples and the labeled data. In this embodiment, the discrimination model can include one of the deep learning model and the machine learning model, or can include both the deep learning model and the machine learning model.

[0078] In this embodiment, for the types of the discrimination model provided, users can select according to the actual situation, which can improve the applicability of the training sample acquisition method.

[0079] In one embodiment, when the discrimination model includes a deep learning model, the unlabeled samples and the labeled data are used as input parameters and input into the deep learning model, and the output of the deep learning model is the evaluation score.

[0080] When the discrimination model includes a machine learning model, the features obtained by performing feature extraction processing on the unlabeled samples and the labeled data are used as input parameters and input into the machine learning model, and the output of the machine learning model is the evaluation score.

[0081] Please refer to Figure 3 , in one embodiment, which relates to a possible implementation manner of inputting input parameters into a discrimination model to obtain an evaluation score when the discrimination model includes a deep learning model and a machine learning model, including:

[0082] Step 300: Input the input parameters into the deep learning model and the machine learning model respectively.

[0083] Step 310: Perform a mean value processing on the output of the deep learning model and the output of the machine learning model to obtain the evaluation score.

[0084] The computer device inputs the unlabeled samples and the labeled data as input parameters into the deep learning model to obtain the output result of the deep learning model, denoted as the first evaluation score. The computer device inputs the features obtained by performing feature extraction processing on the unlabeled samples and the labeled data as input parameters into the machine learning model to obtain the output result of the machine learning model, denoted as the second evaluation score. By performing a mean value processing on the first evaluation score and the second evaluation score, the evaluation score can be obtained.

[0085] In this embodiment, the discrimination model includes a deep learning model and a machine learning model, and the evaluation score obtained according to the mean value of the output results of these two models is more accurate, so that high-quality labeled samples can be obtained, and further the reliability and practicality of the training sample acquisition method can be improved.

[0086] In one embodiment, the training sample acquisition method further includes:

[0087] Perform sample evaluation on multiple initial labeled samples, and obtain labeled samples from the multiple initial labeled samples based on the evaluation results.

[0088] Multiple initial labeled samples may refer to a small number of artificially labeled initial labeled samples obtained by staff. The multiple initial labeled samples can be pre-stored in the memory of a computer device, and the computer device can directly obtain them from the memory when needed. The multiple initial labeled samples can also be pre-stored in a specific storage device, and when the computer device needs them, it can obtain them from the specific storage device. This embodiment does not limit the method for the computer device to obtain multiple initial labeled samples, as long as its function can be realized.

[0089] After the computer device obtains multiple initial labeled samples, it obtains labeled samples from the multiple initial labeled samples according to the evaluation results. In other words, the computer device evaluates each initial labeled sample and determines whether each initial labeled sample is a high-quality labeled sample according to the evaluation results. If it is determined according to the evaluation results that the initial labeled sample is a high-quality labeled sample, then the initial labeled sample is added to the initial sample set as a labeled sample. If it is determined according to the evaluation results that the initial labeled sample is not a high-quality labeled sample, then the staff determines whether to re-label the initial labeled sample or reject it. If it is re-labeled, the labeled sample is re-evaluated. This embodiment does not limit the method for the computer device to evaluate multiple initial labeled samples, as long as its function can be realized.

[0090] In this embodiment, by evaluating the multiple initial labeled samples obtained, it can be ensured that the quality of the obtained labeled samples is higher, so that the target model and the discriminant model trained by the labeled samples are more accurate, the labeling of unlabeled samples is more accurate, and thus a large number of high-quality labeled samples, that is, training samples, can be obtained.

[0091] Please refer to Figure 4 In one embodiment, there is provided a possible implementation manner for evaluating multiple initial labeled samples and obtaining labeled samples from the multiple initial labeled samples based on the evaluation results, including:

[0092] After the computer device has multiple initial labeled samples, it performs the following screening process on the multiple initial labeled samples:

[0093] Step 400: Determine L groups of sample sets according to the multiple initial labeled samples, determine a training set and a validation set according to the L groups of sample sets, and train the initial model corresponding to the target model according to the training set and the validation set to obtain N groups of sample screening models; both L and N are integers greater than zero.

[0094] The computer device performs grouping processing on the multiple initial labeled samples obtained to obtain L groups of sample sets. Specifically, the computer device performs an equal division process on the multiple initial labeled samples, that is, the number of initial labeled samples in each group of sample sets is the same.

[0095] After the computer device determines the L sets of sample sets, it determines the training set and the validation set according to the L sets of sample sets, trains the initial model corresponding to the target model according to the training set, validates the trained initial model according to the validation set, and obtains N sets of sample screening models. The N sets of sample screening models can be used to screen multiple initial labeled samples. This embodiment does not limit the specific method for determining the N sets of sample screening models, as long as its function can be realized.

[0096] Please refer to Figure 5 , in one embodiment, it relates to a possible implementation manner of determining the training set and the validation set according to the L sets of sample sets, and training the initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models, including:

[0097] Step 500: Traverse each set of sample sets in the L sets of sample sets, use one set of sample sets as the validation set, and use the remaining sample sets in the L sets of sample sets as the training set.

[0098] After the computer device obtains the L sets of sample sets, it traverses each set of sample sets in the L sets of sample sets, uses one set of sample sets as the validation set, and the remaining sample sets in the L sets of sample sets as the training set. That is to say, each set of sample sets in the L sets of sample sets can be used as the training set and the validation set.

[0099] Suppose there are 3 sets of sample sets, which are respectively denoted as the first sample set, the second sample set, and the third sample set. The first sample set is used as the validation set, and the second sample set and the third sample set are used as the training set; the second sample set is used as the validation set, and the first sample set and the third sample set are used as the training set; the third sample set is used as the validation set, and the first sample set and the second sample set are used as the training set.

[0100] Step 510: Use the training set to train the initial model to obtain a training result, and use the validation set to validate the trained initial model to obtain a validation result.

[0101] The computer device first trains the initial model corresponding to the target model according to the obtained training set to obtain a training result; then uses the validation set corresponding to the training set to validate the trained initial model to obtain a validation result. This embodiment does not limit the specific training process and validation process, as long as its function can be realized.

[0102] Step 520: Determine the sample screening model corresponding to one set of sample sets according to the training result, the validation result, and the preset constraint conditions.

[0103] Based on the training results obtained from training, the verification results obtained from verification, and the preset constraint conditions set in advance, a computer device can determine a sample screening model corresponding to a set of sample sets. The preset constraint conditions are related to the number of sample screening models corresponding to a set of sample sets to be obtained. The preset constraint conditions can include constraints on the training results and constraints on the verification results. This embodiment does not limit the specific method for determining a sample screening model corresponding to a set of sample sets, as long as its function can be achieved.

[0104] Step 530: Determine N groups of sample screening models according to the sample screening models corresponding to each group of sample sets in the L groups of sample sets.

[0105] After the computer device traverses each group of sample sets in the L groups of sample sets and determines the sample screening model corresponding to each group of sample sets, it can determine N groups of sample screening models. For the sample screening model corresponding to each group of sample sets, N sample screening models can be obtained. Then, for the sample screening models corresponding to the L groups of sample sets, N groups of sample screening models can be obtained. That is to say, the first sample screening model in the sample screening model corresponding to each group of sample sets is used as the first group of sample screening models, the second sample screening model in the sample screening model corresponding to each group of sample sets is used as the second group of sample screening models, and so on. The Nth sample screening model in the sample screening model corresponding to each group of sample sets is used as the Nth group of sample screening models. That is, the number of groups of sample screening models corresponding to the L groups of sample sets is related to the number of sample screening models corresponding to a group of sample sets.

[0106] In an optional embodiment, the preset constraint conditions include constraints corresponding to the verification results and constraints corresponding to the training results. The constraint corresponding to the verification result can be that the error between the verification result and the training result reaches a preset threshold. The constraint corresponding to the training result can be that the error between the verification result and the training result no longer decreases or the overlap coefficient between the verification result and the training result no longer increases. In this way, according to the training results, the verification results, and the preset constraint conditions, 2 sample screening models corresponding to a set of sample sets can be obtained. Then, according to the sample screening models corresponding to each group of sample sets in the L groups of sample sets, 2 groups of sample screening models can be obtained.

[0107] Step 410: Input multiple initial labeled samples into the N groups of sample screening models, determine an evaluation result according to the outputs of the N groups of sample screening models, and determine the abnormal samples among the multiple initial labeled samples according to the evaluation result.

[0108] After obtaining N sets of sample screening models, the computer device inputs each initial labeled sample in multiple initial labeled samples into the N sets of sample screening models respectively, determines the evaluation result for each initial labeled sample according to the outputs of the N sets of sample screening models, and determines the abnormal samples among the multiple initial labeled samples according to the evaluation result. This embodiment does not limit the method for determining the abnormal samples among the multiple initial labeled samples as long as its function can be achieved.

[0109] In an alternative embodiment, the computer device inputs each initial labeled sample into the N sets of sample screening models. For each set of sample screening models, there is an evaluation result. The evaluation results of the N sets of sample screening models are comprehensively evaluated (if the evaluation result corresponding to the sample screening model is a numerical value, the comprehensive evaluation is to calculate the mean value of the evaluation results of the N sets of sample screening models). The comprehensive evaluations corresponding to all the initial labeled samples are sorted from largest to smallest, and the initial sample label corresponding to the worst comprehensive evaluation is used as the abnormal sample. The worst comprehensive evaluation can refer to the last 10%-15% (the 2 or 3 with smaller mean values) among all the comprehensive evaluations. If the comprehensive evaluations corresponding to all the initial labeled samples are less than the preset condition, it means that there are no abnormal samples among the multiple initial labeled samples. The preset condition can be set by the staff according to the actual situation. Specifically, the preset condition can be less than the mean threshold. That is to say, if the mean value of the evaluation results corresponding to all the initial labeled samples is less than the threshold, it means that there are no abnormal samples among the multiple initial labeled samples.

[0110] Step 420: Update the multiple initial labeled samples according to the abnormal samples, and perform screening processing on the updated multiple initial labeled samples until the multiple initial labeled samples do not contain abnormal samples.

[0111] When the computer device determines the abnormal samples existing in the multiple initial labeled samples, it can remove the abnormal samples from the multiple initial labeled samples to obtain a new set of multiple initial labeled samples; or the staff can re-label the abnormal samples and add the re-labeled samples to the multiple initial labeled samples to obtain a new set of multiple initial labeled samples. After the computer device obtains the new set of multiple initial labeled samples, it performs the above screening processing process on the new set of multiple initial labeled samples until the multiple initial labeled samples do not contain abnormal samples. At this time, the multiple initial labeled samples are the labeled samples.

[0112] The method provided by this embodiment can improve the quality of the labeled samples in the initial sample set, so as to obtain accurate target models and discrimination models, and further can accurately label the unlabeled samples to obtain a large number of high-quality labeled samples, that is, training samples.

[0113] Please refer to Figure 6, in one embodiment, the training sample acquisition method further includes:

[0114] Step 600, receiving a subjective evaluation score for the labeled data and the modification result of the labeled data; the subjective evaluation score is used to represent the subjective evaluation of the labeled data.

[0115] The subjective evaluation score is used to represent the subjective evaluation of the labeled data, that is, the evaluation of the labeled data by the staff. The modification result of the labeled data, that is, the data after the staff modifies the labeled data. In other words, the staff evaluates the labeled data obtained by the target model and modifies it, and inputs the subjective evaluation score and the modified labeled data into the computer device.

[0116] Step 610, if the subjective evaluation score is less than the evaluation score, training the target model and the discriminant model according to the modification result, the unlabeled samples, and the subjective evaluation score.

[0117] After obtaining the subjective evaluation score, the computer device compares it with the evaluation score obtained by the discriminant model. If it is determined that the subjective evaluation score is less than the evaluation score, it means that the target model and the discriminant model may be inaccurate. Then, the target model is trained according to the labeled samples composed of the modification result and the unlabeled samples, and the discriminant model is trained according to the labeled samples composed of the modification result and the unlabeled samples and the subjective evaluation score, so as to optimize the target model and the discriminant model. If it is determined that the subjective evaluation score is greater than or equal to the evaluation score, it means that the target model and the discriminant model are relatively accurate.

[0118] In this embodiment, without introducing a large number of labeled samples, by receiving the evaluation and modification of the labeled data by the staff in daily situations to optimize the target model and the discriminant model, more accurate target model and discriminant model can be obtained, thereby improving the practicability of the training sample acquisition method.

[0119] It should be understood that although the steps in the flowchart in the figure are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the figure may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0120] Please refer to Figure 7, an embodiment of the present application provides a training sample acquisition device 10, which includes an acquisition module 11, a first determination module 12, a second determination module 13, and a third determination module 14. Among them,

[0121] The acquisition module 11 is used to acquire an initial sample set, train a target model and a discrimination model according to the labeled samples in the initial sample set; the discrimination model is used to evaluate the target model;

[0122] The first determination module 12 is used to input the unlabeled samples in the initial sample set into the target model to obtain labeled data;

[0123] The second determination module 13 is used to determine the input parameters of the discrimination model according to the labeled data and the unlabeled samples; and input the input parameters into the discrimination model to obtain an evaluation score;

[0124] The third determination module 14 is used to determine whether to update the labeled samples of the initial sample set according to the evaluation score and the labeled data and the unlabeled samples.

[0125] In one embodiment, the third determination module 14 is specifically used to, if the evaluation score is greater than or equal to a preset threshold, add the samples composed of the unlabeled samples and the labeled data as new labeled samples to the initial sample set, and the preset threshold is used to characterize the accuracy of the output result of the target model.

[0126] In one embodiment, the second determination module 13 is specifically used to use the unlabeled samples and the labeled data as input parameters; and / or perform feature extraction processing on the unlabeled samples and the labeled data, and use the extracted features as input parameters.

[0127] In one embodiment, the discrimination model includes a deep learning model and / or a machine learning model.

[0128] In one embodiment, the second determination module 13 is specifically further used to input the input parameters into the deep learning model and the machine learning model respectively; perform an averaging process on the outputs of the deep learning model and the machine learning model to obtain an evaluation score.

[0129] In one embodiment, the training sample acquisition device 10 further includes an evaluation module. The evaluation module is used to evaluate multiple initial labeled samples, and obtain labeled samples from the multiple initial labeled samples based on the evaluation results.

[0130] In one embodiment, the evaluation module is specifically used to perform the following screening process on multiple initial labeled samples:

[0131] Determine L sets of sample sets based on multiple initial labeled samples, determine a training set and a validation set based on the L sets of sample sets, train an initial model corresponding to a target model according to the training set and the validation set to obtain N sets of sample screening models; both L and N are integers greater than zero; input the multiple initial labeled samples into the N sets of sample screening models, determine an evaluation result according to the output of the N sets of sample screening models, and determine abnormal samples among the multiple initial labeled samples according to the evaluation result; update the multiple initial labeled samples according to the abnormal samples, and perform screening processing on the updated multiple initial labeled samples until there are no abnormal samples among the multiple initial labeled samples.

[0132] In one embodiment, the evaluation module is specifically further configured to traverse each set of sample sets in the L sets of sample sets, use one set of sample sets as the validation set, and use the remaining sample sets in the L sets of sample sets as the training set; use the training set to train the initial model to obtain a training result, and use the validation set to validate the trained initial model to obtain a validation result; determine a sample screening model corresponding to one set of sample sets according to the training result, the validation result, and a preset constraint condition; determine N sets of sample screening models according to the sample screening models corresponding to each set of sample sets in the L sets of sample sets.

[0133] In one embodiment, the training sample acquisition device 10 further includes a receiving module and a training module.

[0134] The receiving module is configured to receive a subjective evaluation score for the labeled data and a modification result for the labeled data; the subjective evaluation score is used to represent the subjective evaluation of the labeled data.

[0135] The training module is configured to, if the subjective evaluation score is less than the evaluation score, train the target model and the discriminant model according to the modification result, the unlabeled samples, and the subjective evaluation score.

[0136] For the specific limitations of the above training sample acquisition device 10, reference can be made to the limitations of the training sample acquisition method in the foregoing text, which will not be elaborated here. Each module in the training sample acquisition device 10 can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned devices, modules, or units can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned devices or modules.

[0137] Please refer to Figure 8 , in one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 8As shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store an initial sample set, an initial model, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer device is executed by the processor, it implements a method for obtaining training samples.

[0138] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0139] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0140] Obtain an initial sample set, and train a target model and a discrimination model according to the labeled samples in the initial sample set; the discrimination model is used to evaluate the target model;

[0141] Input the unlabeled samples in the initial sample set into the target model to obtain labeled data;

[0142] Determine the input parameters of the discrimination model according to the labeled data and the unlabeled samples; and input the input parameters into the discrimination model to obtain an evaluation score;

[0143] According to the evaluation score, determine whether to update the labeled samples of the initial sample set according to the labeled data and the unlabeled samples.

[0144] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if the evaluation score is greater than or equal to a preset threshold, then use the samples composed of the unlabeled samples and the labeled data as new labeled samples to be added to the initial sample set, and the preset threshold is used to characterize the accuracy of the output result of the target model.

[0145] In one embodiment, when the processor executes the computer program, the following steps are further implemented: use the unlabeled samples and the labeled data as input parameters; and / or, perform feature extraction processing on the unlabeled samples and the labeled data, and use the extracted features as input parameters.

[0146] In one embodiment, when the processor executes the computer program, the following steps are further implemented: input the input parameters into the deep learning model and the machine learning model respectively; perform a mean value processing on the outputs of the deep learning model and the machine learning model to obtain an evaluation score.

[0147] In one embodiment, when the processor executes the computer program, the following steps are further implemented: perform a sample evaluation on a plurality of initial labeled samples, and obtain labeled samples from the plurality of initial labeled samples based on the evaluation results.

[0148] In one embodiment, when the processor executes the computer program, the following steps are further implemented: perform the following screening process on a plurality of initial labeled samples: determine L sets of sample sets according to the plurality of initial labeled samples, determine a training set and a validation set according to the L sets of sample sets, train the initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models; both L and N are integers greater than zero; input the plurality of initial labeled samples into the N sets of sample screening models, determine the evaluation results according to the outputs of the N sets of sample screening models, and determine the abnormal samples among the plurality of initial labeled samples according to the evaluation results; update the plurality of initial labeled samples according to the abnormal samples, and perform the screening process on the updated plurality of initial labeled samples until the plurality of initial labeled samples do not contain abnormal samples.

[0149] In one embodiment, when the processor executes the computer program, the following steps are further implemented: traverse each set of sample sets in the L sets of sample sets, use one set of sample sets as the validation set, and use the remaining sample sets in the L sets of sample sets as the training set; use the training set to train the initial model to obtain a training result, and use the validation set to validate the trained initial model to obtain a validation result; determine the sample screening model corresponding to one set of sample sets according to the training result, the validation result and the preset constraint conditions; determine N sets of sample screening models according to the sample screening models corresponding to each set of sample sets in the L sets of sample sets.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0151] Obtain an initial sample set, and train a target model and a discrimination model according to the labeled samples in the initial sample set; the discrimination model is used to perform a model evaluation on the target model;

[0152] Input the unlabeled samples in the initial sample set into the target model to obtain labeled data;

[0153] Determine the input parameters of the discrimination model according to the labeled data and the unlabeled samples; and input the input parameters into the discrimination model to obtain an evaluation score;

[0154] Determine whether to update the labeled samples of the initial sample set based on the labeled data and unlabeled samples according to the evaluation score.

[0155] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: If the evaluation score is greater than or equal to a preset threshold, the samples composed of the unlabeled samples and the labeled data are used as new labeled samples and added to the initial sample set, and the preset threshold is used to characterize the accuracy of the output result of the target model.

[0156] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Use the unlabeled samples and the labeled data as input parameters; and / or, perform feature extraction processing on the unlabeled samples and the labeled data, and use the extracted features as input parameters.

[0157] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Input the input parameters into the deep learning model and the machine learning model respectively; perform an averaging process on the outputs of the deep learning model and the machine learning model to obtain an evaluation score.

[0158] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Perform sample evaluation on multiple initial labeled samples, and obtain labeled samples from the multiple initial labeled samples based on the evaluation results.

[0159] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Perform the following screening process on multiple initial labeled samples: Determine L sets of sample sets according to the multiple initial labeled samples, determine a training set and a validation set according to the L sets of sample sets, train the initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models; both L and N are integers greater than zero; Input the multiple initial labeled samples into the N sets of sample screening models, determine the evaluation results according to the outputs of the N sets of sample screening models, and determine the abnormal samples among the multiple initial labeled samples according to the evaluation results; Update the multiple initial labeled samples according to the abnormal samples, and perform the screening process on the updated multiple initial labeled samples until there are no abnormal samples among the multiple initial labeled samples.

[0160] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: Traverse each set of sample sets in the L sets of sample sets, use one set of sample sets as the validation set, and use the remaining sample sets in the L sets of sample sets as the training set; Use the training set to train the initial model to obtain a training result, and use the validation set to validate the trained initial model to obtain a validation result; Determine the sample screening model corresponding to one set of sample sets according to the training result, the validation result, and the preset constraint conditions; Determine N sets of sample screening models according to the sample screening models corresponding to each set of sample sets in the L sets of sample sets.

[0161] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0162] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0163] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for obtaining training samples, characterized in that, including obtaining an initial sample set, training a target model and a discrimination model according to labeled samples in the initial sample set; the discrimination model is used to evaluate the target model; inputting unlabeled samples in the initial sample set into the target model to obtain labeled data; determining input parameters of the discrimination model according to the labeled data and the unlabeled samples; inputting the input parameters into the discrimination model to obtain an evaluation score; determining whether to update the labeled samples in the initial sample set according to the labeled data and the unlabeled samples according to the evaluation score; performing sample evaluation on a plurality of initial labeled samples, and obtaining the labeled samples from the plurality of initial labeled samples based on the evaluation result; wherein, the performing sample evaluation on a plurality of initial labeled samples and obtaining the labeled samples from the plurality of initial labeled samples based on the evaluation result includes: performing the following screening process on the plurality of initial labeled samples: determining L sets of sample sets according to the plurality of initial labeled samples, determining a training set and a validation set according to the L sets of sample sets, and training an initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models; both L and N are integers greater than zero; inputting the plurality of initial labeled samples into the N sets of sample screening models, determining the evaluation result according to the outputs of the N sets of sample screening models, and determining abnormal samples among the plurality of initial labeled samples according to the evaluation result; updating the plurality of initial labeled samples according to the abnormal samples, and performing the screening process on the updated plurality of initial labeled samples until the plurality of initial labeled samples do not include abnormal samples.

2. The training sample acquisition method according to claim 1, wherein The determining whether to update the labeled samples in the initial sample set according to the labeled data and the unlabeled samples according to the evaluation score includes: if the evaluation score is greater than or equal to a preset threshold, taking the samples composed of the unlabeled samples and the labeled data as new labeled samples and adding them to the initial sample set, and the preset threshold is used to characterize the accuracy of the output result of the target model.

3. The training sample acquisition method according to claim 1, wherein, The determining the input parameters of the discrimination model according to the labeled data and the unlabeled samples includes: taking the unlabeled samples and the labeled data as the input parameters; and / or, performing feature extraction processing on the unlabeled samples and the labeled data, and taking the extracted features as the input parameters.

4. The training sample acquisition method according to claim 3, wherein The discrimination model includes a deep learning model and a machine learning model.

5. The training sample acquisition method according to claim 4, wherein When the discrimination model includes a deep learning model and a machine learning model, the inputting the input parameters into the discrimination model to obtain an evaluation score includes: inputting the input parameters into the deep learning model and the machine learning model respectively; performing a mean value processing on the outputs of the deep learning model and the machine learning model to obtain the evaluation score.

6. The training sample acquisition method according to claim 1, characterized in that The determining the training set and the validation set according to the L sets of sample sets, and training the initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models includes: Traverse each set of samples in the L sets of sample sets, use a set of sample sets as the validation set, and use the remaining sample sets in the L sets of sample sets as the training set; Use the training set to train the initial model to obtain a training result, and use the validation set to validate the trained initial model to obtain a validation result; According to the training result, the validation result, and a preset constraint condition, determine the sample screening model corresponding to the set of sample sets; Determine the N sets of sample screening models according to the sample screening models corresponding to each set of sample sets in the L sets of sample sets.

7. The training sample acquisition method according to claim 1, wherein The method further includes: Receiving a subjective evaluation score for the labeled data and a modification result for the labeled data; the subjective evaluation score is used to represent the subjective evaluation of the labeled data; If the subjective evaluation score is less than the evaluation score, train the target model and the discriminative model according to the modification result, the unlabeled samples, and the subjective evaluation score.

8. A training sample acquisition device, characterized in that Including: An acquisition module, configured to acquire an initial sample set, and train a target model and a discriminative model according to the labeled samples in the initial sample set; the discriminative model is used to evaluate the target model; A first determination module, configured to input the unlabeled samples in the initial sample set into the target model to obtain labeled data; A second determination module, configured to determine the input parameters of the discriminative model according to the labeled data and the unlabeled samples; And input the input parameters into the discriminative model to obtain an evaluation score; A third determination module, configured to determine whether to update the labeled samples of the initial sample set according to the labeled data and the unlabeled samples according to the evaluation score; An evaluation module, configured to perform sample evaluation on a plurality of initial labeled samples, and obtain the labeled samples from the plurality of initial labeled samples based on the evaluation result; The evaluation module is specifically configured to perform the following screening process on the plurality of initial labeled samples: Determine L sets of sample sets according to the plurality of initial labeled samples, determine a training set and a validation set according to the L sets of sample sets, train the initial model corresponding to the target model according to the training set and the validation set to obtain N sets of sample screening models; both L and N are integers greater than zero; input the plurality of initial labeled samples into the N sets of sample screening models, determine the evaluation result according to the output of the N sets of sample screening models, and determine the abnormal samples among the plurality of initial labeled samples according to the evaluation result; Update the plurality of initial labeled samples according to the abnormal samples, and perform the screening process on the updated plurality of initial labeled samples until there are no abnormal samples in the plurality of initial labeled samples.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images