Sample data determination method, data processing method, apparatus, device, and medium

Abstract

The application discloses a sample data determination method, a data processing method, a device, equipment and a medium, wherein the method comprises: acquiring an unlabeled data set; taking part of the unlabeled data from the unlabeled data set and labeling the part of the unlabeled data to obtain a first data set with labels corresponding to the part of the unlabeled data; training a type recognition model by using the first data set, the type recognition model being used for recognizing the unlabeled data to obtain a recognition result; acquiring corresponding remaining unlabeled data from the unlabeled data set; and determining a labeled sample data set based on the first data set, the type recognition model and the remaining unlabeled data. The application can save labor cost and time cost in the data labeling process, thereby improving the efficiency of data labeling.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to a method for determining sample data, a data processing method, an apparatus, equipment, and a medium. Background Technology

[0002] In the era of artificial intelligence, deep learning has achieved remarkable results in computer vision fields such as image classification and object detection. The foundation for building high-quality deep learning models lies in training them using large-scale datasets with precise annotations. Therefore, the training process requires significant manual input for data annotation. This manual data annotation method is time-consuming, inefficient, and requires substantial manpower. Summary of the Invention

[0003] This application provides an implementation scheme that differs from related technologies, in order to solve the technical problem that in the related technologies, a large amount of manual labor is required to annotate the data during the model training process. This method of relying on manual data annotation requires a long data annotation cycle and a large amount of human resources, resulting in low efficiency in data annotation.

[0004] Firstly, this application provides a method for determining sample data, including:

[0005] Obtain the unlabeled dataset;

[0006] Take a portion of the unlabeled data from the unlabeled dataset and label it to obtain a first labeled dataset corresponding to the portion of unlabeled data;

[0007] A type recognition model is trained using the first dataset. The type recognition model is used to identify unlabeled data and obtain recognition results. The recognition results include the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data.

[0008] Obtain the corresponding remaining unlabeled data from the unlabeled dataset;

[0009] A labeled sample dataset is determined based on the first data set, the type recognition model, and the remaining unlabeled data.

[0010] Secondly, this application provides a data processing method, including:

[0011] Obtain a labeled sample dataset;

[0012] The second sample data to be cleaned in the labeled sample dataset is traversed, and the second sample data is input into the type recognition model to obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data.

[0013] Obtain the second target threshold of the type recognition model for the second type;

[0014] When the second confidence level is less than the second target threshold, the second sample data is removed from the labeled sample dataset; when the second confidence level is not less than the second target threshold, the second sample data is retained, and the cleaned sample dataset is obtained.

[0015] The labeled sample dataset is obtained by removing a portion of the unlabeled data from the unlabeled dataset for labeling by relevant personnel, resulting in a first set of labeled data corresponding to this portion of unlabeled data; the remaining unlabeled data in the unlabeled dataset is then processed based on the first set of labeled data.

[0016] Thirdly, this application provides a sample data determination apparatus, comprising:

[0017] The first acquisition unit is used to acquire the unlabeled dataset;

[0018] The annotation unit is used to extract a portion of the unlabeled data from the unlabeled dataset and annotate it to obtain a first set of labeled data corresponding to the portion of unlabeled data;

[0019] The training unit is used to train a type recognition model using the first dataset. The type recognition model is used to identify unlabeled data and obtain a recognition result. The recognition result includes the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data.

[0020] The second acquisition unit is used to acquire the corresponding remaining unlabeled data from the unlabeled dataset;

[0021] The determining unit is used to determine a labeled sample dataset based on the first data set, the type recognition model, and the remaining unlabeled data.

[0022] Fourthly, this application provides an electronic device, comprising:

[0023] Processor; and

[0024] Memory for storing the executable instructions of the processor;

[0025] The processor is configured to execute the first aspect, the second aspect, or any method in the first aspect, the second aspect, by executing the executable instructions.

[0026] Fifthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the possible implementations of the first aspect and the second aspect, or any method in the possible implementations of the first aspect and the second aspect.

[0027] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the possible implementations of the first aspect and the second aspect, or any method in the possible implementations of the first aspect and the second aspect.

[0028] This application provides a scheme for obtaining an unlabeled dataset; extracting a portion of unlabeled data from the unlabeled dataset and labeling it to obtain a first labeled dataset corresponding to the portion of unlabeled data; training a type recognition model using the first dataset, the type recognition model being used to identify the unlabeled data and obtaining a recognition result, the recognition result including the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data; obtaining the corresponding remaining unlabeled data from the unlabeled dataset; and determining a labeled sample dataset based on the first dataset, the type recognition model, and the remaining unlabeled data. This scheme can save manpower and time costs in the data labeling process, thereby improving the efficiency of data labeling. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0030] Figure 1 A schematic flowchart illustrating a sample data determination method provided in an embodiment of this application;

[0031] Figure 2 A schematic diagram illustrating a scenario of determining data in a labeled sample pool according to an embodiment of this application;

[0032] Figure 3This is a schematic diagram showing the relationship between the first data set, the third data set, the fourth data set, and the second data set when the unlabeled dataset meets the preset conditions after one loop.

[0033] Figure 4 A schematic diagram of a scenario in which three prediction accuracies are determined when the preset number of times is three, according to an embodiment of this application;

[0034] Figure 5 A schematic diagram illustrating the entire process of data annotation and data cleaning provided in an embodiment of this application;

[0035] Figure 6 A schematic flowchart illustrating a data processing method provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of the structure of a sample data determination device provided in an embodiment of this application;

[0037] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of this application;

[0038] Figure 9 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. 。 Detailed Implementation

[0039] The embodiments of this application are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0040] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the solution can be implemented in a different order than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] First, some terms used in the embodiments of this application will be explained below to facilitate understanding by those skilled in the art.

[0042] Noisy dataset: A dataset containing samples with incorrect labels.

[0043] Supervised learning is a machine learning task that infers a function from a labeled training dataset. The training data consists of a set of training instances. In supervised learning, each instance is a pair consisting of an input object (usually a vector) and a desired output value (also known as a supervision signal).

[0044] In the era of artificial intelligence, deep learning has achieved remarkable results in computer vision fields, including image classification and object detection. Currently, deep learning-based models are primarily trained using supervised learning, which means using precisely labeled datasets.

[0045] A common approach to deep learning model development involves labeling specific data from massive datasets within a particular business scenario. Algorithm developers then use this dataset for algorithm development and model training. Supervised model training requires large-scale, accurately labeled datasets. Furthermore, to obtain higher-quality models, the model size often needs to be increased, which in turn requires a larger training dataset. This necessitates significant investment in manual labeling, resulting in long labeling cycles, high manpower costs, and low efficiency.

[0046] In addition, different annotators have different preferences and levels of expertise, so it is inevitable that there will be some mislabeled data in manually annotated datasets. Directly using noisy data with a large number of mislabeled data for model training will greatly affect the performance of the final model. In order to quickly iterate the model, the simplest and most direct solution is to manually clean these "dirty data". However, manually cleaning data from massive amounts of data still has the problems of high annotation costs and long annotation cycles.

[0047] Therefore, this application provides a sample data determination method, data processing method, apparatus, device, and medium to solve at least one of the aforementioned technical problems.

[0048] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0049] Figure 1 A flowchart illustrating a sample data determination method provided for an exemplary embodiment of this application, the method comprising at least S11-S15:

[0050] S11. Obtain the unlabeled dataset;

[0051] In some embodiments, the unlabeled dataset is a collection of multiple unlabeled datasets.

[0052] In some embodiments, the unlabeled data in the unlabeled dataset can be data of any of the following data types: image data, video data, text data, and audio data.

[0053] S12. Take out a portion of the unlabeled data from the unlabeled dataset and label it to obtain a first labeled dataset corresponding to the portion of unlabeled data;

[0054] In some embodiments, the ratio of the number of unlabeled data to the total number of unlabeled data in the aforementioned unlabeled dataset can be a preset ratio, which can be flexibly set by relevant personnel, for example, the preset ratio can be 10%. The aforementioned unlabeled data can be labeled by relevant personnel.

[0055] S13. A type recognition model is trained using the first data set. The type recognition model is used to identify unlabeled data and obtain a recognition result. The recognition result includes the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data.

[0056] For example, unlabeled data contains images of cats, and the type is cat.

[0057] In some embodiments, the type recognition model trained using the first dataset can be represented as M0.

[0058] Specifically, the type recognition model can recognize data types that are consistent with the data types in the first data set.

[0059] In some embodiments, the unlabeled dataset is stored in an unlabeled sample pool, which can be represented as P0, the unlabeled data in the unlabeled sample pool can be represented as ui, and the unlabeled dataset can be represented as {u1,u2,…,ui,…,un}, where n represents the number of unlabeled data in the unlabeled sample pool P0.

[0060] The first data set can be added to the labeled sample pool. The first data set can be represented as: {(x1,y1),(x2,y2),…,(xi,yi)…,(xm,ym)}, where xi represents the i-th labeled data, yi is the label of xi, and m represents the number of labeled data in the labeled sample pool P1.

[0061] S14. Obtain the corresponding remaining unlabeled data from the unlabeled dataset;

[0062] S15. Determine a labeled sample dataset based on the first data set, the type recognition model, and the remaining unlabeled data;

[0063] In some embodiments provided in this application, S15, determining the labeled sample dataset based on the first data set, the type recognition model, and the remaining unlabeled data, further includes at least S151-S152:

[0064] S151. Based on the type recognition model and the remaining unlabeled data, determine the second set of labeled data corresponding to the remaining unlabeled data;

[0065] In some embodiments, the second set of labeled data corresponding to the remaining unlabeled data can be represented as: (u1,p1),(u2,p2),…,(ui,pi,…,(uk,pk), where pi is the label of the unlabeled data ui in the determined unlabeled sample pool P0.

[0066] In some embodiments, the method further includes adding the first dataset to a labeled sample pool. The labeled sample pool stores labeled data.

[0067] In some embodiments, S151, determining the second set of labeled data corresponding to the remaining unlabeled data based on the type recognition model and the remaining unlabeled data includes the following S01-S08:

[0068] S01, Data Determination Step: The remaining unlabeled data is taken as the unlabeled data to be analyzed;

[0069] S02. Input step: Input the unlabeled data to be analyzed into the type recognition model to obtain the first recognition result corresponding to the unlabeled data to be analyzed. The first recognition result includes the first type to which the unlabeled data to be analyzed belongs, and the first confidence level for the first type. The first type is related to the content of the unlabeled data to be analyzed.

[0070] Among them, the unlabeled data to be analyzed corresponds one-to-one with the first identification result.

[0071] In some embodiments, a first identification result corresponding to an unlabeled data ai to be analyzed can be represented as fi, and fi can be further represented as (fi1, fi2), where fi1 represents the first type to which the unlabeled data to be analyzed belongs, and fi2 represents the first confidence level for fi1.

[0072] S03, Analysis steps: Take at least a portion of the first unlabeled data with a first confidence level greater than a first preset threshold from the unlabeled data to be analyzed, and take the first identification result corresponding to the first unlabeled data as the label of the first unlabeled data to obtain a labeled third data set.

[0073] In some embodiments, the first preset threshold can be set flexibly, and this application does not impose specific limitations on it.

[0074] S04, Manual annotation step: Take out the second unlabeled data in the unlabeled data to be analyzed whose first confidence level is less than the second preset threshold from the unlabeled dataset, and have it annotated by the labelers to obtain a fourth labeled dataset;

[0075] In some embodiments, the second preset threshold can be set flexibly, and this application does not impose specific limitations on it.

[0076] S05, Adding step: Add the third data set and the fourth data set to the labeled sample pool;

[0077] Specifically, the data in the third dataset are relatively simple samples for the type recognition model because the confidence level of their corresponding recognition results is greater than the first preset threshold, i.e., the confidence level is high. These samples can also be called low-information samples. Such samples are very important in the early stage of model training. They can quickly expand the training set, and as the model iterates, they can enable the model to learn more similar data in different forms, thereby increasing the model's generalization ability.

[0078] The confidence level of the recognition results corresponding to the data in the fourth data set is less than the second preset threshold, that is, the confidence level is low. These are relatively difficult samples to identify for the type recognition model, and can also be called high information samples. Such samples are very important for improving the performance of the model in the later stage of model training. They can make the model pay more attention to difficult data and improve the performance of the model.

[0079] The second preset threshold is less than the first preset threshold.

[0080] Specifically, the aforementioned process can also be found in Figure 2 As shown, Figure 2 The following is a scenario-based schematic diagram of the process of determining data in a labeled sample pool provided by an exemplary embodiment of this application corresponding to the above steps, with a more intuitive description of S11-S14 and S01-S05.

[0081] In some embodiments, the first preset threshold can be 0.85, and the second preset threshold can be 0.53. In this case, low-information samples with a first confidence level greater than 0.85 in the first recognition result and their labels are added to the labeled sample pool. Figure 2 The unlabeled data to be analyzed with a first confidence level of 0.99 and 0.9 are considered low-information samples. High-information samples with a first confidence level less than 0.53 in the first identification results are manually labeled to obtain labels. These high-information samples and their labels are then added to the labeled sample pool. Figure 2 The unlabeled data to be analyzed with a first confidence level of 0.52 and 0.48 are considered high-information samples.

[0082] In some embodiments, the first confidence level can be any of the following: 0.82, 0.89, 0.76, or 0.90.

[0083] S06. Training steps: Train the type recognition model based on the data in the labeled sample pool to obtain a new type recognition model;

[0084] Specifically, the new type recognition model will achieve better type recognition results, that is, it can more accurately identify low-information samples and high-information samples in the remaining unlabeled data in the unlabeled sample pool, thereby improving the efficiency of data labeling.

[0085] S07. Loop Determination Step: Determine whether the unlabeled dataset meets the preset conditions. If yes, then take the set of data in the labeled sample pool excluding the data included in the first data set as the second labeled data set corresponding to the remaining unlabeled data. If no, then execute the following loop steps.

[0086] In some embodiments, Figure 3 This diagram illustrates the relationship between the first, third, and fourth data sets and the second data set in the labeled sample pool, assuming that the unlabeled dataset meets the preset conditions after one iteration. Figure 3 It can more intuitively describe the relationships between data sets during data processing.

[0087] In some embodiments, the unlabeled dataset here is the collection of remaining unlabeled data in the unlabeled sample pool. The remaining unlabeled data in the unlabeled sample pool refers to unlabeled data other than that included in the labeled sample pool.

[0088] The preset conditions include: the number of unlabeled data in the unlabeled dataset, excluding the data included in the labeled sample pool, is not greater than a preset value.

[0089] In some embodiments, the preset value can be 0.

[0090] In some embodiments, the preset value may also be other values.

[0091] Specifically, if the unlabeled dataset does not meet the preset condition, that is, the number of data in the unlabeled dataset is greater than the preset value, it means that there is too much unlabeled data remaining in the unlabeled dataset. In other words, the labeling of the unlabeled data in the unlabeled dataset has not been completed, and the remaining unlabeled data in the unlabeled dataset still needs to be labeled until the unlabeled dataset meets the preset condition.

[0092] S08. Looping step: Take the unlabeled data in the unlabeled dataset other than the data included in the labeled sample pool as new unlabeled data to be analyzed; return to execute the aforementioned input step, analysis step, manual annotation step, addition step, training step, and loop judgment step until the second set of labeled data corresponding to the remaining unlabeled data is determined.

[0093] Specifically, after the aforementioned cyclic steps, if the number of unlabeled data in the unlabeled dataset, excluding the data included in the labeled sample pool, meets the preset condition in S07 above, then the data in the unlabeled dataset can be considered to have been labeled.

[0094] S152. The set of data in the first data set and the second data set is used as a labeled sample dataset.

[0095] In deep learning model development, the strong dependence of models on precisely labeled datasets leads to high dataset acquisition costs, long development cycles, and the impact of noisy datasets on model performance. The solution proposed in this application has good versatility and can be widely applied in fields such as image classification and object detection. The data self-labeling scheme provided in this application automatically labels data with minimal human assistance, effectively reducing the human cost of dataset labeling and accelerating the model development process.

[0096] Specifically, addressing the high cost of dataset collection and labeling, this paper starts with model structure or training strategy to enable the model to quickly mine data that can help improve algorithm performance from a large number of unlabeled samples, automatically label them, and add them to supervised training to improve model performance.

[0097] In some embodiments provided in this application, cross-validation technology is also introduced. Based on this, the above-mentioned sample data determination method further includes the following S31-S38:

[0098] S31. According to the preset grouping rules, the labeled sample dataset is grouped multiple times to obtain multiple training sample datasets and multiple verification sample datasets corresponding to the multiple training sample datasets; wherein, each group of training sample datasets corresponds to one group of verification sample datasets.

[0099] In some embodiments, each pair of labeled sample datasets is grouped once to obtain a training sample dataset and a corresponding validation sample dataset. The union of the training sample dataset and the validation sample dataset is the labeled sample dataset.

[0100] In some embodiments, the preset grouping rule can be to use four-fifths of the labeled sample dataset as the training sample dataset and the remaining one-fifth as the validation sample dataset corresponding to the training sample dataset. The number of groupings can also be a preset number, for example, three times.

[0101] S32. Based on the multiple training sample datasets and the type recognition model, multiple candidate recognition models are determined;

[0102] The type recognition model is the most recently determined type recognition model M1 in step S06.

[0103] Specifically, the multiple training sample datasets are respectively input into the type recognition model for model training to obtain multiple new type recognition models corresponding to the multiple training sample datasets, and the multiple new type recognition models corresponding to the multiple training sample datasets are determined as the multiple candidate recognition models.

[0104] S33. Based on the multiple verification sample datasets, determine a new type recognition model from the multiple candidate recognition models;

[0105] In some embodiments, S33, determining a new type recognition model from the plurality of candidate recognition models based on the plurality of verification sample datasets includes:

[0106] The multiple verification sample datasets are input into the multiple candidate recognition models to obtain multiple output results corresponding to the multiple verification sample datasets;

[0107] Based on the multiple output results, determine the multiple prediction accuracies of the multiple candidate recognition models for the types of validation sample data in the multiple validation sample datasets;

[0108] The candidate identification model corresponding to the highest prediction accuracy among the multiple prediction accuracies is determined as the new type identification model.

[0109] In this dataset, a set of validation samples corresponds to one candidate recognition model, one set of output results, and one prediction accuracy. A set of training samples corresponds to one set of validation samples.

[0110] Furthermore, methods for determining multiple prediction accuracies can be found in [link to relevant documentation]. Figure 4 As shown, Figure 4 This is a schematic diagram illustrating how to determine the accuracy of three predictions when the preset number of attempts is 3.

[0111] S34. Obtain multiple preset types that the type recognition model can recognize;

[0112] For example, each labeled sample data in the labeled sample dataset is image data, and the preset type is related to the content of the image data.

[0113] For example, the preset type can be the category of objects in the image data content, such as: cat, dog, eagle, elephant, airplane, plant, vehicle, football, etc.

[0114] S35. Traverse each preset type among the multiple preset types, and obtain multiple first sample data from the labeled sample dataset that belong to the preset type and have a confidence level greater than a third preset threshold for the preset type.

[0115] As mentioned above, the labeled sample dataset also includes the labels for each labeled sample data. The labels include the type to which the labeled sample data belongs, as well as the confidence level for that type.

[0116] The labels for some of the labeled sample data were manually labeled, while the labels for other labeled sample data were determined based on the output of the corresponding type recognition model.

[0117] In some embodiments, the third preset threshold can be flexibly set, and this application does not impose specific limitations on it.

[0118] For example, the third preset threshold can be 0.9.

[0119] S36. Input the plurality of first sample data into the type recognition model to obtain the plurality of prediction confidence levels corresponding to the plurality of first sample data;

[0120] S37. Determine the first target threshold of the type identification model for the preset type based on the multiple prediction confidence scores;

[0121] In one feasible embodiment provided in this application, the specific implementation of determining the first target threshold of the type recognition model for the preset type based on the plurality of prediction confidences, i.e., the process of determining the first target threshold of the preset type, can be as follows:

[0122] The average of the multiple prediction confidence scores is used to obtain the first target threshold of the type recognition model for the preset type.

[0123] The average value obtained by averaging the multiple prediction confidences is determined as the first target threshold.

[0124] In another feasible embodiment provided in this application, the specific implementation of determining the first target threshold of the type recognition model for the preset type based on the plurality of prediction confidences, i.e., the process of determining the first target threshold of the preset type, can also be S371-S373:

[0125] S371. Average the multiple prediction confidence scores to obtain the unprocessed threshold of the type recognition model for the preset type;

[0126] S372. Determine the threshold range of the type recognition model for the preset type based on the threshold to be processed and the preset step size;

[0127] The preset step size can be flexibly set, and this application does not impose a specific limitation on it. For example, if the preset step size is set to 0.05, then the range between the value of the threshold to be processed plus 0.05 and the value of the threshold to be processed minus 0.05 is taken as the threshold range.

[0128] In some embodiments, it may be as shown in Table 1:

[0129] Table 1. Correspondence between multiple preset types and multiple first target thresholds

[0130] Preset type First target threshold Preset type 0 0.48 Preset type 1 0.56 Preset type 2 0.76 Preset type 3 0.42 Preset type 4 0.81 ... ...

[0131] S373. Take any value within the threshold range as the first target threshold of the type recognition model for the preset type.

[0132] S38. Based on the type recognition model and multiple first target thresholds for the multiple preset types, perform data cleaning on the data in the labeled sample dataset to obtain the cleaned sample dataset.

[0133] In some embodiments provided in this application, the step of cleaning the data in the labeled sample dataset according to the type recognition model for multiple first target thresholds of multiple preset types to obtain a cleaned sample dataset includes S381-S383:

[0134] S381. Traverse the second sample data to be cleaned in the labeled sample dataset, input the second sample data into the type recognition model, and obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data.

[0135] For example, the content of the second sample data is an image of a cat, and the second type is a cat.

[0136] S382. Obtain the second target threshold of the type recognition model for the second type;

[0137] Specifically, obtaining the second target threshold of the type recognition model for the second type includes:

[0138] Find a target preset type that is the same as the second type from the plurality of preset types; use the first target threshold of the target preset type as the second target threshold of the second type.

[0139] S383. When the second confidence level is less than the second target threshold, the second sample data is removed from the labeled sample dataset. When the second confidence level is not less than the second target threshold, the second sample data is retained to obtain the cleaned sample dataset.

[0140] Specifically, when the second confidence level is less than the second target threshold, it indicates that the label of the second sample data is unreasonable and can be considered as a noisy data sample, which should be cleaned up.

[0141] In some embodiments, the aforementioned labeled sample dataset can be represented as D0, and the cleaned dataset can be represented as D1.

[0142] This solution can address the technical problem of noisy data in the dataset affecting model performance, and achieve automatic cleaning of noisy data, thus improving the efficiency of data cleaning.

[0143] Furthermore, the aforementioned scheme can also be found in [reference needed]. Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the entire process of data annotation and data cleaning provided for an exemplary embodiment of this application.

[0144] Figure 6 A flowchart illustrating a data processing method provided for an exemplary embodiment of this application is shown, the method comprising at least the following steps S51-S55:

[0145] S51. Obtain a labeled sample dataset;

[0146] The labeled sample dataset is obtained by removing a portion of the unlabeled data from the unlabeled dataset for labeling by relevant personnel, resulting in a first set of labeled data corresponding to this portion of unlabeled data; the remaining unlabeled data in the unlabeled dataset is then processed based on the first set of labeled data.

[0147] S52. Traverse the second sample data to be cleaned in the labeled sample dataset, input the second sample data into the type recognition model, and obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data.

[0148] S53. Obtain the second target threshold of the type recognition model for the second type;

[0149] S54. When the second confidence level is less than the second target threshold, the second sample data is removed from the labeled sample dataset; when the second confidence level is not less than the second target threshold, the second sample data is retained, and the cleaned sample dataset is obtained.

[0150] The labeled sample dataset is obtained by removing a portion of the unlabeled data from the unlabeled dataset for labeling by relevant personnel, resulting in a first set of labeled data corresponding to this portion of unlabeled data; the remaining unlabeled data in the unlabeled dataset is then processed based on the first set of labeled data.

[0151] The specific implementation method in this embodiment can be found in the foregoing content, and will not be repeated here.

[0152] In summary, the data self-labeling technology proposed in this paper can effectively solve the technical problems of high labeling costs and long cycles in the data collection process, thereby saving labeling manpower costs and rapidly developing algorithm models.

[0153] Furthermore, the data self-cleaning proposed in this paper can effectively solve the technical problem of noisy data affecting model entries, and can effectively improve the performance of the model.

[0154] Figure 7 A schematic diagram of a sample data determination device provided for an exemplary embodiment of this application is shown. The sample data determination device includes:

[0155] First Acquisition Unit 61: Acquire the unlabeled dataset;

[0156] Labeling unit 62: Extracts a portion of unlabeled data from the unlabeled dataset and labels it to obtain a first set of labeled data corresponding to the portion of unlabeled data;

[0157] Training Unit 63: Trains a type recognition model using the first dataset. The type recognition model is used to identify unlabeled data and obtain a recognition result. The recognition result includes the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data.

[0158] Second acquisition unit 64: Acquires the corresponding remaining unlabeled data from the unlabeled dataset;

[0159] Determining unit 65: Determines a labeled sample dataset based on the first data set, the type recognition model, and the remaining unlabeled data.

[0160] In some embodiments, when the sample data determining device is used to determine a labeled sample dataset based on the first dataset, the type recognition model, and the remaining unlabeled data, it is specifically used for:

[0161] Based on the type recognition model and the remaining unlabeled data, determine the second set of labeled data corresponding to the remaining unlabeled data;

[0162] The set of data from the first data set and the second data set is used as the labeled sample dataset.

[0163] In some embodiments, the sample data determining device is further configured to: add the first data set to the labeled sample pool.

[0164] In some embodiments, when the sample data determining device is used to determine the second labeled data set corresponding to the remaining unlabeled data based on the type recognition model and the remaining unlabeled data, it is specifically used for:

[0165] Data determination step: The remaining unlabeled data is taken as the unlabeled data to be analyzed;

[0166] Input steps: Input the unlabeled data to be analyzed into the type recognition model to obtain the first recognition result corresponding to the unlabeled data to be analyzed. The first recognition result includes the first type to which the unlabeled data to be analyzed belongs, and the first confidence level for the first type, wherein the first type is related to the content of the unlabeled data to be analyzed.

[0167] Analysis steps: Extract at least a portion of the first unlabeled data whose first confidence level is greater than a first preset threshold from the unlabeled dataset, and use the first identification result corresponding to the first unlabeled data as the label of the first unlabeled data to obtain a labeled third dataset;

[0168] Manual annotation steps: Take out the second unlabeled data in the unlabeled data to be analyzed whose first confidence level is less than the second preset threshold from the unlabeled dataset, and have it annotated by the labelers to obtain a fourth labeled dataset;

[0169] Addition step: Add the third data set and the fourth data set to the labeled sample pool;

[0170] Training steps: Train the type recognition model based on the data in the labeled sample pool to obtain a new type recognition model;

[0171] Loop determination step: Determine whether the unlabeled dataset meets the preset conditions. If yes, then take the set of data in the labeled sample pool other than the data included in the first data set as the second labeled data set corresponding to the remaining unlabeled data. If no, then execute the following loop step.

[0172] Looping step: Take the unlabeled data in the unlabeled dataset other than the data included in the labeled sample pool as new unlabeled data to be analyzed; return to execute the aforementioned input step, analysis step, manual annotation step, addition step, training step, and loop judgment step until the second set of labeled data corresponding to the remaining unlabeled data is determined.

[0173] In some embodiments, the sample data determining device is further configured to:

[0174] According to the preset grouping rules, the labeled sample dataset is grouped multiple times to obtain multiple training sample datasets and multiple validation sample datasets corresponding to the multiple training sample datasets.

[0175] Based on the multiple training sample datasets and the type recognition model, multiple candidate recognition models are determined;

[0176] Based on the multiple verification sample datasets, a new type recognition model is determined from the multiple candidate recognition models.

[0177] In some embodiments, the sample data determining device is further configured to:

[0178] Obtain multiple preset types that the type recognition model can recognize;

[0179] Iterate through each of the multiple preset types, and obtain multiple first sample data from the labeled sample dataset that belong to the preset type and have a confidence level greater than a third preset threshold for the preset type;

[0180] The multiple first sample data are input into the type recognition model to obtain multiple prediction confidence levels corresponding to the multiple first sample data;

[0181] Based on the multiple prediction confidence scores, a first target threshold for the type identification model for the preset type is determined;

[0182] Based on the type recognition model and multiple first target thresholds for the multiple preset types, the data in the labeled sample dataset is cleaned to obtain the cleaned sample dataset.

[0183] In some embodiments, when the sample data determining device is used to determine the first target threshold of the type recognition model for the preset type based on the plurality of prediction confidences, it is specifically used for:

[0184] The average of the multiple prediction confidence scores is used to obtain the first target threshold of the type recognition model for the preset type.

[0185] In some embodiments, when the sample data determining device is used to determine the first target threshold of the type recognition model for the preset type based on the plurality of prediction confidences, it is specifically used for:

[0186] The average of the multiple prediction confidence scores is used to obtain the threshold to be processed by the type recognition model for the preset type;

[0187] The threshold range of the type recognition model for the preset type is determined based on the threshold to be processed and the preset step size;

[0188] Any value within the threshold range is used as the first target threshold of the type recognition model for the preset type.

[0189] In some embodiments, when the sample data determining device is used to perform data cleaning on the data in the labeled sample dataset according to the type recognition model for multiple first target thresholds of the multiple preset types, and to obtain the cleaned sample dataset, it is specifically used for:

[0190] The second sample data to be cleaned in the labeled sample dataset is traversed, and the second sample data is input into the type recognition model to obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data.

[0191] Obtain the second target threshold of the type recognition model for the second type;

[0192] When the second confidence level is less than the second target threshold, the second sample data is removed from the labeled sample dataset; when the second confidence level is not less than the second target threshold, the second sample data is retained, thus obtaining the cleaned sample dataset.

[0193] Accordingly, the specific implementation of this embodiment can be found in the foregoing content, and will not be repeated here.

[0194] An exemplary embodiment of this application also provides a data processing apparatus, the data processing apparatus comprising:

[0195] The acquisition unit is used to acquire labeled sample datasets.

[0196] The traversal unit is used to traverse the second sample data to be cleaned in the labeled sample dataset, input the second sample data into the type recognition model, and obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data.

[0197] The acquisition unit is further configured to acquire the second target threshold of the type recognition model for the second type;

[0198] A cleaning unit is used to remove the second sample data from the labeled sample dataset when the second confidence level is less than the second target threshold, and to retain the second sample data when the second confidence level is not less than the second target threshold, thereby obtaining the cleaned sample dataset.

[0199] The labeled sample dataset is obtained by removing a portion of the unlabeled data from the unlabeled dataset for labeling by relevant personnel, resulting in a first set of labeled data corresponding to this portion of unlabeled data; the remaining unlabeled data in the unlabeled dataset is then processed based on the first set of labeled data.

[0200] Accordingly, the specific implementation of this embodiment can be found in the foregoing content, and will not be repeated here.

[0201] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device correspond to the corresponding processes in the various methods in the above method embodiments, which will not be repeated here for the sake of brevity.

[0202] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. In some embodiments, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0203] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of this application. The electronic device may include:

[0204] The system includes a memory 701 and a processor 702. The memory 701 stores computer programs and transfers the program code to the processor 702. In other words, the processor 702 can retrieve and run the computer programs from the memory 701 to implement the methods described in the embodiments of this application.

[0205] For example, the processor 702 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0206] In some embodiments of this application, the processor 702 may include, but is not limited to:

[0207] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0208] In some embodiments of this application, the memory 701 includes, but is not limited to:

[0209] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0210] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 701 and executed by the processor 702 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0211] like Figure 8 As shown, the electronic device may further include:

[0212] Transceiver 703, which can be connected to processor 702 or memory 701.

[0213] The processor 702 can control the transceiver 703 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 703 may include a transmitter and a receiver. The transceiver 703 may further include antennas, and the number of antennas may be one or more.

[0214] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0215] Figure 9 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 90 stores a computer program, which, when executed by a computer, enables the computer to perform the methods described in the above-described method embodiments. Alternatively, this application also provides a computer program product containing instructions, which, when executed by a computer, cause the computer to perform the methods described in the above-described method embodiments.

[0216] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0217] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0219] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0220] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining sample data, characterized in that, include: Obtain an unlabeled dataset, wherein the unlabeled data in the unlabeled dataset is any of the following data types: image data, video data, text data, and audio data; Take a portion of the unlabeled data from the unlabeled dataset and label it to obtain a first labeled dataset corresponding to the portion of unlabeled data; A type recognition model is trained using the first dataset. The type recognition model is used to identify unlabeled data and obtain recognition results. The recognition results include the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data. Obtain the corresponding remaining unlabeled data from the unlabeled dataset; Based on the first dataset, the type recognition model, and the remaining unlabeled data, a labeled sample dataset is determined. Determining a labeled sample dataset based on the first data set, the type recognition model, and the remaining unlabeled data includes: determining a second labeled data set corresponding to the remaining unlabeled data based on the type recognition model and the remaining unlabeled data; and using the set of data in the first data set and the second data set as the labeled sample dataset. The method further includes: adding the first data set to a labeled sample pool; determining a labeled second data set corresponding to the remaining unlabeled data based on the type recognition model and the remaining unlabeled data, including: a data determination step: using the remaining unlabeled data as unlabeled data to be analyzed; an input step: inputting the unlabeled data to be analyzed into the type recognition model to obtain a first recognition result corresponding to the unlabeled data to be analyzed, the first recognition result including a first type to which the unlabeled data to be analyzed belongs, and a first confidence level for the first type, wherein the first type is related to the content of the unlabeled data to be analyzed; an analysis step: extracting at least a portion of the first unlabeled data with a first confidence level greater than a first preset threshold from the unlabeled dataset, and using the first recognition result corresponding to the first unlabeled data as the label of the first unlabeled data, to obtain a labeled third data set; a manual annotation step: manually annotating the first type of the unlabeled data to be analyzed. The second unlabeled data with a confidence level less than the second preset threshold is taken from the unlabeled dataset and labeled by the labelers to obtain a fourth labeled dataset; Addition step: the third dataset and the fourth dataset are added to the labeled sample pool; Training step: the type recognition model is trained based on the data in the labeled sample pool to obtain a new type recognition model; Loop determination step: it is determined whether the unlabeled dataset meets the preset conditions. If so, the set of data in the labeled sample pool excluding the data included in the first dataset is taken as the second labeled dataset corresponding to the remaining unlabeled data; if not, the following loop step is executed; Loop step: the unlabeled data in the unlabeled dataset excluding the data included in the labeled sample pool is taken as new unlabeled data to be analyzed; return to execute the aforementioned input step, analysis step, manual labeling step, addition step, training step, and loop determination step until the second labeled dataset corresponding to the remaining unlabeled data is determined; The method further includes: obtaining multiple preset types that the type recognition model can recognize; traversing each preset type among the multiple preset types, and obtaining multiple first sample data from the labeled sample dataset that belong to the preset type and have a confidence level greater than a third preset threshold for the preset type; inputting the multiple first sample data into the type recognition model to obtain multiple predicted confidence levels corresponding to the multiple first sample data; determining a first target threshold for the type recognition model for the preset type based on the multiple predicted confidence levels; and cleaning the data in the labeled sample dataset based on the multiple first target thresholds for the multiple preset types to obtain a cleaned sample dataset.

2. The method according to claim 1, characterized in that, The method further includes: According to the preset grouping rules, the labeled sample dataset is grouped multiple times to obtain multiple training sample datasets and multiple verification sample datasets corresponding to the multiple training sample datasets. Based on the multiple training sample datasets and the type recognition model, multiple candidate recognition models are determined; Based on the multiple verification sample datasets, a new type recognition model is determined from the multiple candidate recognition models.

3. The method according to claim 1, characterized in that, Determining a first target threshold for the type recognition model for the preset type based on the multiple prediction confidence scores includes: The average of the multiple prediction confidence scores is used to obtain the first target threshold of the type recognition model for the preset type.

4. The method according to claim 1, characterized in that, Determining a first target threshold for the type recognition model for the preset type based on the multiple prediction confidence scores includes: The average of the multiple prediction confidence scores is used to obtain the threshold to be processed by the type recognition model for the preset type; The threshold range of the type recognition model for the preset type is determined based on the threshold to be processed and the preset step size; Any value within the threshold range is used as the first target threshold of the type recognition model for the preset type.

5. The method according to claim 1, characterized in that, Based on the type recognition model and multiple first target thresholds for the multiple preset types, the data in the labeled sample dataset is cleaned to obtain a cleaned sample dataset, including: The second sample data to be cleaned in the labeled sample dataset is traversed, and the second sample data is input into the type recognition model to obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data. Obtain the second target threshold of the type recognition model for the second type; When the second confidence level is less than the second target threshold, the second sample data is removed from the labeled sample dataset; when the second confidence level is not less than the second target threshold, the second sample data is retained, thus obtaining the cleaned sample dataset.

6. A sample data determination device, characterized in that, include: The first acquisition unit is used to acquire an unlabeled dataset, wherein the unlabeled data in the unlabeled dataset is data of any of the following data types: image data, video data, text data, and audio data; The annotation unit is used to extract a portion of the unlabeled data from the unlabeled dataset and annotate it to obtain a first set of labeled data corresponding to the portion of unlabeled data; The training unit is used to train a type recognition model using the first dataset. The type recognition model is used to identify unlabeled data and obtain a recognition result. The recognition result includes the type to which the unlabeled data belongs and the confidence level for the corresponding type, wherein the type is related to the content of the unlabeled data. The second acquisition unit is used to acquire the corresponding remaining unlabeled data from the unlabeled dataset; The determining unit is configured to determine a labeled sample dataset based on the first data set, the type identification model, and the remaining unlabeled data. When the sample data determining device is used to determine a labeled sample dataset based on the first data set, the type recognition model, and the remaining unlabeled data, it is specifically used to: determine a second labeled data set corresponding to the remaining unlabeled data based on the type recognition model and the remaining unlabeled data; and take the set of data in the first data set and the second data set as the labeled sample dataset; The sample data determination device is further configured to: add the first data set to the labeled sample pool; when the sample data determination device determines the labeled second data set corresponding to the remaining unlabeled data based on the type recognition model and the remaining unlabeled data, it is specifically configured to: a data determination step: using the remaining unlabeled data as unlabeled data to be analyzed; an input step: inputting the unlabeled data to be analyzed into the type recognition model to obtain a first recognition result corresponding to the unlabeled data to be analyzed, the first recognition result including a first type to which the unlabeled data to be analyzed belongs, and a first confidence level for the first type, wherein the first type is related to the content of the unlabeled data to be analyzed; an analysis step: extracting at least a portion of the first unlabeled data with a first confidence level greater than a first preset threshold from the unlabeled dataset, and using the first recognition result corresponding to the first unlabeled data as the label of the first unlabeled data to obtain a labeled third data set; a manual annotation step: manually annotating the first data set corresponding to the unlabeled data to be analyzed. A second set of unlabeled data with a confidence level less than a second preset threshold is taken from the unlabeled dataset and labeled by the labelers to obtain a fourth set of labeled data. The second preset threshold is less than the first preset threshold. Addition step: The third and fourth sets of data are added to the labeled sample pool. Training step: The type recognition model is trained based on the data in the labeled sample pool to obtain a new type recognition model. Loop determination step: It is determined whether the unlabeled dataset meets the preset conditions. If so, the set of data in the labeled sample pool excluding the data included in the first set is taken as the second set of labeled data corresponding to the remaining unlabeled data. If not, the following loop step is executed: Loop step: The unlabeled data in the unlabeled dataset excluding the data included in the labeled sample pool is taken as new unlabeled data to be analyzed. The process of inputting, analyzing, manually labeling, adding, training, and looping is repeated until the second set of labeled data corresponding to the remaining unlabeled data is determined. The sample data determination device is further configured to: acquire multiple preset types that the type recognition model can recognize; traverse each preset type among the multiple preset types, and acquire multiple first sample data from the labeled sample dataset that belong to the preset type and have a confidence level greater than a third preset threshold for the preset type; input the multiple first sample data into the type recognition model to obtain multiple predicted confidence levels corresponding to the multiple first sample data; determine a first target threshold for the type recognition model for the preset type based on the multiple predicted confidence levels; and perform data cleaning on the data in the labeled sample dataset based on the multiple first target thresholds for the multiple preset types to obtain a cleaned sample dataset.

7. The apparatus according to claim 6, characterized in that, The sample data determination device is also used for: According to the preset grouping rules, the labeled sample dataset is grouped multiple times to obtain multiple training sample datasets and multiple verification sample datasets corresponding to the multiple training sample datasets. Based on the multiple training sample datasets and the type recognition model, multiple candidate recognition models are determined; Based on the multiple verification sample datasets, a new type recognition model is determined from the multiple candidate recognition models.

8. The apparatus according to claim 6, characterized in that, When the sample data determining device is used to determine the first target threshold of the type recognition model for the preset type based on the plurality of prediction confidence scores, it is specifically used for: The average of the multiple prediction confidence scores is used to obtain the first target threshold of the type recognition model for the preset type.

9. The apparatus according to claim 6, characterized in that, When the sample data determining device is used to determine the first target threshold of the type recognition model for the preset type based on the multiple prediction confidence scores, it is specifically used for: The average of the multiple prediction confidence scores is used to obtain the threshold to be processed by the type recognition model for the preset type; The threshold range of the type recognition model for the preset type is determined based on the threshold to be processed and the preset step size; Any value within the threshold range is used as the first target threshold of the type recognition model for the preset type.

10. The apparatus according to claim 6, characterized in that, The sample data determination device is used to perform data cleaning on the labeled sample dataset according to the type recognition model for multiple first target thresholds of the multiple preset types, and to obtain the cleaned sample dataset, specifically for: The second sample data to be cleaned in the labeled sample dataset is traversed, and the second sample data is input into the type recognition model to obtain the second recognition result corresponding to the second sample data. The second recognition result includes the second type to which the second sample data belongs, and the second confidence level for the second type, wherein the second type is related to the content of the second sample data. Obtain the second target threshold of the type recognition model for the second type; When the second confidence level is less than the second target threshold, the second sample data is removed from the labeled sample dataset; when the second confidence level is not less than the second target threshold, the second sample data is retained, thus obtaining the cleaned sample dataset.

11. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-5 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method as described in any one of claims 1-5.

Images