Training sample processing method, data classification method, device and electronic device

Abstract

The embodiments of the present application provide a training sample processing method, a data classification method, a device, and an electronic device, which relate to the fields of artificial intelligence, big data, multimedia, and cloud technology. The processing method includes: obtaining an initial neural network model and an initial training data set including n sample subsets; training the initial neural network model based on each sample subset to obtain n first neural network models; for each sample subset, using each first neural network model other than the first neural network model corresponding to the subset to identify each training sample in the subset, to obtain n-1 first prediction labels; determining the first target sample in the initial training data set based on the labeled label and the first prediction label of each training sample; and constructing a target training data set based on the determined first target samples. The method provided by the embodiments of the present application can effectively improve the quality of the training data set.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, cloud technology, and smart transportation technology. Specifically, the present application relates to a training sample processing method, a data classification method, a device, and an electronic device. Background Art

[0002] With the advancement of artificial intelligence (AI) technology, research and application have been extensively applied across a wide range of fields. Machine learning is the core of AI and the fundamental way to make computers intelligent. Through machine learning, neural network models can be trained to meet application requirements. Supervised learning is currently a widely studied machine learning approach for training neural network models.

[0003] Supervised learning, also known as supervised training, uses labeled training samples to learn a neural network model that meets performance requirements. Supervised learning requires a large number of labeled training samples, and these labels are often manually assigned. While current labeling methods can meet application requirements to a certain extent, due to manual labeling, it is difficult to ensure consistent labeling standards. Furthermore, the labeling process is likely to encounter unexpected data samples, making them difficult to distinguish during labeling. Due to the lack of professionalism among labelers, it is difficult to ensure the accuracy of the labels for these samples, which is detrimental to subsequent model training. Therefore, improving the quality of training samples is a key issue that needs to be addressed in machine learning. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a training sample processing method that can effectively improve the quality of training samples, as well as a data classification method, device and electronic device based on the training sample processing method. In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0005] In one aspect, an embodiment of the present application provides a training sample processing method, the method comprising:

[0006] Obtaining an initial neural network model to be trained and an initial training data set corresponding to the initial neural network model, wherein the initial training data set includes n sample subsets, n≥2, and each sample subset includes multiple training samples with annotated labels;

[0007] Training the initial neural network model based on each of the sample subsets to obtain n first neural network models;

[0008] For each of the sample subsets, the n-1 first neural network models corresponding to the sample subset are used to respectively identify each training sample in the sample subset to obtain n-1 first prediction labels for each training sample, wherein the n-1 first neural network models corresponding to each of the sample subsets refer to the n-1 models among the n first neural network models excluding the first neural network model corresponding to the sample subset;

[0009] For each of the training samples, if a first number of labels in the n-1 first predicted labels of the training sample that are identical to the labeled label of the training sample is not less than a first set number, then the training sample is determined as a first target sample;

[0010] Based on the determined first target samples, a target training data set corresponding to the initial neural network model is constructed.

[0011] On the other hand, an embodiment of the present application provides a data classification method, the method comprising:

[0012] Obtaining data to be classified; inputting the data to be classified into a target classification model to obtain a category recognition result of the data to be classified;

[0013] In which, the target classification model is obtained by training the initial classification model using a target training data set with annotated labels, and the target training sample set is obtained by processing the initial training data set of the initial classification model using the training sample processing method provided in an embodiment of the present application, and the annotated label of each training sample in the target training data set represents the true category of the training sample.

[0014] On the other hand, an embodiment of the present application provides a training sample processing device, the device comprising:

[0015] An initial training set acquisition module is used to acquire an initial neural network model to be trained and an initial training data set corresponding to the initial neural network model, wherein the initial training data set includes n sample subsets, n≥2, and each sample subset includes multiple training samples with annotated labels;

[0016] The sample processing module is configured to perform the following processing based on the initial training data set:

[0017] Training the initial neural network model based on each of the sample subsets to obtain n first neural network models;

[0018] For each of the sample subsets, the n-1 first neural network models corresponding to the sample subset are used to identify each training sample in the sample subset to obtain n-1 first prediction labels for each training sample, wherein the n-1 first neural network models corresponding to each of the sample subsets refer to the n-1 models among the n first neural network models excluding the first neural network model corresponding to the sample subset;

[0019] For each of the training samples, if a first number of labels in the n-1 first predicted labels of the training sample that are identical to the labeled label of the training sample is not less than a first set number, then the training sample is determined as a first target sample;

[0020] The target training set acquisition module is used to construct a target training data set corresponding to the initial neural network model based on the determined first target samples.

[0021] Optionally, the sample processing module is further configured to: use all the determined first target samples as a first target sample set, train the initial neural network model based on the first target sample set, and obtain a second neural network model; use the training samples other than the first target samples in the initial training data set as pending samples, identify each of the pending samples based on the second neural network model, and obtain a second predicted label for each of the pending samples; and determine a second target sample in each of the pending samples based on the second predicted label of each of the pending samples;

[0022] Correspondingly, the target training set acquisition module can be used to construct the target training data set based on the determined first target samples and second target samples.

[0023] Optionally, the sample processing module may be configured to: for each of the n sample subsets, take all the to-be-determined samples in the sample subset as a first to-be-determined subset; divide the n first to-be-determined subsets into m second to-be-determined subsets, where n≥m≥2, each second to-be-determined subset includes at least one first to-be-determined subset, and each first to-be-determined subset belongs to only one second to-be-determined subset; merge the first target sample set with the m second to-be-determined subsets respectively to obtain m first mixed sample sets, and train the initial neural network model based on each of the first mixed sample sets to obtain m second neural network models;

[0024] For each of the pending samples, the m-1 second neural network models corresponding to the pending sample are used to respectively identify the pending sample to obtain m-1 second prediction labels for the pending sample, wherein the m-1 second neural network models corresponding to the pending sample refer to the m-1 models of the m second neural network models excluding the first model, and the first model is the second neural network model corresponding to the first mixed sample set to which the pending sample belongs;

[0025] Based on the m-1 second predicted labels of each of the pending samples, a second target sample in each of the pending samples is determined.

[0026] Optionally, when determining the second target sample in each of the pending samples, the sample processing module can be used to: for each of the pending samples, if the second number of labels in the m-1 second predicted labels of the pending sample that are the same as the marked label of the pending sample is not less than a second set number, then the pending sample is determined as the second target sample.

[0027] Optionally, the sample processing module is further configured to: for each of the first undetermined subsets, treat each undetermined sample in the undetermined subset except the second target sample as a rare sample, reacquire a new annotated label for each of the rare samples, and use the rare sample with the new annotated label as a third target sample;

[0028] Correspondingly, the target training set acquisition module can be used to construct the target training data set based on the first target samples, the second target samples and the third target samples.

[0029] Optionally, when reacquiring a new annotation label for each of the rare samples, the sample processing module may be configured to:

[0030] Merge all the determined first target samples and all the second target samples to obtain a second target sample set; for each of the first undetermined subsets, take all the rare samples in the undetermined subset as a first rare sample set; divide the n first rare sample sets into k second rare sample sets, where n≥k≥2, each second rare sample set includes at least one first rare sample set, and each first rare sample set belongs to only one second rare sample set; merge the second target sample set with each of the second rare sample sets respectively to obtain k second mixed sample sets; train the initial neural network model based on each of the second mixed sample sets respectively to obtain k third neural network models; for each of the second rare sample sets, use k-1 third neural network models other than the third neural network model corresponding to the second rare sample set to identify each rare sample in the second rare sample set respectively, and obtain k-1 third predicted labels for each rare sample; for each of the rare samples, determine the predicted label with the highest proportion among the n-1 third predicted labels of the rare sample as the new label label of the rare sample.

[0031] Optionally, when determining the second target sample in each of the pending samples based on the m-1 second prediction labels of each of the pending samples, the sample processing module may be configured to:

[0032] For each of the pending samples, the predicted label with the highest proportion among the m-1 second predicted labels of the pending sample is determined as the new labeled label of the pending sample; and each of the pending samples with the new labeled label is used as the second target sample.

[0033] Optionally, the sample processing module can also be used to:

[0034] At least one sample subset of the n sample subsets is used as a reference dataset, and for each reference dataset, based on the n sample subsets, a first training set, a second training set, a third training set, and a test set corresponding to the reference dataset are constructed, wherein the first training set includes at least part of the training samples in the reference dataset and at least part of the training samples in n-1 sample subsets other than the reference dataset, the second training set is at least part of the training samples in n-1 sample subsets other than the reference dataset, the training samples in the third training set and the test set are at least part of the training samples in the reference dataset, and the training samples in the test set are different from the training samples in the third training set;

[0035] For each of the reference data sets, respectively training the initial neural network model based on the three training sets corresponding to the reference data set to obtain three fourth neural network models, and performing performance evaluation on each of the fourth neural network models based on the test set and the model evaluation index to obtain an index value for each of the fourth neural network models, and determining the data quality of the initial training data set corresponding to the reference data set based on the three index values ​​corresponding to the three fourth neural network models;

[0036] If the data quality corresponding to each of the reference data sets meets the preset data quality condition, the initial training data set is used as the target training data set;

[0037] The sample processing module can be used to: if the data quality corresponding to each reference data set does not meet the preset data quality conditions, then the initial neural network model is trained based on each sample subset.

[0038] Optionally, the initial neural network model is an initial classification model, and the annotated label of each training sample is the true category of the annotated training sample; corresponding to each sample subset, the sample processing module uses the n-1 first neural network models of the sample subset to identify each training sample in the subset respectively, and obtains n-1 first prediction labels for each training sample. It can be used to: use the n-1 first neural network models corresponding to the sample subset to classify and identify each training sample in the sample subset respectively, and obtain n-1 first prediction labels for each training sample, wherein the first prediction label is the predicted category of the training sample.

[0039] On the other hand, an embodiment of the present application further provides a data classification device, the device comprising:

[0040] A data acquisition module, used to acquire data to be classified;

[0041] A data processing module is used to input the data to be classified into a target classification model to obtain a category recognition result of the data to be classified;

[0042] In which, the target classification model is obtained by training the initial classification model using a target training data set with annotated labels, and the target training sample set is obtained by processing the initial training data set of the initial classification model using the training sample processing method provided in any optional embodiment of the present application, and the annotated label of each training sample in the target training data set represents the true category of the training sample.

[0043] On the other hand, an embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the training sample processing method provided in the embodiment of the present application, or implements the data classification method provided in the embodiment of the present application.

[0044] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training sample processing method provided in the embodiment of the present application, or implements the data classification method provided in the embodiment of the present application.

[0045] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the training sample processing method provided in the embodiment of the present application, or implements the data classification method provided in the embodiment of the present application.

[0046] The beneficial effects of the technical solution provided by the embodiments of the present application are as follows:

[0047] The training sample processing method provided in the embodiment of the present application can automatically, efficiently and quickly screen out high-quality target samples (i.e., first target samples) in the initial training data set based on the training samples in the initial training data set themselves, so that a high-quality target training data set can be constructed based on these screened high-quality target samples, providing a basis for training a high-performance neural network model. Since this process can be implemented without relying on manpower, while improving the efficiency of sample verification, it can also greatly reduce human resources and better meet the needs of actual applications.

[0048] When the training sample processing method is implemented, for each sample subset in the initial training data set, the first neural network model obtained by training multiple other sample subsets in the initial training data set can be used to predict and identify each training sample in the sample subset to obtain a first predicted label. Since each training sample has an annotated label, if the annotated label of the training sample is correct and the annotation consistency of the training samples in each sample subset is relatively high, then the first predicted label of the training sample predicted by the model trained by other sample subsets should be consistent with the annotated label of the training sample. Therefore, the annotation quality of the first target sample screened out based on the first predicted label and the true label of the sample is guaranteed and can be used as the sample in the target training data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0050] Figure 1 A flowchart of a method for evaluating the quality of sample annotations in a training dataset provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram illustrating the principle of constructing multiple training sets provided in an embodiment of the present application;

[0052] Figure 3 A graph showing the changing trends of the evaluation indicators of the model corresponding to each training set when the model is trained by sampling multiple training sets provided in an embodiment of the present application;

[0053] Figure 4 A flowchart of a training sample processing method provided in an embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of the principle of preliminary screening of high-quality samples provided in the examples of this application;

[0055] Figure 6 This is a schematic diagram of the principle of further screening high-quality samples provided in the embodiments of this application;

[0056] Figure 7 A schematic diagram of the structure of a data processing system in an application scenario of the sample processing method provided in an embodiment of the present application;

[0057] Figure 8 A flowchart of a training sample processing method provided in an embodiment of the present application;

[0058] Figure 9 A flowchart of a neural network model training method provided in an embodiment of the present application;

[0059] Figure 10 A schematic diagram of the structure of a training sample processing device provided in an embodiment of the present application;

[0060] Figure 11 A schematic structural diagram of an electronic device applicable to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0062] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B". When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A including A1 or A2 or A3, and can also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0063] In the field of artificial intelligence, in order to obtain a neural network model that meets application requirements, the model needs to be trained. Using labeled training samples to train the neural network model is one of the common training methods. At present, the labeling of training samples is usually done by experts who provide labeling criteria and labeling rules. Labelers then label according to these criteria and rules. After the labeling is completed, a sampling manual inspection is carried out. If the set accuracy is achieved, the model passes. Otherwise, it is re-labeled. The quality of the labeling of training samples is crucial to the subsequent model training. However, the existing solutions do not have at least the following problems:

[0064] 1) A very detailed explanation is required on the labeling standards and rules. However, even so, it is still possible to encounter data samples that were not anticipated before (hereinafter referred to as "rare samples") during the labeling process. These samples are difficult to distinguish during labeling, and the labelers need to communicate with the experts again based on their own judgment to confirm the labels of the rare samples. However, since labelers often lack professionalism, it is difficult to accurately and comprehensively select all rare samples, or to submit non-rare samples to experts for labeling. If all rare samples are not selected, the accuracy of their labels will be greatly challenged, which is not conducive to subsequent model training. In addition, since it is impossible to perform the next step of oversampling on rare samples that have not been selected, their number in the training set is small, and the trained model has poor predictive power for such samples. If the labelers mistakenly submit non-rare samples to experts for labeling, the labor cost of the experts will increase.

[0065] 2) Since the number of training samples is usually large, the labels of the samples are usually labeled by multiple different labelers. However, due to the different understandings of labeling standards and criteria by different labelers, the accuracy and consistency of the labels of the training samples obtained in the end may also be problematic, requiring manual verification.

[0066] 3) After labeling, manual labor is still too dependent, resulting in high labor costs. Furthermore, manual inspection is inefficient and subjective, leading to potential bias in the results.

[0067] Therefore, how to more efficiently and accurately verify the samples with annotations and improve the sample quality of the training data set (including improving the accuracy and consistency of the annotation labels of the training samples in the training data set) is still a technical problem that needs to be improved. The embodiment of the present application is a training sample processing method, data classification method, device and electronic device proposed to solve at least one of the problems existing in the existing solutions. The training sample processing method provided by the embodiment of the present application can effectively improve the quality of the training data set. Furthermore, when the target training data set obtained by the method provided by the embodiment of the present application is used to train the model, the model performance can be effectively improved, so that when the trained model is used to process the data to be processed, the data processing effect can be effectively improved.

[0068] The solutions provided in the embodiments of the present application relate to artificial intelligence (AI) technology, and specifically to the training and application of neural network models. For example, based on the training sample processing methods provided in the embodiments of the present application, a higher-quality target training data set for model training can be obtained. Based on the target training data set, a machine learning (ML) method can be used to train a neural network model with better performance. The trained neural network model can also be used to process data to be processed, for example, data can be classified and identified based on a trained classification model.

[0069] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. With the research and advancement of AI technology, it has been researched and applied in a variety of fields, including common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robotics, smart healthcare, intelligent customer service, connected vehicles, autonomous driving, and smart transportation. We believe that as the technology develops, this technology will be applied in even more fields and play an increasingly important role.

[0070] Optionally, the data processing involved in the methods provided in the embodiments of the present application can be implemented based on cloud technology. For example, the training sample processing method provided in this application can be implemented based on cloud technology, and cloud technology can be used to process the initial training data set to obtain the target training data set. For example, various data calculations during the processing process can use cloud computing. Optionally, the training of a neural network model based on the target training data set can also be implemented based on cloud technology, and the storage of the initial training data set or the target training data set can also use cloud storage.

[0071] It should be noted that the data modes of the training samples, data to be processed (such as data to be classified) involved in this application are not limited in the embodiments of this application and can be any type of data, for example, can include but not be limited to one or more of text, video, image or voice data. The specific data modes of the training samples and the data to be processed are determined by the actual application requirements and application scenarios. For example, the neural network model in the embodiments of this application can be a text classification model. Accordingly, the training samples in the initial training data set are sample texts, and the annotation labels of the sample texts are the true categories of the annotated sample texts. For another example, the neural network model in the embodiments of this application can be an image recognition model used to identify whether an image pair is a similarity image pair. The training samples in the initial training data set are sample image pairs, and the annotation labels of the sample image pairs represent the labels of whether the two annotated images are similar. For example, if the label is 1, it means that the two images are annotated as similar images, and if the label is 0, it means that the two images are annotated as non-similar images. For another example, the neural network model can also be a speech classification model, which can be used to identify which object the speech input into the model belongs to. Then the training samples are sample speech, and the labels of the sample speech represent the object to which the annotated sample speech belongs.

[0072] The embodiments of the present application provide a method for processing training samples and a data processing method (such as a data classification method) that can be executed by any electronic device, such as a user terminal or a server. For example, a user can send a data set that needs to be classified to the server through his user terminal, and the server can deploy a trained classification model. The server can implement the data classification method provided by the embodiments of the present application to identify the categories of each data to be classified in the data set, and can send the classification results to the user's user terminal to provide them to the user, for example, to classify the image set stored in the user terminal.

[0073] Among them, the above-mentioned server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The above-mentioned user terminal (also referred to as user equipment) can be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device (such as a smart speaker), a wearable electronic device (such as a smart watch), a car terminal, a smart home appliance (such as a smart TV), an AR / VR device, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0074] Optionally, the method provided in the embodiment of the present application can be implemented as an independent application or a functional module / plug-in of an application. For example, the application can be specialized data classification software or other applications with data classification functions, through which the classification of the data to be processed can be achieved.

[0075] The training sample processing method provided in the embodiment of the present application can theoretically be applied to the processing of training data sets for any scenario / model, and is used to improve the quality of the training data sets, thereby improving the performance of the neural network model trained based on the processed training data sets, and improving the data processing effect of the model.

[0076] For example, the neural network model provided in the embodiment of the present application can be a risk control model, which can predict the risk level of a business (or the risk level of a target object corresponding to the business data) based on business data. Different risk levels in this scenario can be regarded as different categories. For example, if there are two risk levels, then the risk control model is a two-class model. The initial training data set can include training samples of each risk level. The training samples can specifically include sample business data with labeled labels, and the labels represent the risk levels of the sample business data. Through the training sample processing method provided in the embodiment of the present application, the initial training data set can be optimized to obtain a higher-quality target training data set, so that a risk control model with better performance can be trained based on the target training data set.

[0077] It should be noted that, in the optional embodiments of this application, the data related to the object involved, when the embodiments of this application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to the object, this data must be obtained with the authorization and consent of the object and in compliance with the relevant laws, regulations and standards of the country and region.

[0078] The following first introduces and explains some relevant technologies or technical terms involved in the embodiments of this application.

[0079] Annotated labels: also referred to as labels, refer to the labels that are annotated with training samples. They can be manually annotated labels or obtained by other means.

[0080] Data labeling accuracy: whether the data label is accurate. In the embodiment of the present application, the data label is the label of the training sample.

[0081] Data labeling consistency: This involves ensuring that data labeling standards are consistent, specifically, that ambiguous data is consistently labeled. For example, for the text "A Middle School's Canteen," the merchant category could be either a school or a restaurant. Therefore, data from such merchants should be consistently labeled, such as "school" or "restaurant."

[0082] Label balance: The training samples are balanced among various labels, that is, the number of samples of each label is relatively balanced.

[0083] Machine learning model training is highly dependent on data samples. Generally speaking, the following indicators are used to measure the quality of data samples:

[0084] (1) Label balance; (2) Correctness, i.e., the accuracy of the label; (3) Consistency, i.e., whether the data labeling standards are consistent before and after.

[0085] The training sample processing method provided in the embodiment of the present application can be used to check the correctness and consistency of the annotation after the data is labeled, and can improve these two indicators at the same time, thereby improving the effectiveness of data annotation. The method provided in the embodiment of the present application measures the correctness and consistency of data annotation (i.e., the annotation label of the training sample) by using a neural network model obtained by training different sample subsets to predict the same sample. The scheme has low manual dependence. When the neural network model is trained based on the target training data set obtained by adopting this processing method, the performance of the model obtained by effective training can be achieved.

[0086] Optionally, the method provided in this application can also automatically screen out rare samples and improve the accuracy of the annotation labels of rare samples, thereby improving the model's prediction / recognition ability for rare samples.

[0087] Optionally, the method provided in this application, on the premise of being able to automatically screen out rare samples, can also increase the number of rare samples during model training through sampling processing, and further specifically improve the trained model's prediction ability for rare samples.

[0088] In order to better illustrate and understand the solution provided by the embodiment of the present application and its practical value, the concept of the solution of the present application is explained below.

[0089] The solution provided by the embodiments of this application can be divided into two aspects: one is to verify the quality of data annotation (that is, the quality of the annotation labels of the training samples, including the correctness and consistency of the data annotations); the other is to improve the quality of data annotation. The following first explains the implementation principles of these two aspects.

[0090] (1) Automatic verification method for the correctness and consistency of data annotation

[0091] When labeling training samples, the samples to be labeled (the initial training dataset to be labeled) are typically randomly divided and assigned to multiple staff members for labeling. In theory, if the data annotations are accurate and consistent, then two different labeled samples (training samples with labeled labels) should have similar predictive power. In other words, neural network models trained using training samples labeled by different labelers should have similar predictive power (which can be evaluated using model performance evaluation metrics).

[0092] Taking the neural network model as a binary classification model and the model performance evaluation indicator as the F1 score as an example, if the labels of the samples annotated by each staff member are accurate and consistent, then for the samples annotated by one staff member, when the classification model trained with the samples annotated by that staff member and the classification model trained with the samples annotated by other staff members are used to predict the samples annotated by that staff member (that is, to classify and identify the samples annotated by that staff member), they should have similar F1 scores (the F1 score is an indicator used in statistics to measure the accuracy of a classification model. It takes into account both the accuracy and recall of the classification model. The larger the F1 score value, the better the model). It is the harmonic mean of precision and recall. The expression of the F1 score is as follows:

[0093]

[0094]

[0095]

[0096] Among them, precision represents the proportion of actual positive samples in the samples predicted to be positive. The actual positive samples are samples with positive labels. It can be understood that positive and negative are relative concepts. For example, the sample is an email text, and the purpose of training a binary classification model is to identify whether the email text is spam. Then the spam text can be used as a positive sample, and the non-spam text can be used as a negative sample. TP represents the number of positive samples predicted to be correct, FP represents the number of positive samples predicted to be incorrect, FN represents the number of negative samples predicted to be incorrect, and recall represents the proportion of samples predicted to be positive in all actual positive samples. Based on the above principles, an optional implementation method of the automatic verification method for the correctness and consistency of data annotation provided by this application may include the following steps:

[0097] The samples marked by each staff member (all samples marked by a staff member can be called a sample subset) can be separately processed as follows: Figure 1 The following operations of steps S1 to S4 shown in FIG.

[0098] Step S1: Construct multiple training data sets and a test set corresponding to the sample subset.

[0099] The number of training data sets may be 3 or 4. The following takes 4 data sets as an example for explanation. Step S1 can be implemented by the following operations:

[0100] 1. Randomly divide the samples marked by the staff into k1 equal parts (where k1 ≥ 2), randomly select one equal part as the test set, and take the other equal part as the candidate training set.

[0101] 2. Aggregate the samples labeled by other staff members except the samples labeled by the staff member, and randomly divide the aggregated samples into k2 equal parts (where k2 ≥ 2), and randomly select two equal parts as candidate training sets.

[0102] Assume that there are t workers in total (t≥2). If t≥3, optionally, k2=(t-1)k1 to ensure that the number of samples in each candidate training set is as close as possible.

[0103] 3. Based on the candidate training sets in steps 1 and 2, four training sets are obtained. The four training sets correspond to the following four groups of samples: Group 1 includes one candidate training set in step 1 and one candidate training set in step 2; Group 2 includes two candidate training sets in step 2; Group 3 includes the candidate training set in step 1. Optionally, Group 3 may also include another equal portion of Step 1 excluding the candidate training set; Group 4 includes the candidate training set in step 1 and one candidate training set in step 2. Four sets of training data can be obtained through the above combination. Optionally, only one of Group 1 and Group 4 may be used.

[0104] It can be seen that the four training sets corresponding to the subset of images annotated by a staff member are a combination of three different training samples: all the samples in one training set are annotated by the staff member, all the samples in one training set are annotated by other staff members, and the samples in one or two training sets include both the samples annotated by the staff member and the samples annotated by other staff members. Figure 2 As shown in the figure, the rectangular boxes filled with diagonal lines represent candidate training sets annotated by other staff members, the rectangular boxes filled with vertical lines represent candidate training sets annotated by the staff member, and the test set is the sample set annotated by the staff member. It should be noted that the third training set and test set can use different sample sets. The difference here means that at least some of the training samples in the training set and test set are different.

[0105] Step S2: Use each training set to train the initial neural network model to be trained to obtain the neural network model corresponding to each training set.

[0106] Step S3: Use the test set to test each neural network model obtained by training to obtain the index value of the model evaluation index corresponding to each neural network model.

[0107] Step S4: Evaluate the data annotation quality of the initial training data set based on the indicator values ​​corresponding to each neural network model.

[0108] like Figure 2 The four training sets shown in the figure were used to train the initial neural network models. The first set was first trained with samples labeled by other researchers, and then with samples labeled by the researcher. The second set was trained entirely with samples labeled by other researchers. The third set was trained entirely with samples labeled by the researcher. The fourth set was first trained with samples labeled by the researcher, and then with samples labeled by other researchers. When testing the models trained with each training set, the F1 score (a model evaluation metric) of the test set was observed as the training samples varied.

[0109] In theory, if the data annotation quality is poor, the F1 score of the test set will show obvious Figure 3 The differentiation shown in: Figure 3 The curves with numbers 1, 2, 3, and 4 in the figure represent how the F1 scores for the training sets from Groups 1 to 4 change over the training process. The model trained with samples labeled by a single staff member (Group 3) consistently has the highest F1 score. Group 4 initially performed similarly to Group 3, but after training with data labeled by others, its F1 score dropped significantly. The model trained with data labeled by others (Group 2) consistently has the lowest score. Group 1 initially performed similarly to Group 2, but after training with data labeled by the staff member, its score increased significantly until it matched Group 4. If the data annotation quality is high, the difference in F1 scores between these groups should be small.

[0110] Quantitatively, it can be To measure the accuracy and consistency of the data annotated by the annotator, that is, the annotation quality of the sample, among which F1 i represents the F1 score of the model obtained from the i-th training set. The lower this index is, the better the accuracy and consistency of the data. The following proof shows that the better the accuracy and consistency of the data, that is, the closer the three F1 scores are, the following three conditions must be met:

[0111] 1) F13-F12 is smaller, so given F14, Apparently smaller;

[0112] 2) F13-F14 is smaller, so given F12, smaller;

[0113] 3) F14-F12 is smaller, so given F13, Smaller.

[0114] Therefore, the above-mentioned method of grouping the model with different training data sets can be adopted to train the model, and the labeling quality of the labeled training samples can be evaluated using the model evaluation indicators. The evaluation results (i.e., indicator values) of the model evaluation indicators based on multiple sets of training data can be used to measure the labeling quality (also known as the data quality of the training data set).

[0115] The above description of the method of verifying the annotation quality of training samples uses the F1 score as the model evaluation indicator. It can be understood that in actual applications, the corresponding model evaluation indicator can be selected according to the type of model and the actual application requirements. The expression for measuring the annotation quality of the sample based on the indicator value is not unique. As long as the evaluation method conforms to the principle of the above verification method provided in the embodiment of the present application, it is theoretically acceptable. For example, the evaluation can be directly performed based on the difference between multiple groups of corresponding indicator values. The smaller the difference, the better the annotation quality is theoretically.

[0116] (2) Automatically improve the accuracy and consistency of data annotation

[0117] Under the measurement standard of the annotation quality of the above-mentioned data provided in the embodiment of the present application, the embodiment of the present application also proposes a method for automatically improving the correctness and consistency of data using a neural network model (the initial neural network model to be trained).

[0118] If after the above verification, it is found that the annotation quality of the training samples still needs to be improved, for example, There is still room for improvement (for example, if is greater than the expected threshold, indicating that the annotation quality is not ideal and needs to be improved), the data annotation quality improvement method provided in the embodiment of the present application can be used to improve the annotation quality of the training samples. For example, the improvement method can make the classification model corresponding to Reach the expected threshold or fall below it.

[0119] In the actual implementation of the improvement method, the method can be executed once or multiple times on the annotated initial training dataset until the quality of the obtained target training dataset meets the annotation quality requirements, such as If the algorithm is executed multiple times, the initial training dataset for the first execution is the training samples annotated by the annotator, and the initial training dataset for subsequent executions is the target training dataset obtained in the previous execution.

[0120] The following describes a method for improving the annotation quality of training samples provided in an embodiment of the present application. Optionally, Figure 4 As shown, the method may include the following steps:

[0121] Step 1: Screen samples with high accuracy and consistency through group training (first target samples)

[0122] When labeling training samples, the data can be divided into n equal parts and labeled by different staff members. In the following, taking n=4 as an example, the labeled initial training data set includes 4 sample subsets. The first step is to filter out the first target sample with high labeling quality from the 4 sample subsets, and the other samples except the first target sample are regarded as pending samples. This step can be performed by performing the following steps on each of the 4 sample subsets: Figure 4 The operations of steps S11 to S13 shown in the figure filter out the first target sample in each sample subset. The following takes any sample subset as an example to illustrate the implementation principle of the first step. For the sake of description, the any sample subset is called the current sample subset, and the n-1 sample subsets other than the current sample subset are called other sample subsets. The details are as follows:

[0123] Step S11: For the current sample subset, each of n-1 other sample subsets is used to train the initial neural network model respectively to obtain a first neural network model corresponding to each sample subset, and a total of n-1 first neural network models are obtained.

[0124] Step S12: using n-1 first neural network models to identify each training sample in the current sample subset, respectively, to obtain an identification result, ie, a first prediction label, and obtaining n-1 first prediction labels for each training sample.

[0125] Step S13: For each training sample, determine whether the training sample is the first target sample based on the matching degree between the n-1 predicted labels of the training sample and the labeled label of the training sample.

[0126] Taking n=4 as an example, for each sample subset annotated by a staff member, the model can be trained first using the data annotated by the other n-1 staff members (i.e., the other 3 sample subsets) as a training set, and then tested on the data annotated by the staff member (i.e., the sample subset annotated by the staff member). That is to say, for each sample subset, the initial neural network model is first trained using the other 3 sample subsets except the subset to obtain 3 first neural network models, and these 3 models are used to predict / identify each sample in the sample subset respectively to obtain 3 prediction labels.

[0127] Optionally, for each training sample, when the degree of matching between the n-1 predicted labels of the training sample and the labeled label of the training sample meets a condition, the sample is determined to be the first target sample. The condition may include but is not limited to that the n-1 predicted labels are all the same as the labeled label of the training sample, or that the number of predicted labels in the n-1 predicted labels that are the same as the labeled label of the training sample is greater than a first set number, and the maximum value of the first set number is n-1.

[0128] As an alternative approach, when n=4, for each sample, if the three predicted labels for the sample are consistent with its annotated label (annotated label), the sample is directly passed and its label is locked, and the annotated label of the sample is considered correct. The sample can be used as the first target sample, that is, the sample in the target training dataset; if the predicted label and the annotated label are not all consistent, the sample is pending. By performing the above operation on each sample subset separately, the first target sample and the pending sample in each sample subset can be screened out.

[0129] The schematic diagram of the first step is as follows Figure 5 As shown, when n=4, there are 4 sample subsets, corresponding to Figure 5 The four large unfilled rectangular boxes shown on the left in the figure are used for each sample subset. Figure 5 Taking the fourth sample subset (test subset shown in the figure) as an example, the other three sample subsets are used to train the initial neural network model respectively to obtain three first neural network models. These three models are then used to predict each training sample in the fourth sample subset. After that, the first target sample in the fourth sample subset is screened out based on the three predicted labels and labeled labels of each training sample. The other training samples in the fourth sample subset except the first target sample are the pending samples in the subset. In this way, the first target sample and the pending samples in the four subsets can be determined, as shown in the following example: Figure 5 The filled rectangular box on the right side of the figure represents the first target sample in each subset, and the unfilled small rectangular box represents the pending sample in each subset.

[0130] Step 2: Train the model with high-quality samples and filter out rare samples

[0131] After the first step above, a large amount of data with high accuracy and consistency can be screened out, namely Figure 5 The filled rectangle in the figure shows the portion of the data. After verification with data annotated by other researchers, this data still matches the labels of the researcher. This data can be directly used for model training, that is, it can be directly used as samples in the target training dataset. There are two possible reasons for inconsistently labeled data: one is that the data is mislabeled, that is, the sample's label is incorrect; the other is that the data sample is rare, and the model trained with other data samples cannot accurately predict its label.

[0132] Based on the above two points, we take the remaining training samples except the first target sample in the sample subset corresponding to each worker as a set of pending data (to-be-determined subset). Still taking n=4 as an example, there are 4 groups of to-be-determined subsets.

[0133] The purpose of the second step is to train the initial neural network model based on the high-quality samples screened out in the first step, and further screen out the second target samples in each pending subset through the trained model. At the same time, rare samples in each pending subset can be determined, that is, training samples in the pending subset other than the second target samples.

[0134] The second step may include the following operations: Figure 4 Steps S21 to S24 shown in FIG are specifically as follows:

[0135] Step S21: merging the first target samples in the n sample subsets to obtain a first target sample set, taking the training samples other than the first target sample as pending samples, and obtaining pending subsets corresponding to each of the n sample subsets, wherein the pending subset corresponding to each sample subset includes all pending samples in the positive sample subset;

[0136] Step S22: merging the first target sample set with each of the to-be-determined subsets to obtain n first mixed sample sets, and using the n first mixed sample sets to train the initial neural network model to obtain n second neural network models;

[0137] Step S23: For each undetermined subset, n-1 second neural network models other than the second neural network model corresponding to the undetermined subset are used to identify each undetermined sample in the undetermined subset to obtain n-1 second prediction labels;

[0138] Step S24: Determine the second target sample in the pending samples based on the matching degree between the n-1 second predicted labels and the labeled labels of each pending sample.

[0139] like Figure 6 The implementation principle of the second step shown in is to first aggregate all the first target samples filtered out in the first step, as shown in Figure 6 On the left side, the first target samples in the four sample subsets are merged to obtain a high-quality first target sample set, and a group of pending data is added to the first target sample set, that is, the first target sample set is merged with each group of pending data, that is, Figure 6 The undetermined subsets shown in are merged into , resulting in four new sets of data (i.e., four first mixed sample sets). The four first mixed sample sets are used to train the initial neural network model, respectively, to obtain four second neural network models. Subsequently, for each sample in each set of undetermined data (undetermined sample), three second neural network models trained with the mixed sample sets corresponding to the other sets of undetermined data are used to predict it, obtaining three second predicted labels.

[0140] The addition of undetermined data may introduce some noise into the model, but because the first target sample still accounts for the majority of the mixed sample set, the noise introduced is actually minimal. This undetermined data may contain rare samples similar to those in other undetermined data, so adding it as a training set can improve the model's predictive power for these rare samples.

[0141] After the second step, each pending sample should have the second predicted labels predicted by n-1 models and its own annotated labels. If the second predicted labels predicted by n-1 models match its own annotated labels, for example, n-1 second predicted labels are consistent with the annotated labels, or the number of second predicted labels consistent with the annotated labels is not less than the second set number, then it means that the label of the pending sample is consistent with the first target sample and can be passed directly, and the label is locked, that is, the annotated label of the pending sample can be judged to be correct, and the pending sample can also be used as the target sample (second target sample). For other samples that are inconsistent with manual and model judgments (samples other than the second target sample in the pending subset), they are usually rare samples, such as Figure 6 As shown in , through the second step, the second target sample and the rare sample are screened out from the undetermined subset. It can be seen that the method provided by the embodiment of the present application can not only screen out samples with high annotation quality, but also automatically screen out rare samples.

[0142] Step 3: Processing of rare samples

[0143] The third step is to process the rare samples that have been screened out, such as Figure 4In step S31 shown in FIG, the labels of rare samples can be re-acquired, and the rare samples with new labeled labels can be used as the third target samples. Finally, the first target samples, the second target samples, and the third target samples can be combined to obtain a target training dataset with improved accuracy and consistency of the labeled labels, as shown in FIG. Figure 4 Step S32.

[0144] Since the rare samples screened out are usually small in number, as an optional method, they can be directly labeled by experts, that is, new labeling labels can be obtained again. These re-labeled rare samples can be added to the target training dataset as the third target samples.

[0145] As another approach, the present application also provides a method for automatically determining new annotation labels for rare samples. It is understandable that the newly determined annotation labels may be consistent with the original annotation labels or may be inconsistent. The method is implemented as follows:

[0146] The first target samples screened out in the first step and the second target samples screened out in the second step are merged to obtain a second target sample set. All rare samples in each sample subset are taken as a rare sample set to obtain n rare sample sets. The second target sample set and each rare sample set are merged to obtain n second mixed sample sets. The initial neural network model is trained using these n second mixed sample sets to obtain n third neural network models. For each sample in each rare sample set, n-1 models of the n third neural network models, except the third neural network model corresponding to the rare sample set, are sampled to predict the sample to obtain n-1 third prediction labels.

[0147] Through the above operation, for each rare sample, n-1 third predicted labels are obtained. A new label for the rare sample can be re-determined by voting based on these n-1 third predicted labels. Specifically, the mode of the n-1 third predicted labels can be selected as the final label. This approach completely eliminates the need for manual labeling while still ensuring data consistency.

[0148] The new annotation labels are used as the final annotation labels of rare samples. These samples with the new annotation labels can be used as third target samples and added to the target training dataset. The training dataset with improved data quality can be used for training neural network models.

[0149] After obtaining a target training data set with high annotation quality, the neural network model to be trained can be trained based on the training data set to obtain a high-performance neural network model.

[0150] Step 4: Training the neural network model

[0151] Optionally, after obtaining the target training data set, the initial neural network model to be trained can be directly trained in a conventional training manner to obtain a trained neural network model that meets the training end conditions. The trained model can be used to identify the data to be processed. For example, if the initial neural network model is a classification model, then the initial training data set is a training data set with labeled labels corresponding to the classification model. After obtaining the target training data set through the training sample processing method provided in the embodiment of the present application, the classification model to be trained can be trained with the target training data set to obtain a trained classification model, so that the classification model can be used to perform type identification on the data to be classified.

[0152] Since the number of rare samples in the target training data set is relatively small, the trained model's predictive ability for these samples is still relatively weak. In order to further improve the trained model's predictive ability for rare samples, optionally, when training the model, the rare samples can be oversampled to increase the amount of data of the rare samples participating in the model training. As an optional method, boostrap (self-service sampling method) can be used to oversample rare samples to increase the amount of data of this part of the samples. For example, during training, n samples can be randomly selected from the batch of rare samples each time, and repeated m times to obtain new rare samples (that is, rare samples are repeatedly added to the training data set). As another optional method, clustering can also be used to achieve oversampling of rare samples, that is, this batch of rare samples can be used as seeds and clustered in a larger data set to obtain similar rare samples, and these similar rare samples are also added to the target training data set.

[0153] Compared with existing solutions that require manual sampling to check sample data quality, the method provided in the embodiments of the present application can realize automated data quality inspection, and qualitatively and quantitatively describe the correctness and consistency of the data by predicting the same sample with different samples. It has low manual dependence and is more accurate than manual in judging consistency.

[0154] The embodiment of the present application further designs a data quality improvement plan. This method can screen samples with high accuracy and high consistency through group training, and can also detect rare samples through the screened high-quality samples, so that experts can label rare samples or use automation to obtain new labeling labels for rare samples with improved standard label accuracy, so that rare samples can also be involved in model training, thereby improving the model's prediction ability for rare samples during the application process and better meeting actual application needs.

[0155] The method provided in the embodiment of the present application can detect rare samples in the training data set more objectively and accurately, while the existing scheme can only be formulated by experts in detail at the beginning of the labeling. If an unexpected sample is encountered during the labeling process, the labeling personnel and the experts will communicate to label it. After the labeling is completed, if it does not meet the standard, it can only be labeled again. However, due to the lack of professionalism of the labeling personnel, the evaluation of rare samples is often not accurate enough, resulting in the failure of rare samples to be identified, their labels may be inaccurate, and they cannot be oversampled, and the model's predictability for them is also low. In addition, if re-labeling is required, the labor cost will be greatly increased, and the scheme provided in the embodiment of the present application can cleverly solve these two problems.

[0156] The training sample processing method provided in the embodiment of the present application can theoretically be applied to the processing of training data sets of neural network models with various functions, such as but not limited to classification models, similarity discrimination models, etc., wherein the classification model can be applied to any scenario with classification requirements, for example, but not limited to text classification (such as email classification, classification of comment data), image / video classification, business risk control (risk levels can be judged based on business data, and different risk levels correspond to different types of classification models), etc. It can be understood that for different application requirements, the model structure of the neural network model can be constructed according to the requirements, and the data form of the initial training data set corresponding to the model can also correspond to the requirements. For example, if the neural network model is an image classification model, then the training samples in the initial training data set are images of various types, and the annotation labels of the training samples represent the types of the annotated images.

[0157] The following text classification scenario is used to briefly illustrate the practical value of the solution provided by the embodiment of the present application. In this scenario embodiment, the neural network model to be trained is a text classification model. The application of the trained text classification model to email classification (such as spam or non-spam) is used as an example for illustration, then the training sample is a sample email (text data of the email).

[0158] Figure 7 A structural diagram of a data processing system in this application scenario is shown in FIG. Figure 7 As shown in , the data processing system may include a training server 10, a mail server 20 and a terminal device 30, wherein the mail server 20 may be an application server that provides mail applications and can provide mail services to users of the application. The terminal device 30 may be a user terminal of any user. When the user uses the application through the terminal device 30, the terminal device 30 is in communication with the mail server 20. The training server 10 may be used to execute the training sample processing method provided in the embodiment of the present application, based on the initial training data set (such as Figure 7 The target training dataset is obtained by labeling the unlabeled training dataset by multiple staff members (as shown in the figure), and the initial text classification model is trained based on the target training dataset to obtain a trained text classification model (hereinafter referred to as the email recognition model, which is used to identify whether the email is spam). The training server 10 and the email server 20 can be communicatively connected. After the email recognition model is trained, the training server 10 can send the model to the email server. When the email is received, the email server 20 can call the email recognition model to identify whether the email is spam. Optionally, when the email is identified as spam, the email server 20 can provide the recipient with a prompt message that the email may be spam when sending the email to the recipient's terminal device 30.

[0159] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0160] Figure 8 The flowchart of a method for processing training samples provided by an embodiment of the present application is shown. The method can be executed by a user terminal or a server. The following description is made using the server as an example. Figure 8 As shown, the method includes the following steps:

[0161] Step S110: Obtain an initial neural network model to be trained and an initial training data set corresponding to the initial neural network model, wherein the initial training data set includes n sample subsets, n≥2.

[0162] Each sample subset includes multiple labeled training samples. Optionally, the labeled training samples in the initial training dataset may be labeled by multiple labelers, with each sample subset corresponding to one labeler. That is, the number of labelers may be n, and the multiple samples labeled by each labeler constitute a sample subset. For example, if the samples in the training dataset are labeled by four labelers, there will be four sample subsets.

[0163] Due to various reasons such as manual labeling and difficulty in distinguishing samples, it is difficult to ensure that the labels of all training samples are correct. Furthermore, since different people have different understandings of labeling standards and standard principles, even the same labeler may have inconsistent labeling standards for samples that are more difficult to distinguish. Therefore, the data labeling correctness and data labeling consistency of the training samples in the training data set need to be verified to provide labeling consistency and accuracy of the sample data.

[0164] The present application embodiment does not limit the modality of the data in the initial training data set and the model structure of the initial neural network model, and can be determined or constructed according to actual needs and application scenarios. As can be seen from the foregoing description, the training sample can be data of any modality, including but not limited to at least one of text, image or voice, and the initial neural network model can include but is not limited to a model based on a convolutional neural network, for example, it can be a classification model or a recognition model for identifying whether a pair of images is similar or whether they are images of the same object. It can be understood that the modality and annotation labels of the training samples in the training data set correspond to the model function of the neural network model (what the purpose of the training model is to make the model have), such as the training model is to enable the model to recognize the type of image, such as the first type of image (such as a character image) or the second category of image (landscape image). Accordingly, the training sample is the sample image, and the annotation label is the category label of the sample image, such as the annotation label of 1 represents a character image and the annotation label of 2 represents a landscape image.

[0165] After obtaining the initial training data set, the server can process the training samples in the data set by executing the following steps S120 to S150 to obtain the target training data set that is ultimately used for training the neural network model.

[0166] Step S120: training the initial neural network model based on each sample subset to obtain n first neural network models;

[0167] Step S130: For each sample subset, use n-1 first neural network models other than the first neural network model corresponding to the sample subset to identify each training sample in the sample subset, and obtain n-1 first prediction labels for each training sample;

[0168] It can be understood that for each training sample, its first predicted label is the recognition result of the sample predicted by the first neural network model. For example, the labeled label represents the labeled category, and the predicted label represents the category of the model predicted by the model.

[0169] Step S140: For each training sample, if a first number of labels in the n-1 first predicted labels of the training sample that are identical to the labeled label of the training sample is not less than a first set number, then the training sample is determined as a first target sample;

[0170] Step S150: Based on the determined first target samples, a target training data set corresponding to the initial neural network model is constructed.

[0171] Optionally, the target training data set may include part or all of the determined first target samples, and the first set number is an integer not greater than n-1. The first set number may be equal to n-1. At this time, if the n-1 predicted labels of the training sample are consistent with the labeled label, it is determined as the first target sample.

[0172] For the above n sample subsets, if the accuracy and consistency of the data annotation of the training samples in each subset are relatively high, then in theory, the recognition ability (prediction ability) of the n first neural network models trained using the n subsets should be similar. That is to say, for a training subset, the prediction ability of the model trained based on the subset should be similar to the prediction ability of the n-1 models trained using the other n-1 sample subsets.

[0173] Based on the above principles, the method provided in the embodiment of the present application adopts n sample subsets to train the initial neural network model respectively, and obtains a first neural network model corresponding to each subset, with a total of n first neural network models. Afterwards, for the training samples in a sample subset, n-1 first neural network models other than the first neural network model obtained by training the sample subset are adopted to identify each training sample in the sample subset respectively, and n-1 predicted recognition results are obtained, namely the above-mentioned n-1 first predicted labels. Since each training sample has a labeled label, if the labeled label of the training sample is correct and the labeled consistency of the training samples in each sample subset is relatively high, then the recognition result is obtained. Most of the n-1 first predicted labels of the training sample predicted by the other n-1 models should be consistent with the labeled label of the training sample. If the data quality requirement is particularly high, the n-1 first predicted labels should be the same as the labeled label. Therefore, the accuracy and consistency of the labeled label of the training sample can be evaluated based on the first number of labels in the n-1 first predicted labels of the training sample that are the same as the labeled label of the sample. If the first number of labels in the n-1 first predicted labels of the sample that are the same as the labeled label of the sample is not less than the first set number, the sample can be considered to be high-quality sample data and can be directly used as a target sample and added to the target training data set.

[0174] It is understandable that the maximum value of the first set number is n-1. At this time, when the n-1 first predicted labels of the training sample are all the same as the labeled label, the training sample is determined to be the target sample.

[0175] The training sample processing method provided in the embodiment of the present application can automatically, efficiently and quickly screen out high-quality target samples in the initial training data set, so that a high-quality target training data set can be constructed based on these screened high-quality target samples, providing a basis for training a high-performance neural network model. Since this process can be implemented without relying on manpower, while improving the efficiency of sample verification, it can also greatly reduce human resources and better meet the needs of actual applications.

[0176] As an optional solution, the initial neural network model may include an initial classification model. Accordingly, the annotated label of each training sample is the true category of the annotated training sample. However, the annotated true category may be incorrect. The solution provided by the embodiment of the present application can verify the annotated labels of the samples and improve the quality of the samples in the training data set. When the initial neural network model is an initial classification model, the n-1 first predicted labels of each training sample are the predicted category of the training sample.

[0177] In an optional embodiment of the present application, before the initial neural network model is trained based on each sample subset to obtain the first neural network model, the following steps may also be included:

[0178] At least one of the n sample subsets is used as a reference dataset. For each reference dataset, based on the n sample subsets, a first training set, a second training set, a third training set, and a test set corresponding to the reference dataset are constructed, wherein the first training set includes at least part of the training samples in the reference dataset and at least part of the training samples in n-1 sample subsets other than the reference dataset, the second training set is at least part of the training samples in the n-1 sample subsets, the training samples in the third training set and the test set are at least part of the training samples in the reference dataset, and the training samples in the test set are different from the training samples in the third training set;

[0179] For each reference data set, the initial neural network model is trained based on the three training sets corresponding to the reference data set to obtain three fourth neural network models, and the performance of each fourth neural network model is evaluated based on the test set and the model evaluation index to obtain an index value of each fourth neural network model. Based on the three index values, the data quality of the initial training data set corresponding to the reference data set is determined;

[0180] If the data quality corresponding to each reference data set meets the preset data quality conditions, the initial training data set is used as the target training data set;

[0181] The above-mentioned training of the initial neural network model based on each sample subset separately may include: if the data quality corresponding to each reference data set does not meet the preset data quality conditions, the initial neural network model is trained separately based on each sample subset.

[0182] In this optional embodiment, the labeling quality of the training samples in the initial training data set can be tested first. If the test result meets the required data quality conditions (that is, the labeling quality conditions), it means that the consistency and correctness of the labeling labels of the training samples in the initial training data set are relatively high. The initial training data set can be directly used as the target training data set. If the test result does not meet the above conditions, the above-mentioned operation of obtaining the target training data set based on the initial training data set provided in this application can be performed to obtain a target training data set with improved labeling quality. Based on this optional embodiment, automated verification of the labeling quality of the initial training data set can be achieved.

[0183] Among them, for each sample subset, the above three training sets can correspond to Figure 2 In the example shown, the first training set (or the fourth training set), the second training set and the third training set, optionally, the amount of sample data in the three training sets can be kept as balanced as possible, and the difference in the number of samples should not be too large. For example, for a sample subset, 3f samples can be randomly selected from the sample subset to form the third training set, and f samples can be randomly selected from each of the other three sample subsets, and the selected 3f samples can be used as the second training set. 1.5f samples can be randomly selected from the sample subset, and 0.5f samples can be randomly selected from each of the other three sample subsets to obtain the first training set. Of course, the method of constructing multiple training sets corresponding to sample subsets provided in the previous embodiment can also be used.

[0184] The above-mentioned model evaluation indicators can be selected based on the actual application requirements and the model type of the neural network model to be trained. For example, for a classification model, the evaluation indicator can be the F1 score, or it can be the precision or recall rate. The reference dataset can be one or more of the n sample subsets. If multiple sample subsets are selected as reference datasets, three indicator values ​​of the model evaluation indicators corresponding to each reference subset can be calculated. The data quality corresponding to the reference subset (i.e., the quality of the sample annotations in the initial training dataset) can be determined based on the three indicator values. For example, for a classification model, the data quality can be obtained by measuring the annotation quality using the F1 score based on multiple training sets provided above. The quality of the initial training dataset can then be comprehensively evaluated based on the data quality corresponding to multiple reference datasets to determine whether it meets the preset conditions. For example, if the data quality corresponding to multiple reference datasets all meets the conditions or the fusion result of the data quality corresponding to multiple reference datasets (such as the average or other fusion method) meets the conditions, it can be considered to meet the preset conditions. For example, if the smaller the value of the data quality corresponding to the reference data set, the higher the quality, then satisfying the condition may also mean that the data quality is not greater than the first set threshold. If the larger the value of the data quality corresponding to the reference data set, the higher the quality, then satisfying the condition may also mean that the data quality is not less than the second set threshold.

[0185] If the judgment result does not meet the preset data quality conditions, the first target sample in the initial training data set can be screened out by executing the operations of steps S120 to S150 above, and a target training data set with high annotation quality can be constructed based on the first target sample.

[0186] After determining each first target sample in the initial training data set, for the remaining other training samples, the first number of labels in the n-1 first predicted labels of the sample that are the same as the labeled label of the sample is less than the first set number, indicating that the labeled label of the training sample and the multiple labels predicted by the model obtained by training other sample subsets have a large deviation, indicating that there is an error in the labeled label of the sample or that the sample is a sample that is relatively difficult to identify, and the accuracy of the label predicted by the model is not good. Therefore, it cannot be directly determined whether the sample can be used as a target sample.

[0187] For the convenience of description, in the embodiment of the present application, other training samples except the first target sample are referred to as pending samples.

[0188] As an alternative, the pending samples can be discarded directly, that is, the final target training data set can be directly based on the determined first target samples. However, this method will reduce the number of samples in the final training data set to a certain extent, which is not conducive to model training. Considering this problem, as another alternative, the pending samples can be provided to experts, who will manually verify the labels of these samples and then add the verified samples to the target training data set to ensure the quality and quantity of the samples in the target training data set. This solution can solve the problem of reducing the number of samples mentioned above, but if there are many pending samples, a large amount of expert resources will be required, the labor cost will be high, and the efficiency will be low.

[0189] In response to the problems existing in the above two optional solutions, the embodiment of the present application also provides another solution. Optionally, the training sample processing method provided in the embodiment of the present application may further include the following steps S141 to S143.

[0190] Step S141: using all the determined first target samples as a first target sample set, and training the initial neural network model based on the first target sample set to obtain a second neural network model;

[0191] Step S142: taking the training samples other than the first target sample in the initial training data set as pending samples, identifying each pending sample based on the second neural network model, and obtaining a second predicted label for each pending sample;

[0192] Step S143: Based on the second predicted label of each pending sample, determine the second target sample in each pending sample.

[0193] Since the first target samples are all high-quality samples with high consistency and accuracy of annotations screened out from various sample subsets, by training the initial neural network model based on the first sample set obtained by aggregating these high-quality samples, a second neural network model with higher model performance than the first neural network model can be obtained. Based on this, the second neural network model can be used to re-identify the pending samples to obtain the second predicted labels of the pending samples, and further, based on the second predicted labels of each pending sample, the second target sample can be screened out from each pending sample.

[0194] Accordingly, corresponding to this optional solution, in the above step S150, constructing the target training data set corresponding to the initial neural network model may include:

[0195] A target training data set is constructed based on the determined first target samples and second target samples.

[0196] Correspondingly, the target training data set may also include part or all of the second target samples.

[0197] By adopting the solution provided in the embodiment of the present application, high-quality second target samples can be automatically screened out again from the pending samples, thereby expanding the high-quality samples in the target training data set.

[0198] When determining the second target sample based on the labeled label and the second predicted label of the pending sample, as an optional method, if the second predicted label of the pending sample is consistent with its labeled label, the pending sample can be determined as the second target sample. This is because the second neural network model is trained based on a sample set containing a large number of high-quality training samples. Therefore, the second predicted label predicted by the model can be considered relatively accurate. If the second predicted label of the pending sample is consistent with its labeled label, the labeled label of the sample can be considered correct.

[0199] In order to better ensure that the determined second target sample is a high-quality sample, the present application also provides an optional solution. In this optional solution, the initial neural network model is trained based on the first target sample set to obtain the second neural network model, including:

[0200] For each of the n sample subsets, all pending samples in the sample subset are taken as a first pending subset; the n first pending subsets are divided into m second pending subsets, where n≥m≥2, each second pending subset includes at least one first pending subset, and each first pending subset belongs to only one second pending subset;

[0201] Merging the first target sample set with each of the second undetermined subsets to obtain m first mixed sample sets, and training the initial neural network model based on each of the first mixed sample sets to obtain m second neural network models;

[0202] At this time, the above-mentioned identification of each pending sample based on the second neural network model to obtain a second predicted label for each sample includes:

[0203] For each pending sample, use the m-1 second neural network models corresponding to the pending sample to respectively identify the pending sample, and obtain m-1 second prediction labels for the pending sample, wherein the m-1 second neural network models corresponding to the pending sample refer to the m-1 second neural network models other than the first model, and the first model is the second neural network model corresponding to the first mixed sample set to which the pending sample belongs (that is, the second neural network model trained by sampling the first mixed sample set to which the pending sample belongs);

[0204] Accordingly, the above-mentioned determining the second target sample in each of the pending samples based on the second predicted label of each of the pending samples includes:

[0205] Based on the m-1 second predicted labels of each pending sample, a second target sample in each pending sample is determined.

[0206] This step corresponds to the second step of the optional embodiment given above, that is, a scheme for further screening target samples from the pending samples.

[0207] Optionally, the value of m can be equal to n. In this case, the number of first to-be-determined subsets is equal to the number of second to-be-determined subsets, that is, one first to-be-determined subset can be used as a second to-be-determined subset.

[0208] Optionally, the value of m can be less than n. In this case, a second undetermined subset can include one or more first undetermined subsets, and different second undetermined subsets do not contain the same first undetermined subset. That is, a first undetermined subset can only belong to one second undetermined subset. The number of first undetermined subsets contained in different second undetermined subsets can be the same or different. For example, every two first undetermined subsets can be merged into one second undetermined subset. Different second undetermined subsets do not contain data from the same first undetermined subset. Alternatively, some second undetermined subsets can contain one first undetermined subset, while others can contain two first undetermined subsets.

[0209] As an optional approach, m second undetermined subsets can be determined based on the number of undetermined samples in each first undetermined subset, so that the amount of data of the undetermined samples contained in each second undetermined subset is as close as possible, or the number of samples in each second undetermined subset is not less than a set value. For example, the n first undetermined subsets can be sorted in order of the number of undetermined samples contained, from small to large, and the first undetermined subset after sorting is used as the initial current subset. The following operations are repeated until the number of samples contained in the current subset is not less than the set value and the current subset is not a merged subset:

[0210] Determine the number of pending samples contained in the current subset. If the number is less than the set value, merge the current subset into the next first pending subset of the current subset, and use the merged subset as the new current subset. If the number is not less than the set value and the current subset is the merged subset, use the current subset as a second pending subset, and use the next first pending subset after the current subset as the new current subset. If the number is not less than the set value and the current subset is not the merged subset, end the above operation and use the current subset and each first pending subset after the subset as a second pending subset.

[0211] Through this optional method, each second to-be-determined subset can contain a certain number of samples, which can avoid the problem that due to too few samples in the second to-be-determined subset, some first mixed sample sets are too different from the samples in the first target sample set, and these mixed sample sets have a very weak effect on improving the model performance. It is more conducive to improving the recognition ability of the model.

[0212] Optionally, the above-mentioned determining the second target sample in each pending sample based on the m-1 second predicted labels of each pending sample includes:

[0213] For each of the pending samples, the label with the highest proportion among the m-1 second predicted labels of the pending sample is determined as the new label of the pending sample;

[0214] Each pending sample with a new label is used as the second target sample.

[0215] That is to say, for each pending sample except the first target sample in the initial training data set, the label of the pending sample can be re-determined based on the m-1 second predicted labels of the pending sample, and each pending sample with the re-determined label can be used as the second target sample.

[0216] By adopting this optional solution, the consistency and correctness of the annotation labels of all training samples in the initial training data set can be automatically verified without human intervention, and a target training data set with higher quality than the initial training data set can be obtained. The solution is efficient and does not rely on human intervention.

[0217] Optionally, determining the second target sample in each pending sample based on the m-1 second predicted labels of each pending sample includes:

[0218] For each pending sample, if a second number of labels in the m-1 second predicted labels of the pending sample that are identical to the labeled label of the pending sample is not less than a second set number, the pending sample is determined as the second target sample.

[0219] Among them, the value of the second set number is not greater than m-1, and the specific value of the second set number can be configured according to actual needs. For example, the higher the requirement for sample quality, the larger the value of the second set number can be. Optionally, the second set number can be equal to m-1. At this time, all the second predicted labels of the pending sample are consistent with its marked label, and the pending sample will be identified as the second target sample.

[0220] Optionally, the method provided by this application may further include:

[0221] For each first undetermined subset, each undetermined sample in the undetermined subset except the second target sample is regarded as a rare sample, a new label is obtained for each rare sample, and the rare sample with the new label is regarded as a third target sample;

[0222] At this time, constructing the target training data set based on the determined first target samples and second target samples may include:

[0223] A target training data set is constructed based on each first target sample, each second target sample, and each third target sample.

[0224] The target training data set may further include part or all of the third target samples.

[0225] For the initial training data set, after determining the high-quality first target samples and second target samples contained therein, the remaining samples can be considered as samples that are difficult to determine whether their labeled labels are correct, which can be called rare samples. For these samples, one way is to discard them directly and use the determined first target samples and second target samples as the target training data set, but this method will reduce the number of samples in the target training data set compared to the initial training data set, and if the rare samples that are difficult to distinguish are discarded, the model of the neural network model trained based on the target training data set will have weaker predictive ability for data that are difficult to distinguish. Therefore, discarding rare samples does not utilize subsequent model training. In order to avoid these problems, the above-mentioned optional scheme provided in the embodiment of the present application can redetermine the labeled labels of rare samples and add these samples with updated labels as target samples to the target training data set, which not only ensures the number of samples in the data set, but also ensures the quality of the samples in the data set.

[0226] It is understandable that the new annotation labels of rare samples may be the same as or different from the original annotation labels of the samples.

[0227] The acquisition of new annotation labels for rare samples is optional. Since the number of rare samples in the training data set is usually not too large, and the rare samples in the embodiment of the present application are the samples remaining after two target sample screenings, the number of rare samples has been greatly reduced compared to the number of samples in the initial training data set. Therefore, these rare samples can be provided to experts, who will verify the annotation labels of these samples to obtain verified labels.

[0228] Although the method of verification by experts can ensure the accuracy and consistency of sample labels to the greatest extent, this method still requires human participation. In order to achieve full automation of the sample processing process, this application also provides another optional method for obtaining new labeling labels for rare samples. The process of re-obtaining new labeling labels for each rare sample by this method includes:

[0229] Merging all the determined first target samples and all the second target samples to obtain a second target sample set;

[0230] For each first undetermined subset, all rare samples in the undetermined subset are taken as a first rare sample set;

[0231] Divide n first rare sample sets into k second rare sample sets, where n≥k≥2, each second rare sample set includes at least one first rare sample set, and each first rare sample set belongs to only one second rare sample set;

[0232] Merge the second target sample set with each second rare sample set to obtain k second mixed sample sets;

[0233] Training the initial neural network model based on each second mixed sample set to obtain k third neural network models;

[0234] For each second rare sample set, using k-1 third neural network models other than the third neural network model corresponding to the rare sample set, to be determined, to identify each rare sample in the rare sample set, and obtain k-1 third prediction labels for each rare sample;

[0235] For each rare sample, the label with the highest proportion among the n-1 third predicted labels of the rare sample is determined as the new label of the rare sample.

[0236] Since the first target samples and the second target samples are a large number of high-quality training samples selected from the initial training data set through two rounds of screening, a third neural network model with good performance can be trained based on the second target sample set containing these target samples. As an optional solution, the second target sample set can be directly used to train the initial neural network model to obtain a third neural network model. Through the model, each rare sample is identified to obtain a third prediction label. Based on the label, a new annotation label for the rare sample can be determined. For example, the third prediction label of the rare sample can be directly used as the new annotation label of the sample.

[0237] As another optional scheme, a scheme similar to the m-1 second neural network models trained in the previous article can be adopted to divide the n first rare sample sets into k second rare sample sets, where the maximum value of k can be n, and the second target sample set is merged with each second rare sample set. Based on the merged k second mixed sample sets, k third neural network models are trained to obtain k third neural network models. Then, for each rare sample, k-1 third neural network models other than the second model can be used to identify the k-1 third prediction labels of the rare sample. The second model refers to the third neural network model trained using the second mixed data set to which the rare sample belongs. Afterwards, these k-1 third prediction labels can be combined to determine the new label label of the rare sample.

[0238] Optionally, the label with the highest mode (i.e., the highest proportion) among the k-1 third predicted labels can be voted as the label for the rare sample. Of course, during implementation, the label with the highest proportion among the k-1 third predicted labels can be determined first. If the number of labels with the highest proportion is not less than the third number, the label with the highest proportion can be directly used as the label. If the number of labels with the highest proportion is less than the third number, the rare sample can be discarded or provided to experts for further verification.

[0239] For example, assuming n=4, k=n=4, and the k-1 predicted labels of a rare sample are label 1, label 1, and label 0, then the label with the highest proportion is label 1. Label 1 can be used as the new label for the rare sample. Of course, the new label may also be the same as the original label. If they are different, the new label is used to replace the original label to obtain the third target sample.

[0240] At this point, the screened first target sample, the second target sample, and the third rare sample obtained by further processing the rare sample can be aggregated to obtain a target training data set with high annotation quality. As can be seen from the foregoing description, the training sample processing method provided in the embodiment of the present application can be applied to the optimization of training data sets of various neural network models used for data recognition (for example, category recognition of data to be processed, similarity recognition of pairs of data to be processed (such as image pairs), recognition of target objects, etc.). Based on the training sample processing method provided in the present application, the embodiment of the present application also provides a training method for a neural network model, such as Figure 9 As shown, the training method includes:

[0241] Step S910: Obtaining an initial neural network model to be trained and an initial training data set;

[0242] Step S920: Based on the initial training data set, the initial training data set is processed using the training sample processing method provided in any optional embodiment of the present application to obtain a target training data set corresponding to the initial training data set;

[0243] Step S930: Train the initial neural network model based on the target training data set to obtain a trained target neural network model, and recognize the data to be processed based on the target neural network model to obtain a recognition result.

[0244] For example, the initial neural network model is an initial image classification model, and each training sample in the initial training data set is a sample image with an annotated label. The annotated label represents the true category of the annotated sample image. The training processing method provided in any optional embodiment of the present application can be used to verify the annotated labels of the training samples in the initial training data set to obtain a target sample training data set. The annotated labels of some training samples in the target training data set may be modified. Compared with the initial training data set, the use of the target training data set can effectively improve the trained image classification model's ability to predict the category of the image to be identified.

[0245] Optionally, during the process of training the initial neural network model based on the target training data set, the training method can oversample the rare samples in the target training data set, and train the initial classification model based on the rare samples obtained by the oversampling process and the training samples other than the rare samples in the marked training data set.

[0246] The specific method of oversampling rare samples is not limited in the embodiment of the present application, and may include but is not limited to boostrap oversampling or cluster oversampling.

[0247] Taking data classification as an example, an embodiment of the present application further provides a data classification method, which may include:

[0248] Obtain data to be classified;

[0249] Input the data to be classified into the target classification model to obtain the category recognition result of the data to be classified;

[0250] Among them, the target classification model is obtained by training the initial classification model using a target training data set with annotated labels, and the target training sample set is obtained by processing the initial training data set using the training sample processing method provided in any optional embodiment of the present application. The annotated label of each training sample in the target training data set represents the true category of the training sample.

[0251] Corresponding to the training sample processing method provided in the embodiment of the present application, the embodiment of the present application also provides a training sample processing device, such as Figure 10 As shown, the training sample processing apparatus 100 may include an initial training set acquisition module 110 , a sample processing module 120 and a target training set acquisition module 130 .

[0252] An initial training set acquisition module 110 is used to acquire an initial neural network model to be trained and an initial training data set corresponding to the initial neural network model, wherein the initial training data set includes n sample subsets, n ≥ 2, and each sample subset includes multiple training samples with annotated labels;

[0253] The sample processing module 120 is configured to obtain a target training data set by performing the following operations:

[0254] The initial neural network model is trained based on each sample subset to obtain n first neural network models;

[0255] For each sample subset, the n-1 first neural network models corresponding to the sample subset are used to identify each training sample in the subset to obtain n-1 first prediction labels for each training sample. The n-1 first neural network models corresponding to each sample subset refer to the n-1 models in the n first neural network models excluding the first neural network model corresponding to the sample subset;

[0256] For each training sample, if a first number of labels in the n-1 first predicted labels of the training sample that are the same as the labeled label of the training sample is not less than a first set number, then the training sample is determined as a first target sample;

[0257] The target training set acquisition module is used to construct a target training data set corresponding to the initial neural network model based on the determined first target samples.

[0258] Optionally, the sample processing module is further configured to: use all determined first target samples as a first target sample set, train an initial neural network model based on the first target sample set, and obtain a second neural network model; use training samples other than the first target sample in the initial training data set as pending samples, identify each pending sample based on the second neural network model, and obtain a second predicted label for each pending sample; and determine a second target sample in each pending sample based on the second predicted label of each pending sample;

[0259] Accordingly, the target training set acquisition module can be used to construct a target training data set based on the determined first target samples and second target samples.

[0260] Optionally, the sample processing module may be configured to: for each of the n sample subsets, take all the to-be-determined samples in the sample subset as a first to-be-determined subset; divide the n first to-be-determined subsets into m second to-be-determined subsets, where n≥m≥2, each second to-be-determined subset includes at least one first to-be-determined subset, and each first to-be-determined subset belongs to only one second to-be-determined subset; merge the first target sample set with the m second to-be-determined subsets respectively to obtain m first mixed sample sets, and train the initial neural network model based on each first mixed sample set respectively to obtain m second neural network models;

[0261] For each pending sample, use the m-1 second neural network models corresponding to the pending sample to respectively identify the pending sample to obtain m-1 second prediction labels for the pending sample, where the m-1 second neural network models corresponding to the pending sample refer to the m-1 models among the m second neural network models excluding the first model, where the first model is the second neural network model corresponding to the first mixed sample set to which the pending sample belongs;

[0262] Based on the m-1 second predicted labels of each pending sample, a second target sample in each pending sample is determined.

[0263] Optionally, when determining the second target sample among each pending sample, the sample processing module can be used to: for each pending sample, if the second number of labels in the m-1 second predicted labels of the pending sample that are the same as the marked label of the pending sample is not less than a second set number, then the pending sample is determined as the second target sample.

[0264] Optionally, the sample processing module is further configured to: for each first undetermined subset, treat each undetermined sample in the undetermined subset except the second target sample as a rare sample, reacquire a new label for each rare sample, and treat the rare sample with the new label as a third target sample;

[0265] Accordingly, the target training set acquisition module can be used to construct a target training data set based on each first target sample, each second target sample, and each third target sample.

[0266] Optionally, the sample processing module can be used to re-obtain new annotation labels for each rare sample:

[0267] Merge all the determined first target samples and all the second target samples to obtain a second target sample set; for each first undetermined subset, take all the rare samples in the undetermined subset as a first rare sample set; divide the n first rare sample sets into k second rare sample sets, where n≥k≥2, each second rare sample set includes at least one first rare sample set, and each first rare sample set belongs to only one second rare sample set; merge the second target sample set with each second rare sample set respectively to obtain k second mixed sample sets; train the initial neural network model based on each second mixed sample set respectively to obtain k third neural network models; for each second rare sample set, use k-1 third neural network models other than the third neural network model corresponding to the second rare sample set to identify each rare sample in the second rare sample set respectively, and obtain k-1 third predicted labels for each rare sample; for each rare sample, determine the predicted label with the highest proportion among the n-1 third predicted labels of the rare sample as the new annotation label of the rare sample.

[0268] Optionally, when determining the second target sample in each pending sample based on the m-1 second predicted labels of each pending sample, the sample processing module may be configured to:

[0269] For each pending sample, the predicted label with the highest proportion among the m-1 second predicted labels of the pending sample is used to determine the new label of the pending sample; and each pending sample with the new label is used as the second target sample.

[0270] Optionally, the sample processing module can also be used to:

[0271] At least one of the n sample subsets is used as a reference dataset. For each reference dataset, based on the n sample subsets, a first training set, a second training set, a third training set, and a test set corresponding to the reference dataset are constructed, wherein the first training set includes at least part of the training samples in the reference dataset and at least part of the training samples in n-1 sample subsets other than the reference dataset, the second training set includes at least part of the training samples in the n-1 sample subsets other than the reference dataset, the training samples in the third training set and the test set include at least part of the training samples in the reference dataset, and the training samples in the test set are different from the training samples in the third training set;

[0272] For each reference data set, the initial neural network model is trained based on the three training sets corresponding to the reference data set to obtain three fourth neural network models, and the performance of each fourth neural network model is evaluated based on the test set and the model evaluation index to obtain an index value for each fourth neural network model, and the data quality of the initial training data set corresponding to the reference data set is determined based on the index values ​​corresponding to the three fourth neural network models;

[0273] If the data quality corresponding to each reference data set meets the preset data quality conditions, the initial training data set is used as the target training data set;

[0274] The sample processing module can be used to: if the data quality corresponding to each reference data set does not meet the preset data quality conditions, the initial neural network model is trained separately based on each sample subset.

[0275] Optionally, the initial neural network model is an initial classification model, and the annotated label of each training sample is the true category of the annotated training sample; when the sample processing module uses n-1 first neural network models to identify each training sample in the subset respectively and obtains n-1 first prediction labels for each training sample, it can be used to: use n-1 first neural network models to classify and identify each training sample in the subset respectively, and obtain n-1 first prediction labels for each training sample, wherein the first prediction label is the category of the predicted training sample.

[0276] Corresponding to the training method provided in the embodiment of the present application, the embodiment of the present application provides a training device for a neural network model, the device comprising:

[0277] A data acquisition module is used to obtain the initial neural network model to be trained and the initial training data set;

[0278] A data optimization module is configured to process the initial training data set using the training sample processing method provided in any optional embodiment of the present application based on the initial training data set to obtain a target training data set corresponding to the initial training data set;

[0279] The model training module is used to train the initial neural network model based on the target training data set to obtain a trained target neural network model.

[0280] Based on the training sample processing method provided in the embodiment of the present application, the embodiment of the present application further provides a data classification device, which includes:

[0281] A data acquisition module, used to acquire data to be classified;

[0282] The data processing module is used to input the data to be classified into the target classification model to obtain the category recognition result of the data to be classified;

[0283] Among them, the target classification model is obtained by training the initial classification model using a target training data set with annotated labels, and the target training sample set is obtained by processing the initial training data set of the initial classification model using the training sample processing method provided in any optional embodiment of the present application. The annotated label of each training sample in the target training data set represents the true category of the training sample.

[0284] It can be understood that the device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0285] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, the method in any optional embodiment of the present application can be implemented.

[0286] Figure 11 FIG. 1 shows a schematic structural diagram of an electronic device to which an embodiment of the present invention is applicable. Figure 11 As shown, the electronic device may be a server or a user terminal, and the electronic device may be used to implement the method provided in any embodiment of the present invention.

[0287] like Figure 11 As shown in FIG, the electronic device 2000 may mainly include at least one processor 2001 ( Figure 11 1 ), memory 2002, communication module 2003 and input / output interface 2004 and other components, optionally, the components can be connected and communicated through bus 2005. It should be noted that, Figure 11 The structure of the electronic device 2000 shown in the figure is merely illustrative and does not constitute a limitation on the electronic devices to which the method provided in the embodiments of the present application is applicable.

[0288] Memory 2002 can be used to store operating systems and application programs, etc. Application programs can include computer programs that implement the methods described in the embodiments of the present invention when called by processor 2001, and can also include programs for implementing other functions or services. Memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0289] The processor 2001 is connected to the memory 2002 via the bus 2005 and implements corresponding functions by calling the application program stored in the memory 2002. The processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0290] The electronic device 2000 can be connected to a network via a communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as a user terminal or a server) via the network to implement data interaction, such as sending data to or receiving data from other devices. The communication module 2003 may include a wired network interface and / or a wireless network interface, etc., that is, the communication module may include at least one of a wired communication module and a wireless communication module.

[0291] The electronic device 2000 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 2004. The electronic device 2000 itself can have a display device, and can also be connected to other external display devices through the interface 2004. Optionally, a storage device, such as a hard disk, can also be connected through the interface 2004, so that data in the electronic device 2000 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 2002. It can be understood that the input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 2004 can be a component of the electronic device 2000, or it can be an external device connected to the electronic device 2000 when needed.

[0292] Bus 2005, used to connect the various components, may include a path for transmitting information between the components. Bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Depending on their function, bus 2005 may be categorized as an address bus, a data bus, a control bus, or the like.

[0293] Optionally, for the solution provided in the embodiment of the present invention, the memory 2002 can be used to store a computer program for executing the solution of the present invention, and be run by the processor 2001. When the processor 2001 runs the computer program, the actions of the method or device provided in the embodiment of the present invention are implemented.

[0294] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0295] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0296] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0297] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0298] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A training sample processing method, characterized in that: include: Obtaining an initial neural network model to be trained and an initial training data set corresponding to the initial neural network model, wherein the initial training data set includes n sample subsets, n≥2, each sample subset includes a plurality of training samples with annotated labels, and each training sample includes at least one of text, video, image, or speech; Training the initial neural network model based on each of the sample subsets to obtain n first neural network models; For each of the sample subsets, the n-1 first neural network models corresponding to the sample subset are used to respectively identify each training sample in the sample subset to obtain n-1 first prediction labels for each training sample, wherein the n-1 first neural network models corresponding to each of the sample subsets refer to the n-1 models among the n first neural network models excluding the first neural network model corresponding to the sample subset; For each of the training samples, if a first number of labels in the n-1 first predicted labels of the training sample that are identical to the labeled label of the training sample is not less than a first set number, then the training sample is determined as a first target sample; Based on the determined first target samples, a target training data set corresponding to the initial neural network model is constructed.

2. The method according to claim 1, characterized in that The method further comprises: Taking all the determined first target samples as a first target sample set, and training the initial neural network model based on the first target sample set to obtain a second neural network model; Taking the training samples other than the first target sample in the initial training data set as pending samples, identifying each of the pending samples based on the second neural network model to obtain a second predicted label for each of the pending samples; Determining a second target sample in each of the pending samples based on the second predicted label of each of the pending samples; The step of constructing a target training data set corresponding to the initial neural network model based on the determined first target samples includes: The target training data set is constructed based on the determined first target samples and second target samples.

3. The method according to claim 2, characterized in that The training of the initial neural network model based on the first target sample set to obtain a second neural network model includes: For each sample subset among the n sample subsets, taking all pending samples in the sample subset as a first pending subset; Divide the n first undetermined subsets into m second undetermined subsets, where n≥m≥2, each second undetermined subset includes at least one first undetermined subset, and each first undetermined subset belongs to only one second undetermined subset; Merging the first target sample set with each second undetermined subset to obtain m first mixed sample sets, and training the initial neural network model based on each first mixed sample set to obtain m second neural network models; The identifying each of the pending samples based on the second neural network model to obtain a second predicted label for each of the pending samples includes: For each of the pending samples, the m-1 second neural network models corresponding to the pending sample are used to respectively identify the pending sample to obtain m-1 second prediction labels for the pending sample, wherein the m-1 second neural network models corresponding to the pending sample refer to the m-1 models of the m second neural network models excluding the first model, and the first model is the second neural network model corresponding to the first mixed sample set to which the pending sample belongs; The determining, based on the second predicted label of each of the pending samples, a second target sample in each of the pending samples, includes: Based on the m-1 second predicted labels of each of the pending samples, a second target sample in each of the pending samples is determined.

4. The method according to claim 3, characterized in that The determining of the second target sample in each of the pending samples based on the m-1 second predicted labels of each of the pending samples includes: For each of the pending samples, if a second number of labels in the m-1 second predicted labels of the pending sample that are identical to the labeled label of the pending sample is not less than a second set number, the pending sample is determined as the second target sample.

5. The method according to claim 3, characterized in that The method further comprises: For each of the first undetermined subsets, treating each undetermined sample in the undetermined subset except the second target sample as a rare sample, reacquiring a new label for each of the rare samples, and treating the rare sample with the new label as a third target sample; The step of constructing the target training data set based on the determined first target samples and second target samples includes: The target training data set is constructed based on the first target samples, the second target samples and the third target samples.

6. The method according to claim 5, characterized in that The step of re-obtaining a new annotation label for each of the rare samples comprises: Merging all the determined first target samples and all the second target samples to obtain a second target sample set; For each of the first to-be-determined subsets, taking all rare samples in the to-be-determined subset as a first rare sample set; Divide n first rare sample sets into k second rare sample sets, where n≥k≥2, each of the second rare sample sets includes at least one first rare sample set, and each of the first rare sample sets belongs to only one second rare sample set; Merging the second target sample set with each of the second rare sample sets to obtain k second mixed sample sets; Training the initial neural network model based on each of the second mixed sample sets to obtain k third neural network models; For each of the second rare sample sets, using k-1 third neural network models other than the third neural network model corresponding to the second rare sample set, respectively identifying each rare sample in the second rare sample set, and obtaining k-1 third prediction labels for each rare sample; For each of the rare samples, the predicted label with the highest proportion among the n-1 third predicted labels of the rare sample is determined as a new label for the rare sample.

7. The method according to claim 3, characterized in that The determining of the second target sample in each of the pending samples based on the m-1 second predicted labels of each of the pending samples includes: For each of the pending samples, the predicted label with the highest proportion among the m-1 second predicted labels of the pending sample is determined as the new label of the pending sample; Each of the pending samples with a new label is used as a second target sample.

8. The method according to claim 1, characterized in that Also includes: At least one sample subset of the n sample subsets is used as a reference dataset, and for each reference dataset, based on the n sample subsets, a first training set, a second training set, a third training set, and a test set corresponding to the reference dataset are constructed, wherein the first training set includes at least part of the training samples in the reference dataset and at least part of the training samples in n-1 sample subsets other than the reference dataset, the second training set is at least part of the training samples in n-1 sample subsets other than the reference dataset, the training samples in the third training set and the test set are at least part of the training samples in the reference dataset, and the training samples in the test set are different from the training samples in the third training set; For each of the reference data sets, respectively training the initial neural network model based on the three training sets corresponding to the reference data set to obtain three fourth neural network models, and performing performance evaluation on each of the fourth neural network models based on the test set and the model evaluation index to obtain an index value for each of the fourth neural network models, and determining the data quality of the initial training data set corresponding to the reference data set based on the three index values ​​corresponding to the three fourth neural network models; If the data quality corresponding to each of the reference data sets meets the preset data quality condition, the initial training data set is used as the target training data set; The training of the initial neural network model based on each of the sample subsets includes: If the data quality corresponding to each of the reference data sets does not meet the preset data quality conditions, the initial neural network model is trained separately based on each of the sample subsets.

9. The method according to any one of claims 1 to 8, characterized in that The initial neural network model is an initial classification model, and the annotated label of each training sample is the true category of the annotated training sample; The n-1 first neural network models corresponding to the sample subset are used to respectively identify each training sample in the sample subset to obtain n-1 first prediction labels for each training sample, including: The n-1 first neural network models corresponding to the sample subset are used to classify and identify each training sample in the sample subset, and n-1 first prediction labels are obtained for each training sample, wherein the first prediction label is the predicted category of the training sample.

10. A data classification method, characterized in that: The method comprises: Obtain data to be classified; Inputting the data to be classified into a target classification model to obtain a category recognition result of the data to be classified; In which, the target classification model is obtained by training the initial classification model using a target training data set with annotated labels, and the target training sample set is obtained by processing the initial training data set of the initial classification model using the method described in any one of claims 1 to 9, and the annotated label of each training sample in the target training data set represents the true category of the training sample.

11. A training sample processing device, characterized in that: include: An initial training set acquisition module is used to acquire an initial neural network model to be trained and an initial training data set corresponding to the initial neural network model, wherein the initial training data set includes n sample subsets, n ≥ 2, each sample subset includes multiple training samples with annotated labels, and each training sample includes at least one of text, video, image or speech; The sample processing module is configured to perform the following processing based on the initial training data set: Training the initial neural network model based on each of the sample subsets to obtain n first neural network models; For each of the sample subsets, the n-1 first neural network models corresponding to the sample subset are respectively used to identify each training sample in the subset to obtain n-1 first prediction labels for each training sample, wherein the n-1 first neural network models corresponding to each of the sample subsets refer to the n-1 models in the n first neural network models excluding the first neural network model corresponding to the sample subset; For each of the training samples, if a first number of labels in the n-1 first predicted labels of the training sample that are identical to the labeled label of the training sample is not less than a first set number, then the training sample is determined as a first target sample; The target training set acquisition module is used to construct a target training data set corresponding to the initial neural network model based on the determined first target samples.

12. The device according to claim 11, characterized in that The sample processing module is further configured to: Taking all the determined first target samples as a first target sample set, and training the initial neural network model based on the first target sample set to obtain a second neural network model; Taking the training samples other than the first target sample in the initial training data set as pending samples, identifying each of the pending samples based on the second neural network model to obtain a second predicted label for each of the pending samples; Determining a second target sample in each of the pending samples based on the second predicted label of each of the pending samples; When constructing a target training data set corresponding to the initial neural network model based on the determined first target samples, the target training set acquisition module is specifically used to: The target training data set is constructed based on the determined first target samples and second target samples.

13. The device according to claim 12, characterized in that When the sample processing module trains the initial neural network model based on the first target sample set to obtain the second neural network model, the sample processing module is specifically configured to: For each sample subset among the n sample subsets, taking all pending samples in the sample subset as a first pending subset; Divide the n first undetermined subsets into m second undetermined subsets, where n≥m≥2, each second undetermined subset includes at least one first undetermined subset, and each first undetermined subset belongs to only one second undetermined subset; Merging the first target sample set with each second undetermined subset to obtain m first mixed sample sets, and training the initial neural network model based on each first mixed sample set to obtain m second neural network models; When the sample processing module identifies each of the pending samples based on the second neural network model to obtain the second predicted label of each of the pending samples, the sample processing module is specifically configured to: For each of the pending samples, the m-1 second neural network models corresponding to the pending sample are used to respectively identify the pending sample to obtain m-1 second prediction labels for the pending sample, wherein the m-1 second neural network models corresponding to the pending sample refer to the m-1 models of the m second neural network models excluding the first model, and the first model is the second neural network model corresponding to the first mixed sample set to which the pending sample belongs; When determining the second target sample in each of the pending samples based on the second predicted label of each of the pending samples, the sample processing module is specifically configured to: Based on the m-1 second predicted labels of each of the pending samples, a second target sample in each of the pending samples is determined.

14. The device according to claim 13, characterized in that When determining the second target sample in each of the pending samples based on the m-1 second prediction labels of each of the pending samples, the sample processing module is specifically configured to: For each of the pending samples, if a second number of labels in the m-1 second predicted labels of the pending sample that are identical to the labeled label of the pending sample is not less than a second set number, the pending sample is determined as the second target sample.

15. The device according to claim 13, characterized in that The sample processing device is further used for: For each of the first undetermined subsets, treating each undetermined sample in the undetermined subset except the second target sample as a rare sample, reacquiring a new label for each of the rare samples, and treating the rare sample with the new label as a third target sample; When constructing the target training data set based on the determined first target samples and second target samples, the target training set acquisition module is specifically configured to: The target training data set is constructed based on the first target samples, the second target samples and the third target samples.

16. The device according to claim 15, characterized in that When re-obtaining a new annotation label for each of the rare samples, the sample processing module is specifically configured to: Merging all the determined first target samples and all the second target samples to obtain a second target sample set; For each of the first to-be-determined subsets, taking all rare samples in the to-be-determined subset as a first rare sample set; Divide n first rare sample sets into k second rare sample sets, where n≥k≥2, each of the second rare sample sets includes at least one first rare sample set, and each of the first rare sample sets belongs to only one second rare sample set; Merging the second target sample set with each of the second rare sample sets to obtain k second mixed sample sets; Training the initial neural network model based on each of the second mixed sample sets to obtain k third neural network models; For each of the second rare sample sets, using k-1 third neural network models other than the third neural network model corresponding to the second rare sample set, respectively identifying each rare sample in the second rare sample set, and obtaining k-1 third prediction labels for each rare sample; For each of the rare samples, the predicted label with the highest proportion among the n-1 third predicted labels of the rare sample is determined as a new label for the rare sample.

17. The device according to claim 13, characterized in that When determining the second target sample in each of the pending samples based on the m-1 second prediction labels of each of the pending samples, the sample processing module is specifically configured to: For each of the pending samples, the predicted label with the highest proportion among the m-1 second predicted labels of the pending sample is determined as the new label of the pending sample; Each of the pending samples with a new label is used as a second target sample.

18. The device according to claim 11, characterized in that The sample processing module is further configured to: At least one sample subset of the n sample subsets is used as a reference dataset, and for each reference dataset, based on the n sample subsets, a first training set, a second training set, a third training set, and a test set corresponding to the reference dataset are constructed, wherein the first training set includes at least part of the training samples in the reference dataset and at least part of the training samples in n-1 sample subsets other than the reference dataset, the second training set is at least part of the training samples in n-1 sample subsets other than the reference dataset, the training samples in the third training set and the test set are at least part of the training samples in the reference dataset, and the training samples in the test set are different from the training samples in the third training set; For each of the reference data sets, respectively training the initial neural network model based on the three training sets corresponding to the reference data set to obtain three fourth neural network models, and performing performance evaluation on each of the fourth neural network models based on the test set and the model evaluation index to obtain an index value for each of the fourth neural network models, and determining the data quality of the initial training data set corresponding to the reference data set based on the three index values ​​corresponding to the three fourth neural network models; If the data quality corresponding to each of the reference data sets meets the preset data quality condition, the initial training data set is used as the target training data set; When the sample processing module trains the initial neural network model based on each of the sample subsets, the sample processing module is specifically configured to: If the data quality corresponding to each of the reference data sets does not meet the preset data quality conditions, the initial neural network model is trained separately based on each of the sample subsets.

19. The device according to any one of claims 11 to 18, characterized in that The initial neural network model is an initial classification model, and the annotated label of each training sample is the true category of the annotated training sample; The sample processing module is specifically used to: The n-1 first neural network models corresponding to the sample subset are used to classify and identify each training sample in the sample subset, and n-1 first prediction labels are obtained for each training sample, wherein the first prediction label is the predicted category of the training sample.

20. A data classification device, characterized in that: The device comprises: A data acquisition module, used to acquire data to be classified; A data processing module is used to input the data to be classified into a target classification model to obtain a category recognition result of the data to be classified; In which, the target classification model is obtained by training the initial classification model using a target training data set with annotated labels, and the target training sample set is obtained by processing the initial training data set of the initial classification model using the method described in any one of claims 1 to 9, and the annotated label of each training sample in the target training data set represents the true category of the training sample.

21. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method according to any one of claims 1 to 9 or the method according to claim 10 by executing the computer program.

22. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented, or the method according to claim 10 is implemented.

23. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 or the method according to claim 10 is implemented.

Images