Sample data screening method, model training method, device and electronic equipment

Abstract

The present disclosure provides a sample data screening method, a model training method, an apparatus, and an electronic device, which relate to the field of artificial intelligence technology, and in particular to the field of deep learning and intelligent search technology in artificial intelligence. A specific implementation scheme includes: obtaining first sample data from a sample pool, where the first sample data carries first annotation information; inputting the first sample data into N information annotation models for information annotation, and outputting N second annotation information, where the N information annotation models are N different network models pre-trained for information annotation of the first sample data, where N is a positive integer; matching the first annotation information with each second annotation information; and determining the first sample data as first target sample data when the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and intelligent search technology in artificial intelligence, and specifically to a sample data screening method, model training method, device and electronic equipment. Background Art

[0002] With the development of artificial intelligence technology, the application of sample data carrying labeled information in artificial intelligence is becoming more and more extensive, and the accuracy of the above labeled information is becoming more and more concerned by people. It is usually necessary to filter out the sample data corresponding to the labeled information with lower accuracy. In the current use process, manual screening of the sample data corresponding to the above labeled information with lower accuracy is usually adopted. Summary of the Invention

[0003] The present disclosure provides a sample data screening method, a model training method, a device and an electronic device.

[0004] According to a first aspect of the present disclosure, a sample data screening method is provided, comprising:

[0005] Acquire first sample data from a sample pool, where the first sample data carries first labeling information;

[0006] Inputting the first sample data into N information annotation models for information annotation, and outputting N second annotation information, wherein the N information annotation models are N different network models pre-trained for information annotation of the first sample data, where N is a positive integer;

[0007] Matching the first annotation information with each second annotation information;

[0008] If the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold, the first sample data is determined as the first target sample data.

[0009] According to a second aspect of the present disclosure, a model training method is provided, comprising:

[0010] Acquire sample data from a sample pool, where the sample data carries first annotation information;

[0011] Input the sample data into N to-be-trained models for information annotation, and output N second annotation information, where N is a positive integer;

[0012] Matching the first annotation information with the second annotation information output by each to-be-trained model;

[0013] The model to be trained corresponding to the second annotation information that successfully matches the first annotation information is determined as the information annotation model.

[0014] According to a third aspect of the present disclosure, a sample data screening device is provided, comprising:

[0015] A first acquisition module, configured to acquire first sample data from a sample pool, where the first sample data carries first annotation information;

[0016] a first labeling module, configured to input the first sample data into N information labeling models for information labeling, and output N second labeling information, wherein the N information labeling models are N different network models pre-trained for information labeling of the first sample data, where N is a positive integer;

[0017] A first matching module, configured to match the first annotation information with each second annotation information;

[0018] The first determining module is configured to determine the first sample data as first target sample data if the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold.

[0019] According to a fourth aspect of the present disclosure, a model training device is provided, comprising:

[0020] a fifth acquisition module, configured to acquire sample data from the sample pool, wherein the sample data carries first annotation information;

[0021] a sixth labeling module, configured to input the sample data into N to-be-trained models for information labeling, and output N second labeling information, where N is a positive integer;

[0022] A second matching module, configured to match the first annotation information with the second annotation information output by each to-be-trained model;

[0023] The fifth determining module is used to determine the to-be-trained model corresponding to the second annotation information that successfully matches the first annotation information as the information annotation model.

[0024] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0025] at least one processor; and

[0026] a memory communicatively connected to at least one processor; wherein,

[0027] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods in the first aspect or the second aspect.

[0028] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute any one of the methods in the first aspect or the second aspect.

[0029] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any one of the methods in the first aspect or the second aspect when executed by a processor.

[0030] In the embodiment of the present disclosure, the first sample data can be input into N information annotation models for information annotation, and N second annotation information can be output. When the number of successful matches between the N second annotation information and the first annotation information is greater than or equal to a first threshold, the first sample data can be determined as the first target sample data, thereby completing the screening of the first sample data through the N information annotation models, and obtaining the first target sample data with lower accuracy of the first annotation information. There is no need to manually screen the first sample data, thereby improving the efficiency and accuracy of the screening of the first sample data.

[0031] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of a sample data screening method provided by an embodiment of the present disclosure;

[0033] Figure 2 is a flowchart of the model training method provided by an embodiment of the present disclosure;

[0034] Figure 3 It is a flowchart of the sample data screening method and model training method provided by the embodiment of the present disclosure;

[0035] Figure 4 This is one of the structural diagrams of the sample data screening device provided by the embodiment of the present disclosure;

[0036] Figure 5 This is the second structural diagram of the sample data screening device provided by the embodiment of the present disclosure;

[0037] Figure 6 This is the third structural diagram of the sample data screening device provided by the embodiment of the present disclosure;

[0038] Figure 7 This is the fourth structural diagram of the sample data screening device provided by the embodiment of the present disclosure;

[0039] Figure 8 This is the fifth structural diagram of the sample data screening device provided by the embodiment of the present disclosure;

[0040] Figure 9 This is the sixth structural diagram of the sample data screening device provided by the embodiment of the present disclosure;

[0041] Figure 10 This is one of the structural diagrams of the model training device provided in the embodiment of the present disclosure;

[0042] Figure 11 This is the second structural diagram of the model training device provided by the embodiment of the present disclosure;

[0043] Figure 12 is a schematic block diagram of an example electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] See also Figure 1 , Figure 1 A flow chart of a sample data screening method provided in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the sample data screening method includes the following steps:

[0046] Step S101: Acquire first sample data from a sample pool, where the first sample data carries first labeling information.

[0047] Among them, the sample pool can be used to store various sample data. The storage and sorting methods of the sample data in the sample pool are not limited here. For example, the sample data can be sorted and stored in the sample pool according to the sample attribution type. The above-mentioned sample attribution type can be determined according to at least one of the information such as the identity, gender and number of the speaker of the sample data.

[0048] The specific type of the first sample data is not limited here. Optionally, the first sample data may be voice data or video data. For example, the first sample data may be voice call recording data.

[0049] Among them, the specific type of the first annotation information is not limited here. Optionally, the first annotation information can be used to represent various parameters of the first sample data. The above parameters may include the identity, gender, age and other information of the speaker in the first sample data. The above parameters may also include information such as the scene of the first sample data.

[0050] For example: when the above parameters include the identity information of the speaker, the first annotation information can be used to indicate that the identity of the speaker of the first sample data can be an agent or a user; when the above parameters include the scene information of the first sample data, the first annotation information can be used to indicate that the scene information of the first sample data can be a dialogue scene or named entity recognition (NER) and other information.

[0051] The first annotation information may be information annotated by an electronic device, for example, the electronic device annotates the first sample data to obtain the first annotation information.

[0052] It should be noted that the method for determining the sample data in the sample pool is not limited here.

[0053] As an optional implementation, it also includes:

[0054] Obtain a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer;

[0055] Extracting H third sample data from the set of samples to be accepted, where H is a positive integer less than K;

[0056] When the labeling information of the H third sample data meets the preset conditions, the K third sample data included in the to-be-accepted sample set are saved in the sample pool.

[0057] The sample set to be accepted may be part of the sample data to be labeled, and the sample data in the sample set to be accepted may be labeled by an electronic device to obtain labeling information of the sample data in the acceptance sample set.

[0058] Among them, the preset conditions are not limited here, and the preset conditions may include: the accuracy of the labeling information of each third sample data in the H third sample data is greater than the first preset threshold, or the number of successful matches between the labeling information of the H third sample data and the actual labeling information is greater than the second preset threshold.

[0059] In the disclosed embodiment, when the labeling information of H third sample data meets the preset conditions, it indicates that the labeling information of the K third sample data included in the sample set to be accepted is likely to be correct, so that the K third sample data included in the sample set to be accepted can be saved in the sample pool. Compared with the method of accepting each third sample data included in the sample set to be accepted, this embodiment can reduce the workload and cost of acceptance and improve the acceptance efficiency of the K third sample data included in the sample set to be accepted.

[0060] It should be noted that there may be multiple sample sets to be accepted, and the data included in each sample set to be accepted may be referred to as a batch of sample data.

[0061] As an optional implementation, it also includes:

[0062] Obtain J target sample data to be labeled from the sample data to be labeled;

[0063] Label the J target sample data to be labeled;

[0064] When the number of correct labeling information of the J target sample data to be labeled is greater than or equal to a fourth threshold, the sample data to be labeled are labeled to obtain the sample set to be accepted.

[0065] It should be noted that since the sample data in the sample set to be accepted can be labeled by the electronic device, in order to verify whether the electronic device has mastered the standards for labeling the sample data, J target sample data to be labeled can be obtained from the sample data to be labeled, and then the electronic device can label the J target sample data to be labeled. When the number of correct labeling information of the J target sample data to be labeled is greater than or equal to the fourth threshold, it can be indicated that the electronic device has mastered the standards for labeling the sample data. At this time, the electronic device can label the sample data to be labeled to obtain the sample set to be accepted.

[0066] In the embodiment of the present disclosure, J target sample data to be labeled can be selected for trial labeling first. When the number of correct labeling information of the J target sample data to be labeled is greater than or equal to the fourth threshold, it can be indicated that the trial labeling has passed. At this time, the sample data to be labeled can be labeled to obtain a sample set to be accepted, thereby improving the accuracy of the labeling information of the sample data included in the sample set to be accepted.

[0067] Step S102: input the first sample data into N information annotation models for information annotation, and output N second annotation information, wherein the N information annotation models are N different network models pre-trained for information annotation of the first sample data, and N is a positive integer.

[0068] Among them, the N information annotation models can be trained using sample data that is homologous to the first sample data. The homologous data can refer to data belonging to the same speaker as the first sample data or data in the same scenario.

[0069] In addition, the training method of N information annotation models can be found in Figure 2 The training method in the illustrated embodiment will not be described in detail here.

[0070] As an optional implementation, the above method further includes:

[0071] Determining a sample category of the first sample data;

[0072] The N information labeling models are determined according to the sample attribution categories, wherein different sample attribution categories correspond to different information labeling models.

[0073] Among them, the sample attribution category can be understood as the sample attribution category corresponding to the storage position of the first sample data in the sample pool, that is, the sample data can be stored in the sample pool according to the sample attribution category, and each sample data is stored in the corresponding position of the sample attribution category.

[0074] In the embodiment of the present disclosure, different sample attribution categories correspond to different information annotation models. In this way, the corresponding N information annotation models are determined according to the sample attribution category of the first sample data, so that when the N information annotation models perform information annotation on the first sample data, the second annotation information obtained is more accurate.

[0075] Step S103: Match the first annotation information with each piece of second annotation information.

[0076] Among them, the way of matching the first annotation information with the second annotation information is not limited here. For example, the first annotation information and the second annotation information can be directly matched, and the matching degree is determined based on the overlapping content between the two. The more overlapping content between the two, the higher the matching degree.

[0077] For example, the first annotation information and the second annotation information can be normalized, that is, the first annotation information and the second annotation information can be transformed according to the same standard dimension, and the matching degree can be determined based on the error between the transformed first annotation information and the transformed second annotation information. The smaller the error between the transformed first annotation information and the transformed second annotation information, the higher the matching degree.

[0078] Step S104: If the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold, determine the first sample data as the first target sample data.

[0079] Among them, the more the N second annotation information successfully matches the first annotation information, the higher the accuracy of the first annotation information of the first sample data. Correspondingly, the fewer the N second annotation information successfully matches the first annotation information, the lower the accuracy of the first annotation information of the first sample data. Therefore, when the number of the N second annotation information successfully matches the first annotation information is less than or equal to the first threshold, the first sample data can be determined as the first target sample data with incorrect first annotation information.

[0080] It should be noted that when the number of N second annotation information that successfully matches the first annotation information is greater than the first threshold, it indicates that the precision-recall effect (i.e., accuracy and recall) of the N information annotation models is good, and there is no need to execute the step of determining the first sample data as the first target sample data, nor is there any need to re-label the first target sample data.

[0081] It should be noted that after determining the first target sample data, the first target sample data can also be subjected to target processing. The above-mentioned target processing may include one of the following processing methods: re-labeling the first target sample data, correcting the first labeling information of the first target sample data, and deleting the first target sample data.

[0082] For example, as an optional implementation, the above method further includes:

[0083] Relabeling the first target sample data;

[0084] The re-labeled first target sample data is saved in the sample pool.

[0085] The electronic device may be allowed to re-label the first target sample data, thereby reducing the workload of the electronic device in labeling the sample data and improving the accuracy of the labeling information of the sample data.

[0086] Among them, the re-labeled first target sample data can be saved in the sample pool. In this way, the re-labeled first target sample data can also be called the first sample data when it is obtained from the sample pool next time, and the first target sample data can also be used to train the model after being obtained.

[0087] In an embodiment of the present disclosure, after screening out the first target sample data with low accuracy of the first annotation information, the first target sample data can be re-annotated and the re-annotated first target sample data can be saved in the sample pool, thereby increasing the number of sample data with high accuracy of the annotation information in the sample pool.

[0088] As an optional implementation, it also includes:

[0089] Acquire second sample data from the sample pool, where the second sample data does not carry any labeling information and the sample category of the second sample data is the target category;

[0090] Inputting the second sample data into the N information annotation models for information annotation, and outputting N third annotation information;

[0091] When the N third labeling information indicates that the number of the target types is greater than or equal to the second threshold and less than or equal to the third threshold, the second sample data is determined as second target sample data.

[0092] Among them, the labeling information indicating that the third labeling information is a target category can be called a positive sample label, and the labeling information indicating that the third labeling information is a non-target category can be called a negative sample label.

[0093] Among them, the third annotation information can be used to indicate the type of the second sample data, and the target type can refer to the sample attribution type corresponding to the storage position of the second sample data in the sample pool, that is, the sample data can be stored in the sample pool according to the sample attribution type, and the second sample data is stored in the position corresponding to the target type.

[0094] It should be noted that the method for obtaining the target category can be described below. As an optional method, when obtaining the second sample data, the sample pool generates category information for indicating that the sample category of the second sample data is the target category, so that the second sample data and the category information can be obtained at the same time; as another optional method, the category information for indicating that the sample category of the second sample data is the target category can be pre-generated information, and the second sample data can carry the above-mentioned category information. In this way, the above-mentioned category information can be obtained at the same time as the second sample data is obtained.

[0095] Among them, when the number of target categories indicated by N third annotation information is greater than or equal to the second threshold and less than or equal to the third threshold, it indicates that the confidence that the sample category of the second sample data is the target category is low, that is, the second target sample data can be called an untrusted sample or untrusted sample data, indicating that the storage location of the second sample data in the sample pool may be wrong at this time.

[0096] In the embodiment of the present disclosure, when the second sample data does not carry labeled information, untrusted samples can also be screened out through N information labeling models, that is, untrusted samples can be screened out without labeling the second sample data, which reduces the workload and cost of labeling and can improve the screening efficiency and accuracy of untrusted samples.

[0097] It should be noted that, optionally, when the number of target types indicated by the N third annotation information is less than or equal to the second threshold, or greater than or equal to the third threshold, the second sample data is determined as a confident sample. In this way, the efficiency and accuracy of screening confident samples can be improved.

[0098] As an optional implementation, the above method further includes:

[0099] Relabeling the second target sample data;

[0100] The re-labeled second target sample data is saved in the sample pool.

[0101] Among them, the re-labeled second target sample data can be saved in the sample pool. In this way, the re-labeled second target sample data can also be called second sample data when it is obtained from the sample pool next time, and the second target sample data can also be used to train the model after being obtained.

[0102] In the disclosed embodiment, after the second target sample data is obtained through screening, the second target sample data can be re-labeled and the re-labeled second target sample data can be saved in the sample pool, thereby increasing the number of sample data with higher accuracy of the labeled information in the sample pool; at the same time, the second target sample data with lower confidence can be first screened and then re-labeled. Compared with the method of labeling all the second sample data, this embodiment can reduce the workload of labeling.

[0103] It should be noted that one of the first target sample data and the second target sample data may be re-labeled, or both may be re-labeled at the same time. For specific expressions, please refer to the following expressions.

[0104] As an optional implementation, the above method further includes:

[0105] relabeling at least one of the first target sample data and the second target sample data;

[0106] The re-labeled first target sample data or the re-labeled second target sample data is saved in the sample pool.

[0107] In the disclosed embodiment, the re-labeled first target sample data or second target sample data is saved in the sample pool, thereby increasing the number of sample data with higher accuracy of labeling information in the sample pool; at the same time, compared with the method of labeling all the first sample data and the second sample data, this embodiment can reduce the workload and cost of labeling.

[0108] It should be noted that since the first target sample data determined by the information annotation models corresponding to different sample attribution categories may overlap, the first target sample data determined by the information annotation models corresponding to different sample attribution categories can be merged and deduplicated, and then the merged and deduplicated first target sample data can be re-labeled.

[0109] In the embodiment of the present disclosure, through steps S101 to S104, the first sample data can be input into N information annotation models for information annotation, and N second annotation information can be output. When the number of N second annotation information that successfully matches the first annotation information is greater than or equal to a first threshold, the first sample data can be determined as the first target sample data, thereby completing the screening of the first sample data through the N information annotation models, and obtaining the first target sample data with lower accuracy of the first annotation information. There is no need to manually screen the first sample data, thereby improving the efficiency and accuracy of the screening of the first sample data.

[0110] See also Figure 2 , Figure 2 This is a flow chart of a model training method provided by an embodiment of the present disclosure. The information annotation model trained by the embodiment of the present disclosure can be understood as the information annotation model in the above embodiment, such as Figure 2 As shown, the model training method includes the following steps:

[0111] Step S201: Obtain sample data from a sample pool, where the sample data carries first annotation information.

[0112] The sample pool and the first annotation information can refer to the relevant descriptions in the above embodiment, and the sample data can refer to the relevant descriptions of the first sample data in the above embodiment, which will not be repeated here.

[0113] Step S202: input the sample data into N models to be trained for information annotation, and output N second annotation information, where N is a positive integer.

[0114] Step S203: Match the first annotation information with the second annotation information output by each model to be trained.

[0115] Step S204: Determine the to-be-trained model corresponding to the second annotation information that successfully matches the first annotation information as the information annotation model.

[0116] Among them, the way of matching the first annotation information and the second annotation information is not limited here. As an optional implementation method, the matching degree is determined according to the degree of overlap between the contents of the first annotation information and the second annotation information. As another optional implementation method, the first annotation information and the second annotation information can be normalized, and then the error between the normalized first annotation information and the normalized second annotation information is calculated. When the error is smaller, the matching degree between the first annotation information and the second annotation information is higher. When the matching degree exceeds the preset matching degree threshold, it can be said that the match is successful. Otherwise, the match fails.

[0117] It should be noted that the error between the first annotation information after normalization and the second annotation information after normalization can also be understood as a loss function. When the loss function is within the preset interval, it can be said that the first annotation information and the second annotation information are successfully matched, and the model to be trained has converged.

[0118] As an optional implementation, it also includes:

[0119] Determining the sample category of the sample data;

[0120] The N models to be trained are determined according to the sample categories, wherein different sample categories correspond to different models to be trained.

[0121] In the embodiment of the present disclosure, the N models to be trained are determined according to the sample attribution categories of the sample data, so that the labeling information of the sample data output by the models to be trained can be more accurate.

[0122] In the embodiment of the present disclosure, the information annotation model trained through steps S201 to S204 can accurately screen sample data with low annotation information accuracy, eliminating the need for manual screening of the sample data, thereby improving the efficiency and accuracy of sample data screening.

[0123] See also Figure 3 , Figure 3 A specific embodiment provided in the present disclosure is used to explain the above embodiment, see Figure 3 , including the following steps:

[0124] Step S301: Determine sample data T to be labeled, wherein the sample data T to be labeled may also be referred to as a dataset T to be labeled.

[0125] A small amount of sample data may be selected from the sample data to be labeled T for trial labeling. If the trial labeling passes, step S302 is executed.

[0126] The trial marking process is: obtaining J target sample data to be marked from the sample data to be marked, marking the J target sample data to be marked, and reviewing the marking results; when the number of correct marking information of the J target sample data to be marked is greater than or equal to the fourth threshold, it is determined that the trial marking is passed, and then step S302 is executed.

[0127] Step S302: crowd-test annotation;

[0128] The crowd-test annotation is to annotate the sample data to be annotated to obtain a sample set to be accepted, and each sample set to be accepted includes K third sample data carrying annotation information.

[0129] Each sample set to be accepted can be understood as a batch of third sample data, and the number of sample sets to be accepted can be multiple.

[0130] Step S303: perform acceptance on the sample set to be accepted;

[0131] If the labeling information of the H third sample data extracted from the to-be-accepted sample set meets the preset conditions, the acceptance is passed and step S304 is executed; otherwise, the process returns to step S302 to re-label the third sample data included in the to-be-accepted sample set.

[0132] Step S304: save the K third sample data included in the sample set to be accepted into the sample pool to form a part of the data set C;

[0133] The data in dataset C can be divided into multiple sample categories for storage. Figure 3 , the data in dataset C can be divided into five sample attribution categories, namely sample attribution category 1, sample attribution category 2, sample attribution category 3, sample attribution category 4 and sample attribution category 5, and the sample attribution categories can also be called subcategories.

[0134] Among them, each sample belonging category can be trained with N information labeling models. For example, each sample belonging category can be trained with 5 information labeling models. The training process can be: using the idea of ​​cross-validation, the sample data in the sample belonging category is randomly divided into 5 folds, and 4 folds are selected for training each time, thereby obtaining 5 models. When the number of sample belonging categories is n, the total number of models can be 5n.

[0135] Step S305: Determine whether the average precision of the N information labeling models in a certain sample category meets the standard. If so, execute step S306; otherwise, execute step S307.

[0136] Step S306: Determine that N information labeling models in the category to which the sample belongs are available, and no subsequent operation is required;

[0137] Step S307: Verify the labeling information of the data in the dataset C using the N information labeling models in the sample category, or screen the data in the dataset TC using the N information labeling models in the sample category.

[0138] Step S307 may include the following two steps:

[0139] Step S3071: Verify the labeled information of the data in the dataset C using N information labeling models, and execute steps S101 to S104 of the above embodiment, i.e., execute the step of verifying whether the first sample data is the first target sample data, thereby screening and obtaining the first target sample data;

[0140] Step S3072: Screening the data in the dataset TC using the N information labeling models in the sample category, i.e., performing a step of screening whether the second sample data is the second target sample data, thereby screening to obtain the second target sample data;

[0141] The data in TC refers to the data in the sample data to be labeled T after removing the data in the dataset C.

[0142] Step S308: relabel at least one of the first target sample data and the second target sample data obtained by screening, and save the relabeled data into the data set C, that is, into the sample pool.

[0143] In the embodiment of the present disclosure, by Figure 3 The illustrated embodiment also eliminates the need for manual screening of sample data, thereby improving the efficiency and accuracy of sample data screening.

[0144] See also Figure 4 , Figure 4 A schematic diagram of the structure of a sample data screening device provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the sample data screening device 400 includes:

[0145] A first acquisition module 401 is configured to acquire first sample data from a sample pool, where the first sample data carries first annotation information;

[0146] A first labeling module 402 is configured to input the first sample data into N information labeling models for information labeling, and output N second labeling information, wherein the N information labeling models are N different network models pre-trained for information labeling of the first sample data, where N is a positive integer;

[0147] A first matching module 403 is configured to match the first annotation information with each second annotation information;

[0148] The first determining module 404 is configured to determine the first sample data as first target sample data if the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold.

[0149] Alternatively, see Figure 5 The sample data screening device 400 further includes:

[0150] A second acquisition module 405 is configured to acquire second sample data from the sample pool, wherein the second sample data does not carry any labeling information and the sample category of the second sample data is the target category;

[0151] A second labeling module 406 is configured to input the second sample data into the N information labeling models for information labeling, and output N third labeling information;

[0152] The second determining module 407 is configured to determine the second sample data as second target sample data if the number of target types indicated by the N third annotation information is greater than or equal to a second threshold and less than or equal to a third threshold.

[0153] Alternatively, see Figure 6 The sample data screening device 400 further includes:

[0154] A third determination module 408 is configured to determine the sample category of the first sample data;

[0155] The fourth determining module 409 is configured to determine the N information labeling models according to the sample attribution category, wherein different sample attribution categories correspond to different information labeling models.

[0156] Alternatively, see Figure 7 The sample data screening device 400 further includes:

[0157] The third acquisition module 410 is configured to acquire a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer;

[0158] The extraction module 411 is configured to extract H third sample data from the set of samples to be accepted, where H is a positive integer less than K;

[0159] The first saving module 412 is configured to save the K third sample data included in the to-be-accepted sample set into the sample pool if the labeling information of the H third sample data meets a preset condition.

[0160] Alternatively, see Figure 8 The sample data screening device 400 further includes:

[0161] The fourth acquisition module 413 is used to acquire J target sample data to be labeled from the sample data to be labeled;

[0162] The third labeling module 414 is used to label the J target sample data to be labeled;

[0163] The fourth labeling module 415 is configured to label the sample data to be labeled if the number of correct labeling information of the J target sample data to be labeled is greater than or equal to a fourth threshold, so as to obtain the sample set to be accepted.

[0164] Alternatively, see Figure 9 The sample data screening device 400 further includes:

[0165] a fifth labeling module 416, configured to relabel at least one of the first target sample data and the second target sample data;

[0166] The second saving module 417 is configured to save the re-labeled first target sample data or the re-labeled second target sample data into the sample pool.

[0167] The sample data screening device 400 provided in the present disclosure can implement each process implemented in the sample data screening method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0168] See also Figure 10 , Figure 10 A schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure is shown in FIG. Figure 10 As shown, the model training device 1000 includes:

[0169] A fifth acquisition module 1001 is configured to acquire sample data from a sample pool, where the sample data carries first annotation information;

[0170] The sixth labeling module 1002 is configured to input the sample data into N models to be trained for information labeling, and output N second labeling information, where N is a positive integer;

[0171] A second matching module 1003 is configured to match the first annotation information with the second annotation information output by each to-be-trained model;

[0172] The fifth determining module 1004 is configured to determine the to-be-trained model corresponding to the second annotation information that successfully matches the first annotation information as the information annotation model.

[0173] Alternatively, see Figure 11 , the model training device 1000 further includes:

[0174] A sixth determining module 1005 is configured to determine the sample category of the sample data;

[0175] The seventh determination module 1006 is configured to determine the N models to be trained according to the sample categories, wherein different sample categories correspond to different models to be trained.

[0176] The model training device 1000 provided in the present disclosure can implement each process implemented in the embodiment of the model training method and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0177] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0178] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0179] like Figure 12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0180] Various components in device 1200 are connected to I / O interface 1205, including an input unit 1206, such as a keyboard and mouse; an output unit 1207, such as various types of displays and speakers; a storage unit 1208, such as a magnetic disk and optical disk; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 1201 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the sample data screening method or the model training method. For example, in some embodiments, the sample data screening method or the model training method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the sample data screening method or the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the sample data screening method or the model training method in any other appropriate manner (eg, by means of firmware).

[0182] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0188] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0189] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A sample data screening method, comprising: Acquire first sample data from a sample pool, where the first sample data carries first labeling information; Inputting the first sample data into N information annotation models for information annotation, and outputting N second annotation information, wherein the N information annotation models are N different network models pre-trained for information annotation of the first sample data, where N is a positive integer; Matching the first annotation information with each second annotation information; If the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold, determining the first sample data as the first target sample data; The N information annotation models are models trained using sample data that are homologous to the first sample data, where the homologous data refers to data belonging to the same speaker or the same scenario as the first sample data; Also includes: Obtain a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer; Extracting H third sample data from the set of samples to be accepted, where H is a positive integer less than K; If the labeling information of the H third sample data meets the preset conditions, the K third sample data included in the to-be-accepted sample set are saved in the sample pool; Also includes: Obtain J target sample data to be labeled from the sample data to be labeled; Label the J target sample data to be labeled; When the number of correct labeling information of the J target sample data to be labeled is greater than or equal to a fourth threshold, the sample data to be labeled are labeled to obtain the sample set to be accepted.

2. The method according to claim 1, further comprising: Acquire second sample data from the sample pool, where the second sample data does not carry any labeling information and the sample category of the second sample data is the target category; Inputting the second sample data into the N information annotation models for information annotation, and outputting N third annotation information; When the N third labeling information indicates that the number of the target types is greater than or equal to the second threshold and less than or equal to the third threshold, the second sample data is determined as second target sample data.

3. The method according to claim 1, further comprising: Determining a sample category of the first sample data; The N information labeling models are determined according to the sample attribution categories, wherein different sample attribution categories correspond to different information labeling models.

4. The method according to claim 2, further comprising: relabeling at least one of the first target sample data and the second target sample data; The re-labeled first target sample data or the re-labeled second target sample data is saved in the sample pool.

5. A model training method comprising: Acquire sample data from a sample pool, where the sample data carries first annotation information; Input the sample data into N to-be-trained models for information annotation, and output N second annotation information, where N is a positive integer; Matching the first annotation information with the second annotation information output by each to-be-trained model; Determining the to-be-trained model corresponding to the second annotation information that successfully matches the first annotation information as the information annotation model; The sample data and the first sample data are homologous data, where the homologous data refers to data belonging to the same speaker or the same scenario as the first sample data; Also includes: Obtain a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer; Extracting H third sample data from the set of samples to be accepted, where H is a positive integer less than K; If the labeling information of the H third sample data meets the preset conditions, the K third sample data included in the to-be-accepted sample set are saved in the sample pool; Also includes: Obtain a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer; Extracting H third sample data from the set of samples to be accepted, where H is a positive integer less than K; If the labeling information of the H third sample data meets the preset conditions, the K third sample data included in the to-be-accepted sample set are saved in the sample pool; Obtain J target sample data to be labeled from the sample data to be labeled; Label the J target sample data to be labeled; When the number of correct labeling information of the J target sample data to be labeled is greater than or equal to a fourth threshold, the sample data to be labeled are labeled to obtain the sample set to be accepted.

6. The method according to claim 5, further comprising: Determining the sample category of the sample data; The N models to be trained are determined according to the sample categories, wherein different sample categories correspond to different models to be trained.

7. A sample data screening device, comprising: A first acquisition module, configured to acquire first sample data from a sample pool, where the first sample data carries first annotation information; a first labeling module, configured to input the first sample data into N information labeling models for information labeling, and output N second labeling information, wherein the N information labeling models are N different network models pre-trained for information labeling of the first sample data, where N is a positive integer; A first matching module, configured to match the first annotation information with each second annotation information; a first determining module, configured to determine the first sample data as first target sample data if the number of the N second annotation information that successfully matches the first annotation information is less than or equal to a first threshold; The N information annotation models are models trained using sample data that are homologous to the first sample data, where the homologous data refers to data belonging to the same speaker or the same scenario as the first sample data; Also includes: A third acquisition module is configured to acquire a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer; An extraction module is used to extract H third sample data from the set of samples to be accepted, where H is a positive integer less than K; A first saving module is configured to save the K third sample data included in the to-be-accepted sample set into the sample pool if the labeling information of the H third sample data meets a preset condition; Also includes: The fourth acquisition module is used to obtain J target sample data to be labeled from the sample data to be labeled; A third labeling module is used to label the J target sample data to be labeled; The fourth labeling module is used to label the sample data to be labeled when the number of correct labeling information of the J target sample data to be labeled is greater than or equal to a fourth threshold, so as to obtain the sample set to be accepted.

8. The apparatus according to claim 7, further comprising: A second acquisition module is configured to acquire second sample data from the sample pool, wherein the second sample data does not carry any labeling information and the sample category of the second sample data is the target category; A second labeling module is used to input the second sample data into the N information labeling models for information labeling, and output N third labeling information; The second determining module is configured to determine the second sample data as second target sample data when the number of target types indicated by the N third annotation information is greater than or equal to a second threshold and less than or equal to a third threshold.

9. The apparatus according to claim 7, further comprising: a third determining module, configured to determine a sample category of the first sample data; The fourth determining module is configured to determine the N information labeling models according to the sample attribution category, wherein different sample attribution categories correspond to different information labeling models.

10. The apparatus according to claim 8, further comprising: a fifth labeling module, configured to relabel at least one of the first target sample data and the second target sample data; The second saving module is used to save the re-labeled first target sample data or the re-labeled second target sample data into the sample pool.

11. A model training device comprising: a fifth acquisition module, configured to acquire sample data from the sample pool, wherein the sample data carries first annotation information; a sixth labeling module, configured to input the sample data into N to-be-trained models for information labeling, and output N second labeling information, where N is a positive integer; A second matching module, configured to match the first annotation information with the second annotation information output by each to-be-trained model; A fifth determining module, configured to determine the to-be-trained model corresponding to the second annotation information that successfully matches the first annotation information as the information annotation model; The sample data and the first sample data are homologous data, where the homologous data refers to data belonging to the same speaker or the same scenario as the first sample data; The device is also used for: Obtain a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer; Extracting H third sample data from the set of samples to be accepted, where H is a positive integer less than K; If the labeling information of the H third sample data meets the preset conditions, the K third sample data included in the to-be-accepted sample set are saved in the sample pool; The device is also used for: Obtain a sample set to be accepted, wherein the sample set to be accepted includes K third sample data carrying labeled information, where K is a positive integer; Extracting H third sample data from the set of samples to be accepted, where H is a positive integer less than K; If the labeling information of the H third sample data meets the preset conditions, the K third sample data included in the to-be-accepted sample set are saved in the sample pool; Obtain J target sample data to be labeled from the sample data to be labeled; Label the J target sample data to be labeled; When the number of correct labeling information of the J target sample data to be labeled is greater than or equal to a fourth threshold, the sample data to be labeled are labeled to obtain the sample set to be accepted.

12. The apparatus according to claim 11, further comprising: A sixth determination module, configured to determine a sample category of the sample data; The seventh determination module is used to determine the N models to be trained according to the sample belonging categories, wherein different sample belonging categories correspond to different models to be trained.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 4, or to enable the at least one processor to perform the method of claim 5 or 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4, or the computer instructions are used to cause the computer to execute the method according to claim 5 or 6.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 4, or the computer program implements the method according to claim 5 or 6.

Images