Noise sample identification method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a noise sample identification method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence such as deep learning. The specific implementation scheme is as follows: obtaining a sample set used in any round of training; training a training model by using training samples in the sample set; identifying first noise samples and corresponding confidence levels from the sample set by using at least a first noise identification algorithm according to the output of the training model, and identifying second noise samples from the sample set by using multiple second noise identification algorithms respectively; selecting a first target sample from the first noise samples identified by each first noise identification algorithm according to the confidence levels; determining the second noise samples identified by each second noise identification algorithm as second target samples; and determining noise samples identified in the round of training according to the first target samples and the second target samples. Noise samples can be efficiently and accurately identified from the sample set.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, in particular to the technical field of artificial intelligence such as deep learning, and especially to a noise sample identification method and device, an electronic device, and a storage medium. BACKGROUND

[0002] At present, the application of deep learning technology is more and more extensive, and model training based on a large amount of accurately labeled data can obtain a better deep learning model. Therefore, how to efficiently and accurately mine noise samples from labeled data to obtain accurately labeled training data is of great significance for training a better deep learning model using training data. SUMMARY

[0003] The present disclosure provides a noise sample identification method and device, an electronic device, and a storage medium.

[0004] According to an aspect of the present disclosure, a noise sample identification method is provided, comprising: obtaining a sample set used in any round of training; training a training model using training samples in the sample set to obtain an output of the training model; identifying first noise samples and corresponding confidence levels from the sample set using at least a first noise identification algorithm according to the output of the training model, and identifying second noise samples from the sample set using a plurality of second noise identification algorithms respectively; selecting first target samples from the first noise samples identified by each of the first noise identification algorithms according to the confidence levels; determining the second noise samples identified by each of the second noise identification algorithms as second target samples; and determining noise samples identified in the round of training according to the first target samples and the second target samples.

[0005] According to another aspect of the present disclosure, a model training method is provided, comprising: obtaining noise samples identified from a sample set using the noise sample identification method described above; deleting the noise samples from the sample set; and training a training model using the sample set after deleting the noise samples.

[0006] According to another aspect of the present disclosure, there is provided a device for identifying noise samples, comprising: a first obtaining module configured to obtain a sample set used in an arbitrary round of training; a training module configured to train a training model using training samples in the sample set to obtain an output of the training model; an identifying module configured to identify, according to the output of the training model, first noise samples and corresponding confidence levels from the sample set using at least one first noise identifying algorithm, and identify second noise samples from the sample set using a plurality of second noise identifying algorithms respectively; a selecting module configured to select a first target sample from the first noise samples identified by each of the first noise identifying algorithms according to the confidence levels; a first determining module configured to determine the second noise samples identified by each of the second noise identifying algorithms as second target samples; and a second determining module configured to determine noise samples identified in the round of training according to the first target sample and the second target sample.

[0007] According to another aspect of the present disclosure, there is provided a device for training a model, comprising: a second obtaining module configured to obtain noise samples identified from a sample set using the method for identifying noise samples described above; a second deleting module configured to delete the noise samples from the sample set; and a second training module configured to train a training model using the sample set after the noise samples are deleted.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for identifying noise samples described above, or perform the method for training a model described above.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method for identifying noise samples described above, or perform the method for training a model described above.

[0010] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method for identifying noise samples described above, or implements the steps of the method for training a model described above.

[0011] It should be understood that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0013] Figure 1 is a flowchart of a noise sample identification method according to a first embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of a noise sample identification method according to a second embodiment of the present disclosure;

[0015] Figure 3 is an example diagram of a noise sample identification method according to a second embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of a model training method according to a third embodiment of the present disclosure;

[0017] Figure 5 is a structural diagram of a noise sample identification device according to a fourth embodiment of the present disclosure;

[0018] Figure 6 is a structural diagram of a noise sample identification device according to a fifth embodiment of the present disclosure;

[0019] Figure 7 is a structural diagram of a model training device according to a sixth embodiment of the present disclosure;

[0020] Figure 8 is a block diagram of an electronic device for implementing a noise sample identification method or a model training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, including various details in order to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0022] The present disclosure relates to the field of computer technology, particularly to the field of artificial intelligence such as deep learning.

[0023] The following briefly describes the technical field related to the scheme of the present disclosure:

[0024] AI (Artificial Intelligence), is a discipline that studies making computers simulate some thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc. several directions.

[0025] DL (Deep Learning), is to learn the internal rules and representation levels of sample data. The information obtained in these learning processes is very helpful for the interpretation of data such as text, images and sound. The ultimate goal is to enable machines to have analysis and learning capabilities like people, and to be able to recognize text, images and sound data. Deep learning is a complex machine learning algorithm, and the results achieved in speech and image recognition far exceed previous related technologies. Deep learning has achieved a lot in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as vision and thought, solves many complex pattern recognition problems, and enables artificial intelligence-related technologies to make great progress.

[0026] At present, the application of deep learning technology is becoming more and more extensive, and training a deep learning model based on a large amount of accurately labeled data can obtain a better deep learning model. Therefore, how to efficiently and accurately mine noise samples from labeled data to obtain accurately labeled training data is of great significance for training a deep learning model with better effect using training data.

[0027] The method can efficiently and accurately identify noise samples. The recall rate of the noise samples is high, and manual identification of noise samples from a large number of training samples is not required, thereby saving human resources.

[0028] The method for identifying noise samples, the device for identifying noise samples, the electronic device, the non-transitory computer-readable storage medium, and the computer program product are described below with reference to the accompanying drawings.

[0029] To clearly describe the embodiments of the present disclosure, first, the noise identification algorithm involved in the embodiments of the present disclosure is introduced.

[0030] The noise identification algorithm involved in the embodiments of the present disclosure includes a loss method, an AUM (Area Under the Margin) method, a local neighbor comparison method, a sample similarity method, and a forgetting event method. For convenience, the local neighbor comparison method is referred to as FC_LOF (Full Connect-Local Outlier Factor) and the sample similarity method is referred to as FC_SIM (Full Connect Similarity) in the embodiments of the present disclosure.

[0031] In an example embodiment, the process of identifying noise samples using the loss method can be as follows:

[0032] A model, such as a ResNet50 classification neural network model, is trained using a sample set (a set of training samples from which noise samples are to be identified). The average loss value of each training sample in the entire training process is determined. For a certain training sample x i For example, let the training sample x i The training loss at the tth epoch (one generation of training) is Loss (t) (xi ), the average loss value Loss_mean(x i ) is obtained by averaging the loss values of the training sample x i ) in the entire training process. i If the average loss value of each training sample in the entire training process is greater than a first preset threshold, it is determined that the label of the training sample is incorrect, that is, the training sample is a noise sample; if the average loss value is less than or equal to the first preset threshold, it is determined that the label of the training sample is correct, that is, the training sample is a correct sample.

[0033] wherein, epoch (one generation of training) is a hyperparameter, which refers to using all training samples in the sample set to perform one complete training on the model.

[0034] In an example embodiment, the first preset threshold for determining whether a training sample is a noise sample or a correct sample can be obtained in the following manner: a two-component Gaussian mixture model is used to model the average loss values of all training samples to obtain a Gaussian distribution density function of the average loss values of noise samples and a Gaussian distribution density function of the average loss values of correct samples; and the average loss value corresponding to the intersection of the two distribution density functions is taken as the first preset threshold.

[0035] In addition, for the loss method, the disclosure defines a method for determining the confidence level of a noise sample. The confidence level represents the reliability of the judgment result that the training sample is a noise sample.

[0036] Specifically, for each noise sample (taking training sample x i as an example), the confidence score score(x i ) of the noise sample can be determined in the following formula (1) manner:

[0037]

[0038] If the confidence score of a certain noise sample is greater than or equal to 1, the noise sample is determined to be a high-confidence noise; otherwise, the noise sample is determined to be an ordinary noise. Wherein, Loss threshold refers to the first preset threshold.

[0039] In an example embodiment, the process of identifying noise samples using the AUM method can be:

[0040] A model, such as a ResNet50 classification neural network model, is trained using a sample set. Taking a training sample x i , the class of which is y i , let z (t) (x i ) be the corresponding x i in the tth epoch training.logits vector of the sample x k t)(x i ) corresponds to the category k; determine an AUM value of each training sample, and set a second preset threshold to determine whether the AUM value of each training sample is greater than the second preset threshold, if greater, it is determined that the label of the training sample is correct, that is, the training sample is a correct sample, if less than or equal to, it is determined that the label of the training sample is incorrect, that is, the training sample is a noise sample.

[0041] Taking a training sample x i as an example, the process of determining the AUM value of each training sample is described.

[0042] First, the following formula (2) is used to determine the training sample x i corresponding "interval" m (t) (x i ,y i ), wherein the "interval" represents the possibility of the correct label of the sample.

[0043]

[0044] wherein, is the element value of the category y i in the logits vector of the sample x i , is the maximum element value of the category y i in the logits vector of the sample x i .

[0045] Since for the training sample with incorrect label, the interval in formula (2) is usually negative, and for the training sample with correct label, the interval is a positive number, in order to enhance the robustness, taking the training sample x i as an example, formula (3) can be used to average the interval in all training epochs to obtain an AUM value AUM(x i ):

[0046]

[0047] wherein, T' is the number of training epochs, m (t) (x i ,y i ) is the interval corresponding to the training sample x i .

[0048] Since the greater the AUM value of the training sample, the greater the possibility of the correct label of the training sample, therefore, the AUM value of the training sample can be used to determine whether the label of the training sample is correct.

[0049] In the example embodiment, the second preset threshold value can be obtained in the following manner: using a 2-component Gaussian mixture model to model the AUM values of all training samples, obtaining a Gaussian distribution density function of the AUM values of the noise samples and a Gaussian distribution density function of the AUM values of the correct samples; and taking the AUM value corresponding to the intersection of the two distribution density functions as the second preset threshold value.

[0050] In addition, for the AUM method, the disclosure defines a method for determining the confidence level of noise samples. Specifically, for each noise sample (take training sample x i as an example), the confidence score score(x i ) of the noise sample can be determined in the following formula (4) shown manner:

[0051]

[0052] If the confidence score of a certain noise sample is less than or equal to -1, it is determined that the noise sample is a high-confidence noise, otherwise, it is determined that the noise sample is an ordinary noise. Wherein, AUM threshold refers to the second preset threshold value.

[0053] In the example embodiment, the process of identifying noise samples using the FC_SIM method can be:

[0054] Training a model, such as a ResNet50 classification neural network model, using a sample set; determining the class similarity value of each training sample; determining whether the class similarity value of each training sample is greater than a third preset threshold value, if greater, it is determined that the label of the training sample is correct, that is, the training sample is a correct sample, if less than or equal to, it is determined that the label of the training sample is incorrect, that is, the training sample is a noise sample.

[0055] In the example embodiment, the class similarity value SIM(x i ) of each training sample (take training sample x i as an example) can be calculated in the following formula (5) shown manner:

[0056]

[0057]

[0058] Wherein, f(x i ) represents the feature of training sample x i in the FC (Full Connect, full connection) layer of the ResNet50 classification neural network model, f(x j ) represents the feature of training sample x jThe features of the FC layer of the ResNet50 classification neural network model, N k denotes the number of training samples of the class k, S = (.) represents the cosine similarity. i denotes the class of the training sample x i , y j denotes the class of the training sample x j . T(x i , y i ) denotes the average similarity of the training sample x i to all samples of the class.

[0059] The class similarity value of the training sample can be understood as: the average similarity of the training sample to all samples of the class, and the ratio of the average similarity of the sample to all samples of other classes. The larger the ratio, the greater the possibility that the label of the training sample is correct. The smaller the ratio, the greater the possibility that the label of the training sample is incorrect.

[0060] In an example embodiment, the third preset threshold can be obtained in the following manner: using a two-component Gaussian mixture model to model the class similarity values of all training samples, obtaining a Gaussian distribution density function of the class similarity values of noise samples and a Gaussian distribution density function of the class similarity values of correct samples; and taking the class similarity value corresponding to the intersection of the two distribution density functions as the third preset threshold.

[0061] In addition, for the FC_SIM method, the disclosure defines a method for determining the confidence level of noise samples. Specifically, for each noise sample (taking the training sample x i as an example), the confidence score score(x i ) of the noise sample can be determined in the following formula (6):

[0062]

[0063] If the confidence score of a certain noise sample is less than or equal to -1, the noise sample is determined to be a high-confidence noise. Otherwise, the noise sample is determined to be a normal noise. Wherein, SIM threshold denotes the third preset threshold.

[0064] In an example embodiment, the process of identifying noise samples using the FC_LOF method can be:

[0065] Train a model, such as a ResNet50 classification neural network model, using a set of samples; use the features of the FC layer of the ResNet50 classification neural network model as the input of a LOF (Local Outlier Factor) model to obtain an outlier factor value LOF of each training sample, wherein the greater the LOF value of a training sample, the greater the possibility that the label of the training sample is incorrect; determine whether the LOF value of each training sample is greater than a fourth preset threshold value, and if so, determine that the training sample is a noise sample, and if not, determine that the training sample is a correct sample.

[0066] In an example embodiment, in the same way as the first preset threshold value is determined in the loss method, the fourth preset threshold value can be obtained in the following way: use a 2-component Gaussian mixture model to model the LOF values of all training samples to obtain a Gaussian distribution density function of the LOF values of noise samples and a Gaussian distribution density function of the LOF values of correct samples; and take the LOF value at the intersection of the two distribution density functions as the fourth preset threshold value.

[0067] In addition, for the FC_LOF method, the disclosure defines a method for determining the confidence level of noise samples. Specifically, for each noise sample (take training sample x i as an example), the confidence score score(x i ) of the noise sample can be determined in the following formula (7) shown:

[0068]

[0069] If the confidence score of a certain noise sample is greater than or equal to 1, the noise sample is determined to be a high-confidence noise, otherwise, the noise sample is determined to be a normal noise. Wherein, LoF threshold refers to the fourth preset threshold value.

[0070] In an example embodiment, the process of identifying noise samples using the forgetting event method can be:

[0071] Train a model, such as a ResNet50 classification neural network model, using a set of training samples, wherein during the training process, if the label of a certain training sample is correctly predicted in the last round but incorrectly predicted in this round, it is said that the training sample has experienced a forgetting event; record the number of forgetting events of each training sample in the entire training process, and determine the training samples with a forgetting event number greater than or equal to 2 as noise samples, and determine the training samples with a forgetting event number less than 2 as correct samples. Wherein, for the forgetting event method, the disclosure does not distinguish between high-confidence noise and normal noise in the example embodiment.

[0072] The noise sample identification method provided by the present disclosure will be described in detail below. Figure 1 The noise sample identification method provided by the present disclosure will be described in detail below.

[0073] Figure 1 is a flowchart of the noise sample identification method according to the first embodiment of the present disclosure. It should be noted that the noise sample identification method provided by the present disclosure is executed by a noise sample identification device. The noise sample identification device can be an electronic device or software configured in an electronic device, etc., to efficiently and accurately identify noise samples in a sample set. The present disclosure takes the noise sample identification device configured in an electronic device as an example for description.

[0074] The electronic device can be any stationary or mobile computing device capable of data processing, such as a notebook computer, a smart phone, a wearable device, a desktop computer, a server, or other types of computing devices, etc., and the present disclosure does not limit the electronic device.

[0075] As shown in Figure 1 The noise sample identification method can include the following steps:

[0076] Step 101: Obtain a sample set used in any round of training.

[0077] In an exemplary embodiment, the process of identifying noise samples can be divided into one or more stages, and each stage inputs a sample set into a training model for a round of training. For any round of training, the sample set used in the round of training can be obtained first.

[0078] The sample set is a training sample set from which noise samples need to be identified in any round of training, and includes a plurality of training samples. When the process of identifying noise samples is divided into multiple stages, noise samples are identified from the sample set used in the corresponding round of training in each stage.

[0079] It should be noted that the round of training in the present embodiment refers to the round of training in step 101.

[0080] Step 102: Train the training model using the training samples in the sample set to obtain the output of the training model.

[0081] The training model can be any network model in the field of deep learning technology, such as a ResNet50 classification neural network model or other neural network classification model, and the present disclosure does not limit the training model.

[0082] In addition, any training method can be used to train the training model, such as deep learning. The embodiments of the present disclosure do not make any limitation in this regard.

[0083] The output of the training model can include the output data of each network layer of the training model during the training process of the training model, such as the predicted category of the training model for the training sample, the logits vector of the training sample, the feature of the training sample at the FC layer of the training model, and the like.

[0084] In step 103, at least one first noise sample and the corresponding confidence are identified from the sample set by using at least one first noise identification algorithm according to the output of the training model, and second noise samples are identified from the sample set by using multiple second noise identification algorithms.

[0085] The at least one first noise identification algorithm used in this round of training can include at least one of the noise identification algorithms described in the above embodiments, and the multiple second noise identification algorithms used in this round of training can include multiple of the noise identification algorithms described in the above embodiments. For example, the at least one first noise identification algorithm used in this round of training can include multiple of the loss method, the boundary area AUM, the local neighbor comparison FC_LOF, and the sample similarity FC_SIM, and the multiple second noise identification algorithms used in this round of training can include multiple of the loss method, the AUM, the FC_LOF, the FC_SIM, and the forgetting event. The process of identifying noise samples from the sample set by using each noise identification algorithm and determining the corresponding confidence has been described in the above embodiments, and will not be repeated here.

[0086] In step 104, a first target sample is selected from the first noise samples identified by each first noise identification algorithm according to the confidence.

[0087] It can be understood that each first noise identification algorithm will identify first noise samples from the sample set. In the embodiments of the present disclosure, for each first noise identification algorithm, the confidence of the first noise sample identified by the first noise identification algorithm can be determined by using the corresponding method for determining the confidence of the noise sample, and a target noise sample with high confidence can be selected from the first noise samples identified by the first noise identification algorithm. Then, the target noise samples identified by each first noise identification algorithm can be determined as the first target sample.

[0088] For example, assuming that the at least one first noise recognition algorithm used in the round of training includes loss method, boundary area AUM, local neighbor comparison FC_LOF and sample similarity FC_SIM, each first noise recognition algorithm identifies first noise samples from the sample set. Taking the loss method as an example, a noise sample confidence level determination method corresponding to the loss method can be used to select high-confidence noise samples from the first noise samples identified by the loss method as target noise samples. Similarly, for the AUM method, the FC_LOF method and the FC_SIM method, a corresponding noise sample confidence level determination method can be used to select high-confidence target noise samples from the first noise samples identified by each algorithm, respectively. Then, the high-confidence target noise samples selected by each noise recognition algorithm can be determined as the first target samples. That is, the first target samples are the union of the high-confidence target noise samples selected by each first noise recognition algorithm.

[0089] Step 105, determining the second noise samples identified by each second noise recognition algorithm as the second target samples.

[0090] It can be understood that each second noise recognition algorithm identifies second noise samples from the sample set, and in the embodiments of the present disclosure, the second noise samples identified by each second noise recognition algorithm can be determined as the second target samples.

[0091] For example, assuming that the multiple second noise recognition algorithms used in the round of training include loss method, boundary area AUM, local neighbor comparison FC_LOF and sample similarity FC_SIM and forgetting event, each second noise recognition algorithm identifies second noise samples from the sample set. In the embodiments of the present disclosure, the second noise samples identified by the loss method, the boundary area AUM, the local neighbor comparison FC_LOF and the sample similarity FC_SIM and the forgetting event can be determined as the second target samples. That is, the second target samples are the intersection of the second noise samples identified by each second noise recognition algorithm.

[0092] Step 106, determining the noise samples identified by the round of training according to the first target samples and the second target samples.

[0093] In the example embodiments, for the sample set used in any round of training, after the first target samples and the second target samples are determined, the first target samples and the second target samples can be determined as the noise samples identified by the round of training.

[0094] Since multiple noise recognition algorithms are integrated to identify noise samples from the sample set, it can be ensured that more noise samples are recalled with high precision. Moreover, since the noise sample identification for a large number of training samples can be replaced by artificial, human resources are saved.

[0095] To sum up, the noise sample identification method provided by the embodiments of the present disclosure can efficiently and accurately identify noise samples from the sample set with high recall rate, and does not require manual identification of noise samples from a large number of training samples, thereby saving human resources.

[0096] From the above analysis, it can be seen that in the embodiments of the present disclosure, the process of identifying noise samples can be divided into one stage or multiple stages, and each stage inputs the sample set into the training model for one round of training. The following will be further described in combination with Figure 2 the noise sample identification process when the process of identifying noise samples is divided into multiple stages in the noise sample identification method provided by the present disclosure.

[0097] Figure 2 is a flowchart of the noise sample identification method according to the second embodiment of the present disclosure. As shown in Figure 2 the noise sample identification method can include the following steps:

[0098] Step 201: obtaining a sample set used in an arbitrary round of training.

[0099] It should be noted that in this embodiment, the round of training refers to the round of training in step 201, the next round of training refers to the next round of training of the round of training in step 201, and the previous round of training refers to the previous round of training of the round of training in step 201.

[0100] Step 202: training the training model using the training samples in the sample set to obtain the output of the training model.

[0101] Step 203: according to the output of the training model, identifying first noise samples and corresponding confidence levels from the sample set using at least one first noise identification algorithm, and identifying second noise samples from the sample set using multiple second noise identification algorithms.

[0102] Step 204: selecting first target samples from the first noise samples identified by each first noise identification algorithm according to the confidence levels.

[0103] Step 205, determining the second noise samples identified by each second noise identification algorithm as second target samples.

[0104] Step 206, determining noise samples identified by the round of training according to the first target samples and the second target samples.

[0105] The specific implementation process and principles of steps 201-206 can refer to the description of the above embodiments, which will not be repeated here.

[0106] Step 207, deleting the noise samples identified by the round of training from the sample set used by the round of training to obtain an updated sample set used by the next round of training.

[0107] Step 208, performing the next round of training on the training model according to the updated sample set.

[0108] In an exemplary embodiment, for any sample set used by a round of training, after determining the noise samples identified by the round of training, the noise samples identified by the round of training can be deleted from the sample set used by the round of training to obtain an updated sample set after deleting the noise samples identified by the round of training, and the updated sample set after deleting the noise samples identified by the round of training is used as a sample set used by the next round of training. Then, the next round of training is performed on the training model according to the updated sample set, and the noise samples are identified from the updated sample set according to the process of steps 203-206.

[0109] By deleting the noise samples identified by the round of training from the noise samples in the sample set used by the round of training, the noise samples in the updated sample set used by the next round of training will be greatly reduced, and the influence of the noise samples on the noise discrimination performance of the training model during the next round of training will also be reduced. Thus, the training model can more easily identify noise samples from the sample set.

[0110] It can be understood that, since the noise samples are removed from the sample set used in each round of training after the noise samples are identified from the sample set used in the round of training, the number of noise samples in the sample set used in each round of training is different, and the number of noise samples in the corresponding sample set gradually decreases as the number of rounds of training of the trained model increases, and accordingly, the error information brought by the noise samples in the sample set used in each round of training to the training of the trained model also gradually decreases. Then, in the embodiment of the present disclosure, for any round of training, the first noise identification algorithm and the second noise identification algorithm used in the round of training can be determined according to the number of rounds of training of the trained model, so as to recall as many noise samples as possible in different rounds of training. That is, before step 203, the first noise identification algorithm and the second noise identification algorithm used in the round of training can also be determined according to the number of rounds of training of the trained model.

[0111] In the exemplary embodiment, when the number of rounds of training of the trained model is 0, that is, any round of training in step 201 is the first round of training, since the sample set used in the first round of training contains more noise samples, these noise samples will bring more error information to the training of the trained model, which will affect the noise discrimination performance of the trained model, and therefore, in the first round of training, as many noise samples as possible need to be recalled with as high accuracy as possible. In order to achieve this purpose, in the first round of training, a plurality of types of first noise identification algorithms and second noise identification algorithms can be integrated to identify noise samples.

[0112] In the exemplary embodiment, in order to recall as many noise samples as possible with as high accuracy as possible in the first round of training, the first noise identification algorithm used in the first round of training can include a plurality of loss method, boundary area AUM, local neighbor comparison FC_LOF and sample similarity FC_SIM, and the second noise identification algorithm used in the first round of training can include a plurality of loss method, AUM, FC_LOF, FC_SIM and forgetting event.

[0113] In the example embodiment, in the case that the number of rounds in which the training model has been trained is at least one, i.e. any one of the rounds of training in step 201 is not the first round of training, since the number of noise samples contained in the sample set used in this round of training is less, in order to recall more noise samples that are likely to be noise, a less number of noise identification algorithms than that used in the previous round of training can be used for noise sample identification. Specifically, in the case that the number of rounds in which the training model has been trained is at least one, the first noise identification algorithm in step 201 can be determined to be at least part of the at least one first noise identification algorithm used in the previous round of training, and the second noise identification algorithm can be determined to be at least part of the plurality of second noise identification algorithms used in the previous round of training, so as to recall more noise samples that are likely to be noise in this round of training.

[0114] In the example embodiment, in the case that the number of rounds in which the training model has been trained is at least one, in order to recall as many noise samples as possible with as high accuracy as possible, the first noise identification algorithm used in the previous round of training can include a plurality of loss method, boundary area AUM, local neighbor comparison FC_LOF and sample similarity FC_SIM, the first noise identification algorithm used in this round of training can be the same as the first noise identification algorithm used in the previous round of training, i.e. include a plurality of loss method, boundary area AUM, local neighbor comparison FC_LOF and sample similarity FC_SIM; the second noise identification algorithm used in the previous round of training can include a plurality of loss method, AUM, FC_LOF, FC_SIM and forgetting event, and the second noise identification algorithm used in this round of training can include part of the algorithms of loss method, AUM, FC_LOF, FC_SIM and forgetting event.

[0115] Among them, since loss method, AUM and FC_LOF have high recall ability for noise samples and also have high accuracy, one or more combinations of loss method, AUM and FC_LOF can be used as the second noise identification algorithm for noise sample identification in this round of training, so as to recall more noise samples that are likely to be noise with high accuracy.

[0116] The following will be combined Figure 3 The noise sample identification process when the process of identifying noise samples in the embodiments of the present disclosure is divided into two stages will be described.

[0117] As Figure 3As shown, the training process of the training model is divided into two rounds of training. In the first round of training, the sample set used is the sample set containing noise samples. The first noise identification algorithm includes: loss method, boundary area AUM, local neighborhood comparison FC_LOF, and sample similarity FC_SIM. The second noise identification algorithm includes loss method, boundary area AUM, local neighborhood comparison FC_LOF, sample similarity FC_SIM, and forgetting event.

[0118] In the first round of training, the training samples in the sample set are used to train the training model. After obtaining the output of the training model, the loss method, boundary area AUM, local neighborhood comparison FC_LOF, sample similarity FC_SIM, and forgetting event can be used to identify noise samples in the sample set according to the output of the training model. The first noise samples identified by the loss method, boundary area AUM, local neighborhood comparison FC_LOF, and sample similarity FC_SIM from the sample set are determined as the first target samples. The second noise samples identified by the loss method, boundary area AUM, local neighborhood comparison FC_LOF, sample similarity FC_SIM, and forgetting event are determined as the second target samples. The first target samples and the second target samples are the noise samples identified in the first round of training. In the first round of training, the noise samples are identified by using the integrated method of the loss method, boundary area AUM, local neighborhood comparison FC_LOF, sample similarity FC_SIM, and forgetting event, which can recall as many noise samples as possible from the sample set with the highest possible accuracy.

[0119] Further, the noise samples identified in the first round of training can be deleted from the sample set used in the first round of training to obtain an updated sample set used in the second round of training. The first noise identification algorithm used in the first round of training is determined as the first noise identification algorithm of the second round of training, that is, the first noise identification algorithm used in the second round of training includes: loss method, boundary area AUM, local neighborhood comparison FC_LOF, and sample similarity FC_SIM. Part of the second noise identification algorithm used in the first round of training is determined as the second noise identification algorithm used in the second round of training, for example, the second noise identification algorithm used in the second round of training includes: loss method, boundary area AUM, and local neighborhood comparison FC_LOF.

[0120] In the second round of training, the training samples in the updated sample set after removing the noise samples identified in the first round of training are used to train the training model. After obtaining the output of the training model, the loss method, boundary area AUM, local neighborhood comparison FC_LOF, and sample similarity FC_SIM are used to identify noise samples from the sample set, respectively. The first noise samples identified from the sample set by the loss method, boundary area AUM, local neighborhood comparison FC_LOF, and sample similarity FC_SIM are determined as the first target samples. The second noise samples identified by the loss method, boundary area AUM, and local neighborhood comparison FC_LOF are determined as the second target samples. The first target samples and the second target samples are the noise samples identified in the second round of training. In the second round of training, the noise samples are identified by using the integrated method of the loss method, boundary area AUM, local neighborhood comparison FC_LOF, and sample similarity FC_SIM. More noise samples that may be noise can be recalled from the sample set. Therefore, by using the two-stage multiple noise identification algorithm integration method to identify noise samples, as many noise samples as possible can be accurately identified from the sample set.

[0121] As can be seen from Table 1 below, compared with using a single noise identification algorithm, the recall rate, precision, and F1-score of noise samples are improved by using the above-mentioned multiple noise identification algorithm integration method to identify noise samples. Among them, F1-score is a statistical index used to measure the accuracy of a binary classification model, which takes into account both the accuracy and recall rate of the classification model. It should be noted that the data in Table 1 below is obtained by setting the noise sample ratio in a certain public data set to 10% of the total number of training samples in the sample set.

[0122] Table 1: Recall rate, precision, and F1-score of each noise identification algorithm

[0123] Recall Precision F1-score Loss method 83.33％ 76.58％ 79.81％ Boundary area AUM 82.35％ 83.17％ 82.76％ Confidence learning method 78.43％ 63.00％ 69.87％ Local neighborhood comparison FC_LOF 74.51％ 89.41％ 81.28％ Sample similarity FC_SIM 77.45％ 83.16％ 80.20％ Forget event 72.54％ 54.41％ 62.18％ Integrated method of embodiments of the present disclosure 89.22％ 92.86％ 91.00％

[0124] In summary, the noise sample identification method of the embodiments of the present disclosure, after obtaining the sample set used in any round of training, uses the training samples in the sample set to train the training model to obtain the output of the training model, uses at least one first noise identification algorithm to identify the first noise samples and the corresponding confidence from the sample set according to the output of the training model, and uses multiple second noise identification algorithms to identify the second noise samples from the sample set, respectively. According to the confidence, the first target samples are selected from the first noise samples identified by each first noise identification algorithm, and the second noise samples identified by each second noise identification algorithm are determined as the second target samples. According to the first target samples and the second target samples, the noise samples identified in the round of training are determined, and the noise samples identified in the round of training are deleted from the sample set used in the round of training to obtain the updated sample set used in the next round of training. According to the updated sample set, the next round of training of the training model is performed. The noise samples can be efficiently and accurately identified from the sample set, the recall rate of the noise samples is high, and manual identification of noise samples from a large number of training samples is not required, thereby saving human resources.

[0125] According to the embodiments of the present disclosure, a model training method is also provided.

[0126] Figure 4 is a flowchart of the model training method according to the third embodiment of the present disclosure.

[0127] It should be noted that the model training method provided by the embodiments of the present disclosure has a model training device as the execution subject. The model training device can be an electronic device or software configured in an electronic device, etc., to use a high-quality sample set to perform model training on a training model, thereby improving the effect of the trained model. The embodiments of the present disclosure take the model training device configured in an electronic device as an example for description.

[0128] The electronic device can be any stationary or mobile computing device capable of data processing, such as a notebook computer, a smart phone, a wearable device, or a stationary computing device such as a desktop computer, a server, or other types of computing devices, etc. The present disclosure does not limit this.

[0129] As shown in Figure 4 , the model training method can include the following steps:

[0130] Step 401: using a noise sample identification method to obtain noise samples identified from a sample set.

[0131] Step 402: deleting the noise samples from the sample set.

[0132] The sample set is a training sample set used for model training.

[0133] In an example embodiment, the noise sample identification method described in the above embodiment can be used to identify noise samples from the sample set, and the noise samples can be deleted from the sample set to improve the quality of the sample set.

[0134] In step 403, the model training is performed on the training model using the sample set after the noise samples are deleted.

[0135] The training model can be any network model in the field of deep learning technology, such as a ResNet50 classification neural network model or other neural network classification model, and the present disclosure does not limit the training model.

[0136] In an example embodiment, after the noise samples are deleted from the sample set, the model training is performed on the training model using the sample set after the noise samples are deleted, and the trained model is obtained. In the model training, the deep learning method can be used, and the deep learning method has better performance on big data than other training methods. The training process of the training model can refer to related technologies, and the present disclosure does not repeat them.

[0137] It should be noted that the trained model in the embodiments of the present disclosure can be applied to various scenarios, such as at least one of natural language processing, image processing, and speech recognition, and the present disclosure does not limit the trained model. Because the sample set used in the training process has deleted the noise samples identified by the noise sample identification method described in the above embodiment, the quality of the sample set is improved, and the model trained using the sample set after the noise samples are deleted has better performance, such as improving the accuracy or efficiency of natural language processing, image processing, or speech recognition.

[0138] In summary, the noise sample identification method provided by the embodiments of the present disclosure uses the noise sample identification method to obtain the noise samples identified from the sample set, deletes the noise samples from the sample set, and then uses the sample set after the noise samples are deleted to train the training model, thereby training the training model based on the high-quality sample set, obtaining the high-quality trained model, and improving the effect of the trained model.

[0139] The noise sample identification method provided by the embodiments of the present disclosure will be described below. Figure 5 The noise sample identification device provided by the present disclosure will be described.

[0140] Figure 5 FIG. 4 is a structural schematic diagram of a noise sample identification device according to the fourth embodiment of the present disclosure.

[0141] As shown in FIG. 4, the noise sample identification device includes a noise sample identification unit 401 and a model training unit 402. Figure 5As shown, the noise sample identification device 500 provided by the present disclosure includes a first acquisition module 501, a training module 502, an identification module 503, a selection module 504, a first determination module 505, and a second determination module 506.

[0142] The first acquisition module 501 is configured to acquire a sample set used in any one round of training.

[0143] The training module 502 is configured to train the training model by using the training samples in the sample set, so as to obtain the output of the training model.

[0144] The identification module 503 is configured to identify the first noise samples and the corresponding confidence from the sample set by using at least one first noise identification algorithm according to the output of the training model, and identify the second noise samples from the sample set by using a plurality of second noise identification algorithms.

[0145] The selection module 504 is configured to select the first target samples from the first noise samples identified by each first noise identification algorithm according to the confidence.

[0146] The first determination module 505 is configured to determine the second noise samples identified by each second noise identification algorithm as the second target samples.

[0147] The second determination module 506 is configured to determine the noise samples identified in the round of training according to the first target samples and the second target samples.

[0148] It should be noted that the noise sample identification device provided by the present embodiment can execute the noise sample identification method of the foregoing embodiments. The noise sample identification device can be an electronic device or software configured in an electronic device, so as to efficiently and accurately identify the noise samples in the sample set.

[0149] The electronic device can be any stationary or mobile computing device capable of data processing, such as a notebook computer, a smart phone, a wearable device, a desktop computer, a server, or other types of computing devices, without limitation.

[0150] It should be noted that the foregoing description of the embodiments of the noise sample identification method also applies to the noise sample identification device provided by the present disclosure, which will not be described herein again.

[0151] The noise sample identification device provided in this embodiment first obtains an arbitrary sample set used in a training round, then trains a training model using training samples from the sample set to obtain the output of the training model. Based on the output of the training model, at least one first noise identification algorithm is used to identify a first noise sample and its corresponding confidence level from the sample set, and multiple second noise identification algorithms are used to identify second noise samples from the sample set respectively. Then, based on the confidence level, a first target sample is selected from the first noise samples identified by each first noise identification algorithm, and the second noise samples identified by each second noise identification algorithm are determined as the second target samples. Finally, based on the first target sample and the second target sample, the noise samples identified in this training round are determined. This device can efficiently and accurately identify noise samples from the sample set, and has a high recall rate for noise samples. Furthermore, it eliminates the need for manual noise sample identification of a large number of training samples, saving human resources.

[0152] The following is combined with Figure 6 The noise sample identification device provided in this disclosure will be described.

[0153] Figure 6 This is a schematic diagram of the structure of a noise sample identification device according to the fifth embodiment of the present disclosure.

[0154] like Figure 6 As shown, the noise sample identification device 600 may specifically include: a first acquisition module 601, a training module 602, an identification module 603, a selection module 604, a first determination module 605, and a second determination module 606. Figure 6 The first acquisition module 601, training module 602, recognition module 603, selection module 604, first determination module 605, and second determination module 606 are connected to... Figure 5 The first acquisition module 501, training module 502, recognition module 503, selection module 504, first determination module 505 and second determination module 506 have the same function and structure.

[0155] In an exemplary embodiment, the noise sample identification device 600 further includes:

[0156] The third determining module 607 is used to determine the first noise recognition algorithm and the second noise recognition algorithm to be used in the training round based on the number of rounds the training model has been trained.

[0157] In an exemplary embodiment, the noise sample identification device 600 further includes:

[0158] The first deletion module 608 is used to delete the noisy samples identified in the current training round from the sample set used in the current training round, so as to obtain the updated sample set used in the next training round.

[0159] The first training module 609 is configured to perform next round training on the training model according to the updated sample set.

[0160] In an example embodiment, the third determining module 607 comprises:

[0161] The determining unit is configured to, in the case that the number of trained rounds of the training model is at least one round, determine that the first noise recognition algorithm in the round is at least part of the at least one first noise recognition algorithm used in the previous round, and the second noise recognition algorithm is at least part of the plurality of second noise recognition algorithms used in the previous round.

[0162] In an example embodiment, the first noise recognition algorithm used in the previous round and the first noise recognition algorithm used in the round both comprise a plurality of loss method, boundary area AUM, local neighbor comparison FC_LOF and sample similarity FC_SIM.

[0163] In an example embodiment, the second noise recognition algorithm used in the previous round comprises a plurality of loss method, AUM, FC_LOF, FC_SIM and forgetting event.

[0164] The second noise recognition algorithm used in the round comprises one or more combinations of loss method, AUM and FC_LOF.

[0165] It should be noted that the foregoing description of the embodiments of the method for identifying noise samples also applies to the noise sample identification device provided by the present disclosure, which will not be described here.

[0166] The noise sample identification device provided by the embodiments of the present disclosure can efficiently and accurately identify noise samples from the sample set, has a high recall rate of noise samples, and does not require manual identification of noise samples from a large number of training samples, thereby saving human resources.

[0167] According to the embodiments of the present disclosure, a model training device is also provided.

[0168] ​The following describes the model training device provided by the present disclosure. Figure 7 The model training device provided by the present disclosure is described below.

[0169] Figure 7 FIG. 7 is a structural schematic diagram of a model training device according to a sixth embodiment of the present disclosure.

[0170] As shown in FIG. 7, the model training device 700 provided by the present disclosure includes a second acquisition module 701, a second deletion module 702, and a second training module 703. Figure 7

[0171] The second acquisition module 701 is configured to acquire noise samples identified from the sample set by using the identification method of noise samples.

[0172] The second deletion module 702 is configured to delete the noise samples from the sample set.

[0173] The second training module 703 is configured to perform model training on the training model by using the sample set after the noise samples are deleted.

[0174] It should be noted that the model training device 700 provided by the present embodiment can perform the model training method of the foregoing embodiments. The model training device can be an electronic device or software configured in an electronic device, so as to perform model training on the training model by using a high-quality sample set, thereby improving the effect of the trained model.

[0175] The electronic device can be any stationary or mobile computing device capable of data processing, such as a notebook computer, a smart phone, a wearable device, or a stationary computing device such as a desktop computer, a server, or other types of computing devices, without limitation.

[0176] It should be noted that the foregoing description of the embodiments of the model training method also applies to the model training device provided by the present disclosure, which will not be described herein.

[0177] The model training device provided by the present embodiment performs model training on the training model by using the identification method of noise samples, acquiring noise samples identified from the sample set, deleting the noise samples from the sample set, and then performing model training on the training model by using the sample set after the noise samples are deleted. Thus, the model training device provided by the present embodiment can perform model training on the training model based on a high-quality sample set, obtain a high-quality trained model, and improve the effect of the trained model.

[0178] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0179] Figure 8 ​A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0180] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0181] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0182] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the noise sample identification method or the model training method. For example, in some embodiments, the noise sample identification method or the model training method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the noise sample identification method or the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the noise sample identification method or the model training method by any other suitable means, such as by means of firmware.

[0183] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0184] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0185] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a linearly-programmed electrical connection, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0187] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0188] The computer system can include a client and a server. The client is a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server, or simply VPS). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0189] It should be understood that the steps shown above can be reordered, added, or deleted using various forms of flow. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0190] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for identifying noise samples, comprising: Obtain the sample set used in any round of training; The training model is trained using training samples from the sample set to obtain the output of the training model, wherein the training samples include text, image, or sound data; Based on the output of the training model, at least one first noise identification algorithm is used to identify a first noise sample and its corresponding confidence level from the sample set, and multiple second noise identification algorithms are used to identify second noise samples from the sample set respectively, wherein the multiple second noise identification algorithms include multiple different noise identification algorithms; Based on the confidence level, a first target sample is selected from the first noise samples identified by each of the first noise identification algorithms. The first target sample is the union of the high-confidence target noise samples selected by each of the first noise identification algorithms. The second noise samples identified by each of the second noise recognition algorithms are determined as the second target samples. The second target samples are the intersection of the second noise samples identified by each of the second noise recognition algorithms. Based on the first target sample and the second target sample, determine the noise samples identified in this round of training; Before identifying a first noise sample and its corresponding confidence level from the sample set using at least one first noise recognition algorithm based on the output of the training model, and identifying second noise samples from the sample set using multiple second noise recognition algorithms respectively, the method further includes: If the training model has been trained for at least one round, the first noise identification algorithm in this round of training is determined to be at least a portion of the at least one first noise identification algorithm used in the previous round of training, and the second noise identification algorithm is at least a portion of the multiple second noise identification algorithms used in the previous round of training.

2. The identification method according to claim 1, wherein, After determining the noise samples identified in this round of training based on the first target sample and the second target sample, the method further includes: Remove the noisy samples identified in the training round from the sample set used in the training round to obtain the updated sample set used in the next training round; Based on the updated sample set, the training model is trained for the next round.

3. The identification method according to claim 1, wherein, The first noise recognition algorithm used in the previous training round and the first noise recognition algorithm used in this training round both include multiple of the following: loss method, boundary area AUM, local nearest neighbor comparison FC_LOF and sample similarity FC_SIM.

4. The identification method according to claim 1, wherein, The second noise recognition algorithm used in the previous round of training includes: loss method, AUM, FC_LOF, FC_SIM and multiple of the forgotten events; The second noise identification algorithm used in this round of training includes one or more combinations of loss method, AUM and FC_LOF.

5. A model training method, comprising: Using the method described in any one of claims 1-4, noise samples identified from the sample set are obtained; Remove the noisy sample from the sample set; The training model is trained using the sample set after deleting the noise samples.

6. A noise sample identification device, comprising: The first acquisition module is used to acquire the sample set used in any round of training; The training module is used to train the training model using training samples from the sample set to obtain the output of the training model, wherein the training samples include text, image or sound data; The identification module is configured to identify a first noise sample and its corresponding confidence level from the sample set using at least one first noise identification algorithm based on the output of the training model, and to identify a second noise sample from the sample set using multiple second noise identification algorithms, wherein the multiple second noise identification algorithms include multiple different noise identification algorithms; The selection module is used to select a first target sample from the first noise samples identified by each of the first noise recognition algorithms based on the confidence level. The first target sample is the union of the high-confidence target noise samples selected by each of the first noise recognition algorithms. The first determining module is used to determine the second noise samples identified by each of the second noise recognition algorithms as the second target samples, wherein the second target samples are the intersection of the second noise samples identified by each of the second noise recognition algorithms. The second determining module is used to determine the noise samples identified in this round of training based on the first target sample and the second target sample; Also includes: The third determining module is used to determine the first noise recognition algorithm and the second noise recognition algorithm to be used in the training round based on the number of rounds the training model has been trained. The third determining module includes: The determining unit is configured to, when the training model has been trained for at least one round, determine that the first noise identification algorithm in the current training round is at least a subset of at least one first noise identification algorithm used in the previous training round, and the second noise identification algorithm is at least a subset of multiple second noise identification algorithms used in the previous training round.

7. The identification device according to claim 6, further comprising: The first deletion module is used to delete the noisy samples identified in the current training round from the sample set used in the current training round, so as to obtain the updated sample set used in the next training round. The first training module is used to perform the next round of training on the training model based on the updated sample set.

8. The identification device according to claim 6, wherein, The first noise recognition algorithm used in the previous training round and the first noise recognition algorithm used in this training round both include multiple of the following: loss method, boundary area AUM, local nearest neighbor comparison FC_LOF and sample similarity FC_SIM.

9. The identification device according to claim 6, wherein, The second noise recognition algorithm used in the previous round of training includes: loss method, AUM, FC_LOF, FC_SIM and multiple of the forgotten events; The second noise identification algorithm used in this round of training includes one or more combinations of loss method, AUM and FC_LOF.

10. A model training device, comprising: The second acquisition module is used to acquire noise samples identified from the sample set using the method described in any one of claims 1-4; The second deletion module is used to delete the noise samples from the sample set; The second training module is used to train the training model using the sample set after deleting the noise samples.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-4, or to perform the method of claim 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4, or to perform the method according to claim 5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-4, or when executed, implements the steps of the method according to claim 5.

Images