Optimization methods, apparatus, storage media and electronic devices for data annotation

Abstract

This disclosure relates to the field of computer technology, specifically to a method, apparatus, storage medium, and electronic device for optimizing data annotation. The method for optimizing data annotation includes: acquiring a first sample set and a first annotation set, wherein the first sample set includes multiple original samples, and the first annotation set includes multiple annotations corresponding to the original samples; inputting the first sample set into a neural network model, and training the neural network model using the first annotation set as supervision; acquiring loss data of the original samples, and determining a first problem sample based on the loss data of the original samples; determining feedback data based on the first problem sample, and sending the feedback data to a client; receiving an update instruction from the client, and updating the annotations corresponding to the multiple original samples according to the update instruction.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically to methods, apparatus, storage media, and electronic devices for optimizing data annotation. Background Technology

[0002] With the development of deep learning, artificial intelligence (AI) is shining brightly in more and more fields, such as image and speech recognition, and autonomous driving. However, due to the data-driven nature of AI, how to fully utilize data for AI training is a crucial issue. A common AI training method is supervised learning. Supervised learning means that all data is manually labeled, and the AI's optimal learning outcome is achieved when its output matches the manually labeled data.

[0003] Supervised learning requires all data to be manually labeled, which brings two problems: First, if the amount of data is large, manual labeling takes a long time; second, if errors occur in manual labeling, it is not easy to find them. Picking out the erroneous ones from tens of thousands of data is a time-consuming and unproductive task. On the other hand, since artificial intelligence is "confident" in manual labeling during training, labeling errors can easily have a negative impact and affect its performance. Summary of the Invention

[0004] This disclosure provides a data annotation optimization method, apparatus, storage medium, and electronic device, which can solve the problem that errors in manual annotation are difficult to detect.

[0005] In a first aspect, embodiments of this application provide a data annotation optimization method, comprising: acquiring a first sample set and a first annotation set, wherein the first sample set includes multiple original samples and the first annotation set includes multiple annotations corresponding to the original samples; inputting the first sample set into a neural network model, and training the neural network model using the first annotation set as supervision; acquiring loss data of the original samples, and determining a first problem sample based on the loss data of the original samples; determining feedback data based on the first problem sample, and sending the feedback data to a client; receiving an update instruction from the client, and updating the annotations corresponding to the multiple original samples according to the update instruction.

[0006] Optionally, obtaining the loss data of the original samples and determining the first problem sample based on the loss data of the original samples includes: setting the number of training rounds 'a' of the first sample set in the neural network model, and the loss data of the original samples including 'a' loss values; setting a truncation parameter and obtaining a data truncation range from the truncation parameter and 'a'; determining the expected value and variance of the loss values ​​of the original samples based on the 'a' loss values; determining whether the original samples are deviation samples based on the expected value and variance of the loss values; normalizing the 'a' loss values ​​of the deviation samples and obtaining the average value of the data contained in the data truncation range of each deviation sample after normalization; and determining the deviation sample corresponding to the average value as the first problem sample if the average value is greater than a second preset threshold.

[0007] Optionally, determining whether the original sample is a deviation sample based on the expected value and variance of the loss value of the original sample includes: determining whether the a-th loss value in the loss value of the original sample is greater than the sum of the expected value of the original sample and a first product, where the first product is the product of a preset parameter and the variance of the original sample; if the a-th loss value in the loss value of the original sample is greater than the sum of the expected value of the original sample and the first product, the original sample is determined to be the deviation sample.

[0008] Optionally, determining feedback data based on the first problem sample includes: removing the first problem sample from the first sample set to obtain a second sample set, and removing the labels corresponding to the first problem sample from the first label set to obtain a second label set; inputting the second sample set into the neural network model, and training the neural network model using the second label set as supervision; inputting the first problem sample into the neural network model to obtain loss data of the first problem sample; when the loss value in the loss data of the first problem sample is greater than a third preset threshold, determining the first problem sample corresponding to the loss value as the second problem sample; and determining the first problem sample, the label corresponding to the first problem sample, the second problem sample, and the label corresponding to the first problem sample as feedback data.

[0009] Optionally, after determining the first problem sample corresponding to the loss value being greater than a third preset threshold as the second problem sample, the method includes: inputting the second sample set and the first problem sample into a neural network model, training the neural network model using the second annotation set and the annotations corresponding to the first problem sample as supervision, and obtaining loss data of the second sample set and the first problem sample; when the loss value in the loss data of the second sample set and the first problem sample is greater than the third preset threshold, determining the sample corresponding to the loss value as the second problem sample.

[0010] Optionally, after inputting the first sample set into the neural network model and training the neural network model using the first labeled set as supervision, the method further includes: stopping the training of the model when the average loss of the plurality of original samples decreases less than a first preset threshold.

[0011] Optionally, the method further includes: determining the training results of the first problem sample and the second problem sample in the neural network model as feedback data.

[0012] Secondly, embodiments of this application provide a data annotation optimization apparatus, including an acquisition module for acquiring a first sample set and a first annotation set, wherein the first sample set includes multiple original samples and the first annotation set includes multiple annotations corresponding to the original samples; a training module for inputting the first sample set into a neural network model and training the neural network model using the first annotation set as supervision; a determination module for acquiring loss data of the original samples and determining a first problem sample based on the loss data of the original samples; a feedback module for determining feedback data based on the first problem sample and sending the feedback data to a client; and a receiving module for receiving an update instruction from the client and updating the annotations corresponding to the multiple original samples according to the update instruction.

[0013] Thirdly, embodiments of this application provide a storage medium storing computer instructions thereon, which, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0014] Fourthly, embodiments of this application provide an electronic device having a processor and a memory, wherein the memory stores computer instructions, and when executed by the processor, the computer instructions implement the steps of the method described in any of the first aspects above.

[0015] One beneficial effect of this disclosure is that by inputting the original samples and their corresponding annotations into a neural network model for training, and based on the loss data during training, it is determined whether the original sample is a first problem sample. Based on the first problem sample, feedback data is determined and sent to the client. Then, update instructions from the client can be received to update the annotations corresponding to multiple original samples. In this way, samples with data annotation problems in the original samples can be automatically identified and promptly reported, improving the accuracy of identifying data annotation problems. Furthermore, data from potentially problematic samples can be fed back to confirm whether there are annotation errors and update the annotations accordingly.

[0016] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the embodiments of the present disclosure.

[0018] Figure 1 A flowchart illustrating an embodiment of the data annotation optimization method of this disclosure is shown.

[0019] Figure 2 A block diagram of a data annotation optimization apparatus according to an embodiment of the present disclosure is shown. Detailed Implementation

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0021] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0022] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0023] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0025] This application discloses an optimization method for data annotation, such as... Figure 1 As shown, the method includes steps S11-S15.

[0026] Step S11: Obtain the first sample set and the first annotation set. The first sample set includes multiple original samples, and the first annotation set includes multiple annotations corresponding to the original samples.

[0027] In one example of this embodiment, the original samples can be the original data samples used to train the neural network model, and each original sample has a corresponding manual annotation. The manual annotation can be performed by data annotation personnel.

[0028] Step S12: Input the first sample set into the neural network model and train the neural network model using the first labeled set as supervision;

[0029] In one example of this embodiment, the neural network model can be any neural network model. The first sample set, including all the original data, is input into the neural network model for training, and the first labeled set is used as supervision. That is, the neural network model is considered optimal when its output on the original data is consistent with the manual label. In this way, the neural network model is trained.

[0030] In one example of this embodiment, after inputting the first sample set into the neural network model and training the neural network model using the first labeled set as supervision, the method further includes: stopping the training of the model when the average loss of multiple original samples decreases less than a first preset threshold.

[0031] In one example of this embodiment, when each original sample is input into the neural network model, the model outputs a result. The loss value for each original sample is the difference between the output of the neural network model and the corresponding label of that original sample. Each time the neural network model performs a training round, the original samples used for training acquire a loss value. The average loss of multiple original samples is the average loss value of multiple original samples in the first sample set in the current round. When the decrease in the average loss value of multiple original samples in the current round compared to the average loss value in the previous round is less than a threshold, the neural network model can be considered to have been basically trained, and training of the neural network model can be stopped.

[0032] Step S13: Obtain the loss data of the original sample, and determine whether the original sample is the first problem sample based on the loss data of the original sample.

[0033] In one example of this embodiment, obtaining the loss data of the original sample and determining whether the original sample is a first problem sample based on the loss data of the original sample includes: setting the number of training rounds 'a' of the first sample set in the neural network model, and the loss data of the original sample includes 'a' loss values; setting a truncation parameter and obtaining the data truncation range from the truncation parameter and 'a'; determining the expected value and variance of the loss value of the original sample based on the 'a' loss values; determining whether the original sample is a deviation sample based on the expected value and variance of the loss value of the original sample; normalizing the 'a' loss values ​​of the deviation sample; obtaining the average value of the data contained in the data truncation range of each deviation sample after normalization; and determining the deviation sample corresponding to the average value as the first problem sample if the average value is greater than a second preset threshold.

[0034] In one example of this embodiment, the loss data of the original sample is the loss value of the original sample in all training rounds in the neural network model. Specifically, when the original samples in the first sample set are trained in the neural network model for 'a' rounds, the loss data of each original sample includes 'a' loss values.

[0035] In one example of this embodiment, after obtaining the loss value of each original sample in the first sample set, the expected value and variance of the loss value of any given original sample can be calculated based on the 'a' loss values. Furthermore, based on the expected value and variance of the loss value of the original sample, it can be determined whether the original sample is a deviating sample.

[0036] In one example of this embodiment, determining whether the original sample is a deviation sample based on the expected value and variance of the loss value of the original sample includes: determining whether the a-th loss value in the loss value of the original sample is greater than the sum of the expected value of the original sample and a first product, where the first product is the product of a preset parameter and the variance of the original sample; if the a-th loss value in the loss value of the original sample is greater than the sum of the expected value of the original sample and the first product, the original sample is determined to be a deviation sample.

[0037] For example, for the original sample i, the expected value E(loss) of a loss values ​​obtained from the original sample i. i ) and variance D (loss i After that, the loss value of the original sample during the last round of training can be used as the basis for the calculation. i That is, the a-th loss value, which determines whether the original sample i is a deviation sample. According to the formula:

[0038] loss i >E(loss i )+b*D(loss i )

[0039] Where b is a preset parameter, and the first product is the product of the preset parameter b and the variance of the loss value of the original sample. The specific value of b can be set according to actual needs. In one example, the value range of b can be set between [0,5]. When the a-th loss value of the original sample is greater than the sum of the expected loss value of the original sample and the first product, the original sample i can be identified as a deviation sample.

[0040] After identifying the off-target samples, the loss values ​​of those samples can be normalized. For example, if the off-target sample was trained for 10 epochs, resulting in 10 loss values ​​ordered by epoch [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], the normalized loss values ​​would be [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]. After normalizing the loss values ​​of the off-target samples, the average value of the data within the normalized data cutoff range for each off-target sample is obtained.

[0041] In one example of this embodiment, the truncation parameter can be set to 0.9. Based on the truncation parameter and the number of training epochs 'a', the data truncation range can be determined. Specifically, for example, the data truncation range can be from the 0.9ath loss value to the ath loss value, i.e., 0.9a. In this example, it is the average of the 9th loss value and the 10th normalized loss value.

[0042] After determining the average value of the data within the data cutoff range of the deviation sample, we can determine whether the deviation sample is a first-problem sample based on this average value. A first-problem sample is a sample in the original sample that may have labeling issues. Specifically, for example, we can determine whether the deviation sample is a first-problem sample by checking if the average value of the data within the data cutoff range is greater than a second preset threshold. If the average value of the loss within the data cutoff range is greater than the second preset threshold, it indicates that the training result of this sample differs significantly from the labeled result, and this deviation sample is a sample that may have data labeling issues.

[0043] In one example of this embodiment, the second preset threshold can be selected between [0.1, 0.5] according to actual needs.

[0044] Step S14: Determine the feedback data based on the first problem sample and send the feedback data to the client.

[0045] In one example of this embodiment, determining feedback data based on a first problem sample includes: removing the first problem sample from a first sample set to obtain a second sample set, removing the labels corresponding to the first problem sample from a first label set to obtain a second label set, inputting the second sample set into a neural network model, training the neural network model using the second label set as supervision, inputting the first problem sample into the neural network model to obtain loss data for the first problem sample, and when the loss value in the loss data of the first problem sample is greater than a third preset threshold, determining the first problem sample corresponding to the loss value as the second problem sample, and determining the first problem sample, the label corresponding to the first problem sample, the second problem sample, and the label corresponding to the first problem sample as feedback data.

[0046] In one example of this embodiment, after determining the first problematic sample, it can be removed from the first sample set, and the remaining sample set without problems can be used as the second sample set. Correspondingly, the annotations corresponding to the first problematic sample can be removed from the first annotation set, and the annotations of the remaining samples can be used as the second annotation set.

[0047] In one example of this embodiment, a second sample set can be input into the neural network model and trained again to obtain a more accurate neural network model.

[0048] After training to obtain a more accurate neural network model, the first problem sample can be input into the model for prediction. Based on the prediction result of the neural network model and the corresponding label of the first problem sample, the loss data for each first problem sample is determined. Each loss data includes multiple loss values, which are the differences between the prediction result of the neural network model and the corresponding label. The number of loss values ​​is the same as the number of prediction rounds for that sample in the neural network model.

[0049] When the loss value in the loss data of the first problematic sample exceeds the third preset threshold, it proves that the loss value of the first problematic sample has increased significantly, that is, the sample has deviated during the prediction process. Therefore, this sample is identified as the second problematic sample, which has a higher probability of having a labeling problem compared to the first problematic sample.

[0050] After determining the second problem sample, the first problem sample, the corresponding annotation of the first problem sample, the second problem sample, and the corresponding annotation of the second problem sample can be identified as feedback data and fed back to the client.

[0051] In this example, by further training the neural network model and predicting the output of the first problem sample based on the trained model, and by observing whether the loss value of the first problem sample shows a deviation, samples with labeling problems can be further identified from the first problem sample, thus obtaining more accurate results.

[0052] In one example of this embodiment, after determining the first problem sample whose loss value is greater than a third preset threshold as the second problem sample, the method includes: inputting the second sample set and the first problem sample into the neural network, training the neural network model with the second label set and the labels corresponding to the first problem sample as supervision, obtaining the loss data of the second sample set and the first problem sample, and determining the sample corresponding to the loss value as the second problem sample when the loss value in the loss data of the second sample set and the first problem sample is greater than the third preset threshold.

[0053] In one example of this embodiment, a second sample set and a first problem sample are input into a neural network. The neural network model is trained using the second annotation set and the annotations corresponding to the first problem sample as supervision to obtain loss data for the second sample set and the first problem sample. This includes: inputting the first problem sample into the neural network model, training the neural network model using the annotations corresponding to the first problem sample as supervision, obtaining the gradient value of each parameter of the neural network model, selecting some parameters based on the magnitude of the gradient value, and then updating the some parameters; inputting the second sample set and the first problem sample into the neural network, and training the neural network model using the second annotation set and the annotations corresponding to the first problem sample as supervision.

[0054] In one example of this embodiment, the neural network model is trained using a first problem sample and its corresponding annotation, including:

[0055] In one example of this embodiment, the gradient value of each parameter of the artificial intelligence can be obtained by performing error calculation on the first problem sample.

[0056] Error calculation is one of the core aspects of deep learning. Specifically, error calculation can be achieved through the following functions: the loss function, defined on a single sample, calculates the error of that sample; the cost function, defined on the entire training set, is the average error of all samples, i.e., the average of the loss functions; the objective function, the final optimization function, equals the empirical risk plus the structural risk; and the risk function is the expected value of the loss function. In error calculation, it is crucial to prevent overfitting. Overfitting refers to excessively fitting the training set, leading to poor prediction results. One way to address overfitting is through regularization, a rule that prevents overfitting by reducing the complexity of the model.

[0057] The purpose of error calculation is to obtain the gradient values ​​of each parameter in the new sample corresponding to the parameters in the target model. Specifically, this error calculation can be achieved through algorithms such as the Sobel operator, Laplacian operator, and Canny edge detection. Of course, error calculation is not limited to these algorithms. After obtaining the gradient values, some parameters that have a significant impact on the correct detection of new samples can be adjusted based on the magnitude of the gradient values. When the model's performance meets the requirements, updating the model's parameters can be stopped, and training can end.

[0058] After updating the parameters of the neural network model, the second sample set and the first problem sample can be input into the neural network model for simple training, and the loss value of the corresponding sample can be obtained. If the loss value of a sample is greater than the third preset threshold, it proves that the loss value of the sample has increased significantly, and then the sample is identified as the second problem sample.

[0059] In this example, after training the neural network model multiple times to obtain a better-performing neural network model, we further observe the loss values ​​of samples in the second dataset and the first problem sample. Based on the loss values, we screen samples that may have labeling problems again to enhance the accuracy and effectiveness of the screening.

[0060] Step S15: Receive update instructions from the client and update the annotations corresponding to multiple original samples according to the update instructions.

[0061] In one example of this embodiment, after the feedback data is sent to the client, data annotators or AI can determine whether the first and second problem samples have labeling errors based on the feedback data, such as the first problem sample and its corresponding label, and the labels of the second problem sample and the second problem sample. If data labeling errors are found in these samples, the data annotators can update the original samples with the problems.

[0062] In one example of this embodiment, the training results of the first and second problem samples in the neural network model are determined as feedback data.

[0063] In one example of this embodiment, the training results of the first and second problem samples in the neural network model can also be sent to the client as feedback data, so that data annotators or AI can refer to the training results to update the annotations of multiple original samples with standard problems.

[0064] In this example, the original samples and their corresponding labels are input into the neural network model for training. Based on the loss data during training, it is determined whether an original sample is a first-problem sample. Based on the first-problem sample, feedback data is determined and sent to the client. Then, update instructions from the client can be received to update the labels corresponding to multiple original samples. In this way, samples with data labeling problems can be automatically identified and promptly reported, improving the accuracy of identifying data labeling problems. Furthermore, data from potentially problematic samples can be fed back to confirm whether there are labeling errors and update the labels accordingly.

[0065] See Figure 2 As shown, this embodiment provides a data annotation optimization device 100, including an acquisition module 101 for acquiring a first sample set and a first annotation set. The first sample set includes multiple original samples, and the first annotation set includes multiple annotations corresponding to the original samples. A training module 102 is used to input the first sample set into a neural network model and train the neural network model using the first annotation set as supervision. A determination module 103 is used to acquire the loss data of the original samples and determine a first problem sample based on the loss data of the original samples. A feedback module 104 is used to determine feedback data based on the first problem sample and send the feedback data to a client. A receiving module 105 is used to receive an update instruction from the client and update the annotations corresponding to the multiple original samples according to the update instruction.

[0066] Optionally, the determining module includes: a setting submodule, used to set the number of training epochs *a* for training the first sample set in the neural network model, and that the loss data of the original samples includes *a* loss values. A setting submodule is used to set truncation parameters and obtain the data truncation range from these parameters and *a*. A first determining submodule is used to determine the expected value and variance of the loss values ​​of the original samples based on the *a* loss values. A second determining submodule is used to determine whether any sample in the original samples is a deviation sample based on the expected value and variance of the loss values. A normalization submodule is used to normalize the *a* loss values ​​of the deviation samples and obtain the average value of the data contained within the truncation range of each deviation sample after normalization. A third determining submodule is used to determine the deviation sample corresponding to the average value as the first problem sample if the average value is greater than a second preset threshold.

[0067] Optionally, the second determining submodule is specifically used to determine whether the a-th loss value among the loss values ​​of the original sample is greater than the sum of the expected value of the original sample and the first product, where the first product is the product of the preset parameter and the variance of the original sample. If the a-th loss value among the loss values ​​of the original sample is greater than the sum of the expected value of the original sample and the first product, the original sample is determined to be a deviation sample.

[0068] Optionally, the feedback module includes: a removal submodule, used to remove the first problem sample from the first sample set to obtain the second sample set, and to remove the labels corresponding to the first problem sample from the first label set to obtain the second label set; a training submodule, used to input the second sample set into the neural network model and train the neural network model using the second label set as supervision; a first acquisition submodule, used to input the first problem sample into the neural network model to obtain the loss data of the first problem sample; a fourth determination submodule, used to determine the first problem sample corresponding to the loss value in the loss data of the first problem sample as the second problem sample when the loss value is greater than a third preset threshold; and a fifth determination submodule, used to determine the first problem sample, the label corresponding to the first problem sample, the second problem sample, and the label corresponding to the first problem sample as feedback data.

[0069] Optionally, the feedback module further includes: a second obtaining submodule, used to input the second sample set and the first problem sample into the neural network model, and train the neural network model using the annotations corresponding to the second annotation set and the first problem sample as supervision to obtain the loss data of the second sample set and the first problem sample. A sixth determining submodule, used to determine the sample corresponding to the loss value as the second problem sample when the loss value in the loss data of the second sample set and the first problem sample is greater than a third preset threshold.

[0070] Optionally, the training module is also used to stop training the model when the average loss of multiple original samples decreases by less than a first preset threshold.

[0071] Optionally, the feedback module is also used to determine the training results of the first and second problem samples in the neural network model as feedback data.

[0072] This embodiment provides a computer-readable storage medium storing executable commands. When executed by a processor, the executable commands implement the various processes of the above-described data annotation optimization method embodiment and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0073] This embodiment provides an electronic device having a processor and a memory. The memory stores computer instructions. When the computer instructions are executed by the processor, they implement the various processes of the above-described data annotation optimization method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0074] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with authorization from the owner of the relevant device / account.

[0075] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and apparatus embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0076] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] Embodiments of this disclosure may be systems, methods, and / or computer program products. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of this disclosure.

[0078] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0079] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0080] Computer program instructions used to perform the operations of embodiments of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of embodiments of this disclosure.

[0081] Various aspects of embodiments of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0083] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0085] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An optimization method for data annotation, characterized in that, include: Obtain a first sample set and a first annotation set, wherein the first sample set includes multiple original samples and the first annotation set includes multiple annotations corresponding to the original samples; The first sample set is input into the neural network model, and the neural network model is trained using the first labeled set as supervision; Obtain the loss data of the original sample, and determine the first problem sample based on the loss data of the original sample; Feedback data is determined based on the first problem sample, and the feedback data is sent to the client; Receive update instructions from the client, and update the annotations corresponding to the multiple original samples according to the update instructions; The step of obtaining the loss data of the original sample and determining the first problem sample based on the loss data of the original sample includes: Let a be the number of training rounds a in which the first sample set is trained in the neural network model, and let a be the loss data of the original sample, which includes a loss values. Set the truncation parameter and obtain the data truncation range from the truncation parameter and 'a'; Based on the a loss values ​​of the original sample, determine the expected value and variance of the loss value of the original sample; Based on the expected value and variance of the loss value of the original sample, determine whether the original sample is a deviation sample; Normalize the a loss values ​​of the deviation samples to obtain the average value of the data contained within the data cut-off range of each deviation sample after normalization; If the average value is greater than a second preset threshold, the deviation sample corresponding to the average value is determined as the first problem sample.

2. The method according to claim 1, characterized in that, The step of determining whether a sample in the original sample is a deviation sample based on the expected value and variance of the loss value of the original sample includes: Determine whether the a-th loss value among the loss values ​​of the original sample is greater than the sum of the expected value of the original sample and the first product, where the first product is the product of a preset parameter and the variance of the original sample. When the a-th loss value in the original sample is greater than the sum of the expected value of the original sample and the first product, the original sample is identified as the deviation sample.

3. The method according to claim 1, characterized in that, The step of determining feedback data based on the first problem sample includes: Remove the first problem sample from the first sample set to obtain a second sample set, and remove the labels corresponding to the first problem sample from the first label set to obtain a second label set; The second sample set is input into the neural network model, and the neural network model is trained using the second labeled set as supervision; The first problem sample is input into the neural network model to obtain the loss data of the first problem sample; When the loss value in the loss data of the first problem sample is greater than a third preset threshold, the first problem sample corresponding to the loss value is determined as the second problem sample; The first problem sample, the label corresponding to the first problem sample, the second problem sample, and the label corresponding to the second problem sample are determined as feedback data.

4. The method according to claim 3, characterized in that, After determining the first problem sample whose loss value is greater than a third preset threshold as the second problem sample, the method includes: The second sample set and the first problem sample are input into the neural network model, and the neural network model is trained using the labels corresponding to the second label set and the first problem sample as supervision to obtain the loss data of the second sample set and the first problem sample. When the loss value in the loss data of the second sample set and the first problem sample is greater than a third preset threshold, the sample corresponding to the loss value is determined as the second problem sample.

5. The method according to claim 1, characterized in that, After inputting the first sample set into the neural network model and training the neural network model using the first labeled set as supervision, the method further includes: When the average loss of the multiple original samples decreases to less than a first preset threshold, training of the model is stopped.

6. The method according to claim 3 or 4, characterized in that, The method further includes: The training results of the first problem sample and the second problem sample in the neural network model are determined as feedback data.

7. An apparatus for optimizing data annotation, characterized in that, The device includes: An acquisition module is used to acquire a first sample set and a first annotation set, wherein the first sample set includes multiple original samples and the first annotation set includes multiple annotations corresponding to the original samples; The training module is used to input the first sample set into the neural network model and train the neural network model using the first labeled set as supervision. The determination module is used to acquire the loss data of the original sample and determine the first problem sample based on the loss data of the original sample. The feedback module is used to determine feedback data based on the first problem sample and send the feedback data to the client; The receiving module is used to receive update instructions from the client and update the labels corresponding to the multiple original samples according to the update instructions; The determining module includes: The setting submodule is used to set the number of training rounds a of the first sample set in the neural network model, and the loss data of the original sample includes a loss values. The settings submodule is used to set the truncation parameters and obtain the data truncation range from the truncation parameters and 'a'. The first determining submodule is used to determine the expected value and variance of the loss value of the original sample based on the a loss values ​​of the original sample; The second determining submodule is used to determine whether the original sample is a deviation sample based on the expected value and variance of the loss value of the original sample. The normalization submodule is used to normalize the a loss values ​​of the deviation sample and obtain the average value of the data contained in the data cut-off range of each deviation sample after normalization. The third determining submodule is used to determine the deviation sample corresponding to the average value as the first problem sample when the average value is greater than the second preset threshold.

8. A storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method described in any one of claims 1-6.

9. An electronic device, characterized in that, The device has a processor and a memory, the memory storing computer instructions that, when executed by the processor, implement the steps of the method according to any one of claims 1-6.

Images