Method, device and electronic device for data annotation

Abstract

An embodiment of the present application discloses a method, an apparatus, and an electronic device for data labeling, the method comprising: obtaining multiple recognition results after multiple rounds of recognition of data to be labeled, the data to be labeled comprising at least one character; when it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results; judging the recognition status of the data to be labeled based on the candidate results and the interference results, the recognition status comprising recognition success and recognition failure.

Description

[0001] This patent application is a divisional application of the patent application with application number 2018101157802, application date February 6, 2018, and invention name "Method, device and electronic device for data labeling". Technical Field

[0002] The present application relates to the field of computer data processing, and more specifically to a method, device and electronic device for data annotation. Background Art

[0003] At present, in order to protect data, most crawler channels usually display data to users in the form of pictures, which raises the threshold for data collection. Parsing data displayed in pictures (image data) can improve the competitiveness of crawler channels in the market. At present, manual annotation is required in scenarios where image data is recognized (such as optical character recognition scenarios, verification code recognition scenarios, and handwritten text recognition scenarios). The annotation set formed by the recognition results obtained by manual recognition of image data has a very important purpose.

[0004] However, the existing manual labeling process requires a lot of manpower and material resources, and human errors are inevitable, resulting in the inability to guarantee the effective utilization of the recognition results (or labeling results).

[0005] Therefore, a method for data annotation is needed to overcome the above technical problems. Summary of the invention

[0006] The purpose of the present application is to provide a method, device and electronic device for data labeling, which can ensure the effective utilization of recognition results.

[0007] To solve the above technical problems, the embodiments of the present application are implemented as follows:

[0008] In a first aspect, a method for data annotation is provided, comprising:

[0009] Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character;

[0010] When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results;

[0011] According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure.

[0012] In a second aspect, a device for data annotation is provided, comprising:

[0013] An acquisition unit, which acquires a plurality of recognition results after performing multiple rounds of recognition on the data to be labeled, wherein the data to be labeled includes at least one character;

[0014] a determining unit, which, when determining that there are different recognition results among the multiple recognition results, determines candidate results and interference results among the multiple recognition results;

[0015] A judgment unit is configured to judge, according to the candidate result and the interference result, the recognition status of the data to be labeled, wherein the recognition status includes recognition success and recognition failure.

[0016] According to a third aspect, an electronic device is provided, including:

[0017] Processor; and

[0018] a memory arranged to store computer executable instructions which, when executed, cause the processor to:

[0019] Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character;

[0020] When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results;

[0021] According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure.

[0022] In a fourth aspect, a computer-readable medium is provided, wherein the computer-readable medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device performs the following operations:

[0023] Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character;

[0024] When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results;

[0025] According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure.

[0026] It can be seen from the technical solution provided by the above embodiments of the present application that when there are different recognition results among multiple recognition results of the data to be labeled, the embodiments of the present application can judge the recognition situation of the data to be labeled based on the candidate results and the interference results in the multiple recognition results, thereby avoiding the problem of the effective utilization rate of the recognition results cannot be guaranteed due to the fact that the recognition situation of the data to be labeled can only be judged based on the recognition results when multiple recognition results are the same, thereby ensuring the effective utilization rate of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 is a schematic flowchart of a method for data annotation according to an embodiment of the present application.

[0029] Figure 2 It is a schematic flowchart of a method for data annotation according to a specific embodiment of the present application.

[0030] Figure 3 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0031] Figure 4 It is a structural diagram of a device for data annotation according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0033] Figure 1 is a flowchart of a method for data annotation according to an embodiment of the present application. Figure 1 The method is performed by a device for data annotation. It should be understood that the solution of the embodiment of the present application can be applied to the annotation of text image data. Of course, it is not excluded that the method of the embodiment of the present application can be applied to the annotation of other data.

[0034] like Figure 1As shown, at S102, multiple recognition results are obtained after multiple rounds of recognition of the data to be labeled, and the data to be labeled includes at least one character.

[0035] It is understandable that multiple people can identify the data to be labeled in each round of recognition process. For example, three people can identify the data to be labeled in each round. And the participants in any two rounds of recognition process can be different. For example, if the participants in the first round of recognition process are A, B and C, then the participants in the second round of recognition process are D, E and F. Each person identifies the data to be labeled and obtains a recognition result.

[0036] Optionally, in some embodiments, the total number of rounds for identifying the data to be labeled and / or the number of participants in each round of identification is determined according to the timeliness performance requirement. For example, the total number of rounds for identifying the data to be labeled and / or the number of participants in each round of identification is determined according to the rule that the higher the timeliness performance requirement, the fewer the total number of rounds for identification and / or the fewer the number of participants in each round of identification.

[0037] At S104, when it is determined that there are different recognition results among the multiple recognition results, candidate results and interference results among the multiple recognition results are determined.

[0038] Optionally, in some embodiments, if the multiple recognition results are all the same, the accuracy of the recognition results can be directly calculated, and the recognition status of the data to be labeled can be determined based on the accuracy of the recognition results and the accuracy requirement.

[0039] Specifically, in some embodiments, the candidate result and the interference result are determined according to the number of occurrences, wherein the candidate result is the recognition result with the largest number of occurrences among the multiple recognition results, and the interference result is the recognition result other than the interference result among the multiple recognition results. The number of occurrences here can also be understood as the number of votes or the number of people who identify the data to be labeled and obtain the same recognition result.

[0040] For example, the data to be labeled includes 4 characters from 0 to 3. After 6 people recognize the data to be labeled, the recognition results obtained are 1234, 1324, 1234, 1342, 1234 and 123 respectively. Here, 1234 appears 4 times, 1324 appears 1 time, and 123 appears 1 time. Therefore, 1234 is determined as the candidate result, and 1324 and 123 are determined as interference results.

[0041] Further, in some embodiments, the number of characters included in the interference result is the same as the number of characters included in the candidate result. Or it can be understood that when determining the candidate results and interference results in multiple recognition results, the one with the most occurrences is determined as the candidate result, and the other recognition results in which the number of characters included in the multiple recognition results is the same as the number of characters included in the candidate result are determined as interference results, and if there are recognition results in which the number of characters included is different from the number of characters included in the candidate result among the multiple recognition results, then the recognition results in which the number of characters included is different from the number of characters included in the candidate result can be ignored, or it can be understood as misalignment and discarding the recognition results in which the number of characters included is different from the number of characters included in the candidate result.

[0042] For example, the data to be labeled includes 4 characters from 0 to 3. After 6 people recognize the data to be labeled, the recognition results are 1234, 1324, 1234, 1342, 1234 and 123 respectively. Here, 1234 appears 4 times, 1324 appears 1 time, and 123 appears 1 time. Therefore, 1234 is determined as a candidate result, 1324 is determined as an interference result, and 123 is discarded.

[0043] At S106, the recognition status of the data to be labeled is determined according to the candidate results and the interference results, where the recognition status includes recognition success and recognition failure.

[0044] Specifically, in some embodiments, the type of the candidate result is judged according to the distribution algorithm model, the number of occurrences of the candidate result and the number of occurrences of the interference result, the type of the candidate result includes a determined valid result, a determined invalid result and a valid result to be determined, and the distribution algorithm model is trained based on the accuracy requirements and the recognition results of the training data to be labeled; according to the type of the candidate result, the recognition status of the data to be labeled is judged.

[0045] Optionally, as an example, in the process of training the distribution algorithm model, the accuracy requirement is determined, and the total number of recognition rounds and the number of unique participants in each round are determined based on the timeliness performance requirements. Then, the total number of participants is counted, the distribution of all possible recognition results is analyzed, the estimated accuracy of each distribution is determined, and then the distribution algorithm model is determined based on the estimated accuracy of each distribution and the accuracy requirement.

[0046] For example, assuming that the training data to be labeled for training the distribution algorithm model includes ten characters from 0 to 9, and the character recognition accuracy is a (the character recognition accuracy here should be understood as the lower limit of the character recognition accuracy), the probability of the ten characters from 0 to 9 being misrecognized is initially equal to (1-a) / 9, a=0.9, then the estimated accuracy of each distribution situation can be calculated.

[0047] For example, when there are 2 people to be identified, the distribution of the identification results of these 2 people is the same as (2), and the estimated accuracy of (2) is 99.86301369863013%. When there are 3 people to be identified, the distribution of the identification results of these 3 people is the same as (3), and the estimated accuracy of (3) is 99.99830651989839%. When there are 4 people to be identified, the distribution of the identification results of these 4 people is the same as (4), and the estimated accuracy of (4) is 99.99997909248857%. When there are 5 people to be identified, the distribution of the identification results of these 5 people is the same as (5), and the estimated accuracy of (5) is 99.99999974188252%. When there are 6 people to be identified, the distribution of the identification results of these 6 people is the same as (6), and the estimated accuracy of (6) is 99.99999999681336%. When there are 3 people to be identified, 2 of them have the same identification results, and the other person's identification result is different from the other two people, the distribution is (2,1), and the estimated accuracy of (2,1) is 98.66165413533837%. When there are 4 people to be identified, 3 of them have the same identification results, and the other person's identification result is different from the other three people, the distribution is (3,1), and the estimated accuracy of (3,1) is 99.98325588395764%. When there are 5 people to be identified, 3 of them have the same identification results, and the other 2 have the same identification results and are different from the other three people, the distribution is (3,2), and the estimated accuracy of (3,2) is 98.77901897734243%. When there are 5 people to be identified, the distribution of 4 out of the 5 people with the same identification results and the other person with a different identification result is (4,1), and the estimated accuracy of (4,1) is 99.99979324832664%. When there are 6 people to be identified, the distribution of 4 out of the 6 people with the same identification results and the other 2 people with the same identification results and different identification results is (4,2), and the estimated accuracy of (4,2) is 99.9847421648845%. When there are 6 people to be identified, the distribution of 5 out of the 6 people with the same identification results and the other person with a different identification result is (5,1), and the estimated accuracy of (5,1) is 99.999997447505%. When there are 6 people to be identified, the identification results of 4 of the 6 people are the same and the identification results of the other 2 people are different from the identification results of the 4 people, and the identification results of the other 2 people are also different. The distribution is (4,1,1), and the estimated accuracy of (4,1,1) is 99.98994724384438%.When there are 5 people to be identified, the identification results of 3 of them are the same and the identification results of the other 2 are different from the identification results of the 3 people and the identification results of the other 2 are also different, the distribution is (3,1,1), and the estimated accuracy of (3,1,1) is 98.96391181294628%. When there are 4 people to be identified, the identification results of 2 of them are the same and the identification results of the other 2 are different from the identification results of the other 2 and the identification results of the other 2 are also different, the distribution is (2,1,1), and the estimated accuracy of (2,1,1) is 45.67351200835363%. When there are 6 people to be identified, 3 of them have the same identification results and the other 3 have different identification results from these 3 people, and the other 3 have different identification results, the distribution is (3,1,1,1), and the estimated accuracy of (3,1,1,1) is 81.24386157490324%. When there are 6 people to be identified, 3 of them have the same identification results and the other 3 have different identification results from these 3 people, and 2 of the other 3 have the same identification results, the distribution is (3,2,1), and the estimated accuracy of (3,2,1) is 79.73229373797001%.

[0048] Assuming that the accuracy requirement is higher than 99.99%, then combining the estimated accuracy and accuracy requirements of the above distribution situations, the distribution algorithm model can be obtained as follows: (a) when x<=y, the type of candidate result is determined invalid result; (b) when x=2, the type of candidate result is determined invalid result; (c) when xy>2, the type of candidate result is determined valid result; (d) when xy<=2, the type of candidate result is to be determined valid result. Here x represents the number of occurrences of the candidate result, and y represents the number of occurrences of the interference result.

[0049] Optionally, as an embodiment, when it is determined that the type of the candidate result is a determined valid result, it is determined that the data to be labeled is identified successfully, and the candidate result is determined as a labeling result of the data to be labeled.

[0050] Furthermore, when the candidate result is determined to be a valid result, the number of correct recognitions and the number of incorrect recognitions of at least one character included in the data to be labeled are updated according to the number of occurrences of the candidate result and the number of occurrences of the interference result.

[0051] For example, assuming that 6 people identify the data to be labeled, the candidate results appear 5 times, the interference results appear 1 time, and the interference results identify character 6 in the data to be labeled as character 8, then the number of times at least one character included in the data to be labeled is correctly recognized is increased by 5, and the number of times character 6 in the at least one character is recognized as character 8 is increased by 1. This makes it easier to count the probability of each character in the at least one character included in the data to be labeled being recognized as other characters (or the recognition probability between at least one character), and facilitates the subsequent more accurate determination of the accuracy corresponding to the distribution of recognition results, improves the effective utilization of recognition results, reduces costs and enriches sample sets.

[0052] Optionally, as another embodiment, when it is determined that the type of the candidate result is an invalid result, the relationship between the arguments that have been performed on the data to be labeled and the preset recognition arguments is determined; when it is determined that the number of arguments that have been performed on the data to be labeled is less than the preset number of recognition rounds, the recognition operation is performed again on the data to be labeled; based on the result of performing the recognition operation again on the data to be labeled, multiple recognition results are updated, and the recognition status of the data to be labeled is determined based on the candidate results and interference results in the updated multiple recognition results. Or it can be understood that when it is determined that the number of arguments that have been performed on the data to be labeled is less than the preset number of recognition rounds, the recognition operation is repeated. Figure 1 Alternatively, when it is determined that the statement that has been used to perform the recognition operation on the data to be labeled is equal to the preset recognition statement, it is determined that the recognition of the data to be labeled fails.

[0053] Optionally, as another embodiment, when it is determined that the type of the candidate result is a valid result to be determined, it is determined whether the candidate result meets the accuracy requirement; when it is determined that the candidate result meets the accuracy requirement, it is determined that the data to be labeled is identified successfully, and the candidate result is determined as the labeling result of the data to be labeled.

[0054] Specifically, judging whether the candidate result meets the accuracy requirement includes: determining the recognition accuracy of the at least one character; judging whether the candidate result meets the accuracy requirement based on the recognition accuracy of the at least one character and the accuracy requirement. In other words, judging whether the candidate result meets the accuracy requirement is actually dividing the data to be annotated into multiple individual characters (the candidate result is actually the recognition result of these multiple individual characters), and confirming whether the accuracy requirement is met for each character. It can be understood that when the recognition accuracy of each character meets the accuracy requirement, the candidate result meets the accuracy requirement, otherwise the candidate result does not meet the accuracy requirement.

[0055] In some embodiments, determining the recognition accuracy of at least one character includes: judging the type of the at least one character based on the distribution algorithm model, the number of occurrences of the candidate results and the number of occurrences of the interference results, the type of the at least one character including a determined valid character, a determined invalid character and a valid character to be determined; when the at least one character is judged to be a valid character to be determined, determining the recognition accuracy of the at least one character based on the candidate results.

[0056] Or it can be understood that when the type of candidate result is a valid result to be determined, it is necessary to determine whether the recognition accuracy of each character in the data to be labeled meets the accuracy requirements. In this case, the recognition accuracy of each character can be determined in the following way: the voting distribution of the current character is counted according to the number of occurrences of the candidate results and the number of occurrences of the interference results, and the type of the current character is determined according to the distribution algorithm model. If the type of the current character is a confirmed valid character, the voting distribution of the next character is counted according to the number of occurrences of the candidate results and the number of occurrences of the interference results, and subsequent operations are performed according to the voting distribution of the next character; if the type of the current character is a confirmed invalid character, it is determined that the candidate result does not meet the accuracy requirements and needs to be restarted. Figure 1 The method shown; if the type of the current character is a valid character to be determined, the recognition accuracy of the current character is determined, and the voting distribution of the next character is further counted according to the number of occurrences of the candidate results and the number of occurrences of the interference results, and subsequent operations are performed according to the voting distribution of the next character.

[0057] Optionally, as an example, determining the recognition accuracy of at least one character includes: determining a conditional probability formula corresponding to the candidate result; determining the recognition accuracy of the at least one character based on the conditional probability formula and a recognition probability matrix, the elements in the recognition probability matrix being used to describe the probability that one of the at least one character is recognized as another character in the at least one character.

[0058] For example, the elements in the recognition probability matrix W can be expressed as W ij , W ij represents the probability that character j is recognized as character i, and W ij It can be determined by formula (1):

[0059]

[0060] In formula (1), C ij represents the number of times character j is recognized as character i, Represents the total number of times character j is recognized.

[0061] The following will describe the conditional probability formula of the candidate results according to the embodiment of the present application with reference to specific examples.

[0062] For example, if three people participate in each round of recognition, and if a round of recognition is performed on the labeled data, and the recognition results of the three people are the same in this round of recognition, the conditional probability formula under the condition that the recognition results of the three people are the same can be expressed as formula (2):

[0063]

[0064] And P(X1=j, X2=j, X3=j|A=j) can be expressed as: P(X1=jA=j)·P(X2=j|A=j)·P(X3=j|A=j), then formula (2) is transformed into formula (3):

[0065]

[0066] Substituting the elements of the recognition probability matrix W into formula (3), we can determine the recognition accuracy of each character when the recognition results of three people are the same. And if we assume that the probability W of character j being correctly recognized is jj = a, then the probability that character j is not correctly recognized From this we get and then Assuming a=90% and the accuracy requirement is greater than 98%, it can be determined that when the recognition results of three people are the same, the accuracy of each character meets the accuracy requirement. The recognition results of these three people meet the accuracy requirement, and the recognition results of these three people can be determined as the labeling results of the data to be recognized.

[0067] Furthermore, if the recognition results of the above three people are inconsistent, and the next round of recognition is performed on the data to be annotated, if the case where a character is mistakenly recognized as multiple different characters at the same time is not considered, and the recognition results with lower accuracy are removed, the following eight events will occur for a character: Event 1: √×××××; Event 2: √××√××; Event 3: √××√√×; Event 4: √××√√√; Event 5: √√××××; Event 6: √√×√××; Event 7: √√×√√×; Event 8: √√×√√√. Among them, √ and × represent different recognition results, √ represents correct recognition, and × represents incorrect recognition. Analyzing the above 8 events, we can get 4 scenarios, among which, scenario 1: there are two recognition results in both event 1 and event 8, and the number of occurrences of the two recognition results is 5:1; scenario 2: there are two recognition results in both event 2 and event 7, and the number of occurrences of the two recognition results is 4:2; scenario 3: there are two recognition results in both event 4 and event 5, and the number of occurrences of the two recognition results is 4:2; scenario 4: there are two recognition results in both event 3 and event 6, and the number of occurrences of the two recognition results is 3:3.

[0068] Specifically, the conditional probability formula of event 8 in scenario 1 is formula (4):

[0069]

[0070] If we assume that the probability W of character j being correctly recognized jj = a, then the probability that character j is not correctly recognized is (1-a), and then Further assume that character j is recognized as character i as a common error, that is, character j is only recognized as character i in addition to being recognized as itself, W ij =1-a; when character j is recognized as character i, which is a rare error, that is, character j is never recognized as character i, W ij ≈0; a=90%. It can be determined that the accuracy of the recognition result of event 8 is 99.97% when errors are common and approximately 0 when errors are rare.

[0071] Alternatively, the conditional probability formula for event 7 in scenario 2 is formula (5):

[0072]

[0073] And if we assume that the probability W of character j being correctly recognized is jj = a, then the probability that character j is not correctly recognized is (1-a), and then Further assume that character j is recognized as character i as a common error, that is, character j is only recognized as character i in addition to being recognized as itself, W ij=1-a; when character j is recognized as character i, which is a rare error, that is, character j is never recognized as character i, W ij ≈0; a=90%. It can be determined that the accuracy of the recognition result of event 7 is 98.662% when errors are common and approximately 0 when errors are rare.

[0074] Alternatively, the conditional probability formula for event 3 in scenario 4 is formula (6):

[0075]

[0076] Further assume that character j is recognized as character i as a common error, that is, character j is only recognized as character i in addition to being recognized as itself, W ij =1-a; when character j is recognized as character i, which is a rare error, that is, character j is never recognized as character i, W ij ≈0; a=90%. It can be determined that the accuracy of the recognition result of event 3 is 49.727% when there are common errors and is approximately 0 when there are rare errors.

[0077] Therefore, without considering that characters are mistakenly recognized as multiple different numbers at the same time, when the errors are common errors, scenarios 1, 2 and 3 are scenarios where the accuracy of the recognition results meets the requirements or are acceptable scenarios, and when rare errors occur, scenarios 1, 2 and 3 are unacceptable scenarios. Scenario 4 is an unacceptable scenario regardless of whether the errors are common errors or not.

[0078] If we further consider the case where a character is mistakenly recognized as multiple different characters at the same time, for example, 4 people recognize it correctly and 2 people recognize it incorrectly, and the results of the incorrect recognition are different, the same character will have 3 recognition results, the number of recognition results is (4, 1, 1), and the recognition result with the largest number of occurrences is determined as the candidate result, then the conditional probability formula corresponding to the candidate result is formula (7):

[0079]

[0080] In formula (7) It can be expressed by formula (8):

[0081]

[0082] By using the mean inequality to analyze the last two terms in formula (8), we can obtain:

[0083]

[0084]

[0085]

[0086] It follows that:

[0087]

[0088] When i1 and i2 are both common errors, but:

[0089]

[0090] Assuming a = 0.9, when i1 and i2 are both common errors, the accuracy of the candidate result is greater than 99.485%, and when i1 and i2 are both rare errors, W i1j ,W i2j ≈0, the accuracy of the candidate result is close to 0.

[0091] It is understandable that the above description of the conditional probability formula of the candidate results and the determination of the accuracy rate based on the conditional probability formula take into account the extreme error situation. In the actual process, the elements in the recognition probability matrix W are brought into the specific conditional probability formula to obtain the specific recognition accuracy rate of each character in the data to be annotated, and the validity of the candidate results can be determined based on the specific recognition accuracy rate of each character obtained.

[0092] In an embodiment of the present application, in order to obtain an accurate recognition accuracy rate of at least one character, before determining the recognition accuracy rate of at least one character according to the conditional probability formula and the recognition probability matrix, it is determined that the total number of times the at least one character is recognized is greater than or equal to the preset number of recognition times. For example, if the preset recognition is 100 times at this time, then if the total number of times the at least one character is recognized is greater than 100 times, the elements in the recognition probability matrix can be substituted into the corresponding conditional probability formula of the candidate result to obtain the recognition accuracy rate of the at least one character. If the recognition accuracy rate of at least one character meets the accuracy rate requirement, it is determined that the candidate result meets the accuracy rate requirement. In this case, the number of times the at least one character included in the data to be annotated is correctly recognized and the number of times it is incorrectly recognized will not be updated based on the number of times the candidate result appears and the number of times the interference result appears.

[0093] Figure 2 It is a schematic flowchart of a method for data annotation according to a specific embodiment of the present application. Figure 2 The method can be performed by a device for data annotation. Figure 2 As shown, Figure 2 The method shown includes two main processes: distribution algorithm model training and recognition situation judgment. The distribution model training process includes S202 to S210, and the recognition situation judgment process includes S212-S234; wherein:

[0094] At S202 , the accuracy requirement is determined.

[0095] At S204 , the total number of manual identification rounds and the number of unique participants in each round are determined.

[0096] Specifically, the total number of manual identification rounds and the number of unique participants in each round may be determined according to timeliness performance requirements.

[0097] At S206, the cumulative number of participants is counted and the distribution of all possible results is analyzed.

[0098] The distribution of all possible results here can be understood as Figure 1 The number of occurrences in the method shown.

[0099] It can be understood that at S206, when different times of manual identification are performed, the cumulative number of participants will be different, and the distribution of all possible results corresponding to different times of manual identification will be different. Alternatively, it can be understood that multiple result distributions can be obtained at S206.

[0100] At S208, the estimated accuracy of each result distribution is determined.

[0101] Specifically, the estimated accuracy of each result distribution situation can be determined according to the conditional probability formula of each result distribution situation.

[0102] At S210 , a distribution algorithm model is determined based on the accuracy requirement and the estimated accuracy of each division.

[0103] At S212, it is determined whether the number of manual identification rounds that have been executed meets the total number of rounds requirement.

[0104] At S214, if the total number of manual identification rounds performed does not meet the total number of rounds requirement, manual identification is performed once, and the identification results of all rounds are counted to obtain multiple identification results.

[0105] Optionally, if the total number of rounds of manual recognition performed at S212 meets the total number of rounds requirement, it is determined at S214 that the recognition has failed.

[0106] At S216, the number of occurrences of each recognition result is counted, and candidate results and interference results are determined.

[0107] Optionally, the recognition result with the largest number of occurrences is determined as a candidate result, the recognition result with the same length as the candidate result among multiple candidate results is determined as an interference result, and the other recognition results among the multiple recognition results except the candidate result and the interference result are regarded as misplaced and discarded.

[0108] At S218, based on the distribution algorithm model, it is determined whether the candidate result is a confirmed valid result.

[0109] At S220, if the candidate result is a confirmed valid result, the candidate result and the interference result are counted by characters, and the recognition probability matrix between the characters is accumulated.

[0110] At S220, the elements in the recognition probability matrix between characters are used to describe the probability that a character is recognized as other characters. The character statistics of candidate results and interference results can be understood as counting the number of times each character is correctly recognized and the number of times it is incorrectly recognized according to the number of occurrences of candidate results and the number of occurrences of interference results, and the number of times it is incorrectly recognized here specifically refers to the number of times a character is recognized as another character, for example, the number of times 8 is recognized as 6, or the number of times 8 is recognized as 3.

[0111] Further, at S220, if the candidate result is a confirmed valid result, it is returned to confirm that the recognition is successful, and the recognition result is returned. It can be understood that the recognition result returned here is the candidate result.

[0112] At S222, if the candidate result is not a determined valid result, it is determined whether the candidate result is a determined invalid result according to the distribution algorithm model.

[0113] At S224, if the candidate result is not a determined invalid result, the candidate results are looped character by character, and it is determined whether the loop is completed.

[0114] At S224, if the candidate result is determined to be an invalid result, S212 and subsequent steps are executed.

[0115] At S224, if the loop is completed, it is determined that the recognition is successful, and the recognition result is returned. It can be understood that the recognition result returned here is the candidate result.

[0116] At S226, if the loop is not completed, the voting distribution of the current character is counted based on the candidate results and the interference results.

[0117] At S228, it is determined whether the current character is a valid character according to the distribution algorithm model.

[0118] At S230 , if the current character is not a confirmed valid character, it is determined whether the current character is a confirmed invalid character according to the distribution algorithm model.

[0119] Optionally, at S230, if the current character is a confirmed valid character, S224 and subsequent steps are executed.

[0120] At S232, if the current character is not determined to be an invalid character, the recognition accuracy of the current character is determined based on the recognition probability matrix between characters and the conditional probability formula corresponding to the candidate results.

[0121] Optionally, at S232, if the current character is determined to be an invalid character, S212 and subsequent steps are executed.

[0122] At S234, it is determined whether the recognition accuracy of the current character meets the accuracy requirement. If so, S224 and subsequent steps are executed; otherwise, S212 and subsequent steps are executed.

[0123] It should be noted that, although the above embodiment places S222 after S218 for the sake of convenience, in actual situations, the execution order of these two steps can be swapped, that is, first determine whether the candidate result is a confirmed invalid result, if it is not a confirmed invalid result, determine whether the candidate result is a confirmed valid result, if it is a confirmed valid result, execute S220, if it is not a confirmed valid result, execute S224. Similarly, although 230 is placed after S228, in actual situations, the execution order of these two steps can be swapped, that is, first determine whether the current character is a confirmed invalid character, if the current character is not a confirmed invalid character, determine whether the current character is a confirmed valid character, if the current character is a confirmed valid character, execute S224 and its subsequent steps, if the current character is not a confirmed valid character, determine the recognition accuracy of the current character based on the recognition probability matrix between characters and the conditional probability formula corresponding to the candidate results.

[0124] Combination of the above Figure 1 and Figure 2 The method for data annotation according to the embodiment of the present application is described in detail. Figure 3 The electronic device according to the embodiment of the present application is described in detail. Figure 3 At the hardware level, the electronic device includes a processor, and optionally, an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0125] The processor, network interface and memory can be interconnected through an internal bus, which can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0126] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0127] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for data labeling at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0128] Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character;

[0129] When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results;

[0130] According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure.

[0131] The above application Figure 1 and Figure 2The method for executing the device for data annotation disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0132] The electronic device may also perform Figure 2 A method and a device for data annotation are implemented in Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0133] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0134] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 and Figure 2 The method of the embodiment shown is specifically used to perform the following method:

[0135] Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character;

[0136] When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results;

[0137] According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure.

[0138] Figure 4 This is a schematic diagram of a device for data source labeling according to an embodiment of the present application. Figure 4 In a software implementation, the device 400 for data annotation may include: an acquisition unit 401, a determination unit 402 and a judgment unit 403, wherein:

[0139] An acquisition unit 401 acquires a plurality of recognition results after performing multiple rounds of recognition on the data to be labeled, wherein the data to be labeled includes at least one character;

[0140] The determining unit 402 determines candidate results and interference results among the multiple recognition results when it is determined that there are different recognition results among the multiple recognition results;

[0141] The judging unit 403 judges the recognition status of the data to be labeled according to the candidate results and the interference results, where the recognition status includes recognition success and recognition failure.

[0142] According to the device for data labeling of the embodiment of the present application, when there are different recognition results among multiple recognition results of the data to be labeled, it is possible to judge the recognition status of the data to be labeled based on the candidate results and the interference results in the multiple recognition results, thereby avoiding the problem of the effective utilization rate of the recognition results not being guaranteed due to the fact that the recognition status of the data to be labeled can only be judged based on the recognition results when multiple recognition results are the same, thereby ensuring the effective utilization rate of the recognition results.

[0143] Optionally, as an embodiment, the determining unit 402:

[0144] The candidate result and the interference result are determined according to the number of occurrences, wherein the candidate result is the recognition result with the largest number of occurrences among the multiple recognition results, and the interference result is the recognition result other than the interference result among the multiple recognition results.

[0145] Optionally, as an embodiment, the number of characters included in the interference result is the same as the number of characters included in the candidate result.

[0146] Optionally, as an embodiment, the judging unit 403:

[0147] According to the distribution algorithm model, the number of occurrences of the candidate result and the number of occurrences of the interference result, the type of the candidate result is determined, wherein the type of the candidate result includes a determined valid result, a determined invalid result and a to-be-determined valid result, and the distribution algorithm model is trained based on the accuracy requirement and the recognition result of the training data to be labeled;

[0148] The identification status of the data to be labeled is determined according to the type of the candidate result.

[0149] Optionally, as an embodiment, the judging unit 403

[0150] When it is determined that the type of the candidate result is a determined valid result, it is determined that the data to be labeled is successfully identified, and the candidate result is determined as the labeling result of the data to be labeled.

[0151] Optionally, as an embodiment, when determining that the type of the candidate result is a determined valid result, the determining unit 403:

[0152] According to the number of times the candidate result appears and the number of times the interference result appears, the number of times at least one character included in the data to be labeled is correctly recognized and the number of times it is incorrectly recognized is updated.

[0153] Optionally, as an embodiment, the judging unit 403:

[0154] When it is determined that the type of the candidate result is a determined invalid result, determining the relationship between the number of rounds of recognition operations that have been performed on the data to be labeled and a preset number of recognition rounds;

[0155] When it is determined that the number of rounds of the recognition operation performed on the data to be labeled is less than the preset number of rounds of recognition, performing the recognition operation on the data to be labeled again;

[0156] According to the result of performing the recognition operation again on the data to be labeled, the multiple recognition results are updated, and the recognition status of the data to be labeled is determined according to the candidate results and the interference results in the updated multiple recognition results.

[0157] Optionally, as an embodiment, the judging unit 403:

[0158] When it is determined that the number of rounds of identification operations performed on the data to be labeled is equal to the preset number of identification rounds, it is determined that the identification of the data to be labeled fails.

[0159] Optionally, as an embodiment, the judging unit 403:

[0160] When it is determined that the type of the candidate result is a valid result to be determined, determining whether the candidate result meets the accuracy requirement;

[0161] When it is determined that the candidate result meets the accuracy requirement, it is determined that the data to be labeled is successfully identified, and the candidate result is determined as the labeling result of the data to be labeled.

[0162] Optionally, as an embodiment, the judging unit 403:

[0163] determining a recognition accuracy rate of the at least one character;

[0164] According to the recognition accuracy of the at least one character and the accuracy requirement, it is determined whether the candidate result meets the accuracy requirement.

[0165] Optionally, as an embodiment, the judging unit 403:

[0166] Determine the type of the at least one character according to the distribution algorithm model, the number of occurrences of the candidate results, and the number of occurrences of the interference results, wherein the type of the at least one character includes a determined valid character, a determined invalid character, and a valid character to be determined;

[0167] When it is determined that the at least one character is a valid character to be determined, the recognition accuracy of the at least one character is determined.

[0168] Optionally, as an embodiment, the at least one character is at least two characters;

[0169] Wherein, the judging unit 403:

[0170] Determine a conditional probability formula corresponding to the candidate result;

[0171] The recognition accuracy of the at least one character is determined according to the conditional probability formula and the recognition probability matrix, and the elements in the recognition probability matrix are used to describe the probability that one of the at least one character is recognized as another character in the at least one character.

[0172] Optionally, as an embodiment, before determining the recognition accuracy of the at least one character according to the conditional probability formula and the recognition probability matrix, the judging unit 403:

[0173] It is determined that the total number of times the at least one character is recognized is greater than or equal to a preset number of recognition times.

[0174] The apparatus 400 for data annotation may also perform Figure 1 and Figure 2The method of the embodiment shown in the figure and the device for data annotation are implemented in Figure 1 and Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0175] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0176] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0177] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0178] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0179] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A method for data annotation, comprising: Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character; When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results; According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure, including: Determine the type of the candidate result according to the distribution algorithm model, the number of occurrences of the candidate result and the number of occurrences of the interference result, wherein the type of the candidate result includes a determined valid result, a determined invalid result and a to-be-determined valid result; According to the type of the candidate result, determining the recognition status of the data to be labeled; The training steps of the distribution algorithm model include: Determine the total number of manual identification rounds and the number of unique participants in each round; Count the cumulative number of participants and analyze the distribution of recognition results of the training data to be labeled; Determine the estimated accuracy of each distribution; The distribution algorithm model is determined in combination with the accuracy requirement and the estimated accuracy of each distribution situation.

2. The method according to claim 1, wherein determining candidate results and interference results among the plurality of recognition results comprises: The candidate result and the interference result are determined according to the number of occurrences, wherein the candidate result is the recognition result with the largest number of occurrences among the multiple recognition results, and the interference result is the recognition result other than the interference result among the multiple recognition results. 3 . The method according to claim 2 , wherein the number of characters included in the interference results is the same as the number of characters included in the candidate results.

4. According to the method of claim 3, judging the identification status of the data to be labeled according to the type of the candidate result comprises: When it is determined that the type of the candidate result is a determined valid result, it is determined that the data to be labeled is successfully identified, and the candidate result is determined as the labeling result of the data to be labeled.

5. The method according to claim 4, when determining that the type of the candidate result is a confirmed valid result, further comprising: According to the number of times the candidate result appears and the number of times the interference result appears, the number of times at least one character included in the data to be labeled is correctly recognized and the number of times it is incorrectly recognized is updated.

6. The method according to claim 4, further comprising: When it is determined that the type of the candidate result is a determined invalid result, determining the relationship between the number of rounds of recognition operations that have been performed on the data to be labeled and a preset number of recognition rounds; When it is determined that the number of rounds of the recognition operation performed on the data to be labeled is less than the preset number of rounds of recognition, performing the recognition operation on the data to be labeled again; According to the result of performing the recognition operation again on the data to be labeled, the multiple recognition results are updated, and the recognition status of the data to be labeled is determined according to the candidate results and the interference results in the updated multiple recognition results.

7. The method according to claim 6, further comprising: When it is determined that the number of rounds of identification operations performed on the data to be labeled is equal to the preset number of identification rounds, it is determined that the identification of the data to be labeled fails.

8. The method according to claim 3, further comprising: When it is determined that the type of the candidate result is a valid result to be determined, determining whether the candidate result meets the accuracy requirement; When it is determined that the candidate result meets the accuracy requirement, it is determined that the data to be labeled is successfully identified, and the candidate result is determined as the labeling result of the data to be labeled.

9. The method according to claim 8, The determining whether the candidate result meets the accuracy requirement includes: determining a recognition accuracy rate of the at least one character; According to the recognition accuracy of the at least one character and the accuracy requirement, it is determined whether the candidate result meets the accuracy requirement.

10. The method according to claim 9, The determining the recognition accuracy of the at least one character comprises: Determine the type of the at least one character according to the distribution algorithm model, the number of occurrences of the candidate results, and the number of occurrences of the interference results, wherein the type of the at least one character includes a determined valid character, a determined invalid character, and a valid character to be determined; When it is determined that the at least one character is a valid character to be determined, the recognition accuracy of the at least one character is determined.

11. The method according to claim 10, wherein the at least one character is at least two characters; in, The determining the recognition accuracy of the at least one character comprises: Determine a conditional probability formula corresponding to the candidate result; The recognition accuracy of the at least one character is determined according to the conditional probability formula and the recognition probability matrix, and the elements in the recognition probability matrix are used to describe the probability that one of the at least one character is recognized as another character in the at least one character.

12. The method according to claim 11, before determining the recognition accuracy of the at least one character according to the conditional probability formula and the recognition probability matrix, further comprises: It is determined that the total number of times the at least one character is recognized is greater than or equal to a preset number of recognition times.

13. A device for data annotation, comprising: An acquisition unit, which acquires a plurality of recognition results after performing multiple rounds of recognition on the data to be labeled, wherein the data to be labeled includes at least one character; a determining unit, which, when determining that there are different recognition results among the multiple recognition results, determines candidate results and interference results among the multiple recognition results; A judgment unit, which judges the recognition status of the data to be labeled according to the candidate result and the interference result, wherein the recognition status includes recognition success and recognition failure, including: Determine the type of the candidate result according to the distribution algorithm model, the number of occurrences of the candidate result and the number of occurrences of the interference result, wherein the type of the candidate result includes a determined valid result, a determined invalid result and a to-be-determined valid result; According to the type of the candidate result, determining the recognition status of the data to be labeled; The training steps of the distribution algorithm model include: Determine the total number of manual identification rounds and the number of unique participants in each round; Count the cumulative number of participants and analyze the distribution of recognition results of the training data to be labeled; Determine the estimated accuracy of each distribution; The distribution algorithm model is determined in combination with the accuracy requirement and the estimated accuracy of each distribution situation.

14. An electronic device comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character; When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results; According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure, including: Determine the type of the candidate result according to the distribution algorithm model, the number of occurrences of the candidate result and the number of occurrences of the interference result, wherein the type of the candidate result includes a determined valid result, a determined invalid result and a to-be-determined valid result; According to the type of the candidate result, determining the identification status of the data to be labeled; The training steps of the distribution algorithm model include: Determine the total number of manual identification rounds and the number of unique participants in each round; Count the cumulative number of participants and analyze the distribution of recognition results of the training data to be labeled; Determine the estimated accuracy of each distribution; The distribution algorithm model is determined in combination with the accuracy requirement and the estimated accuracy of each distribution situation.

15. A computer-readable medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the following operations: Acquire multiple recognition results after multiple rounds of recognition of the data to be labeled, wherein the data to be labeled includes at least one character; When it is determined that there are different recognition results among the multiple recognition results, determining candidate results and interference results among the multiple recognition results; According to the candidate results and the interference results, the recognition status of the data to be labeled is judged, and the recognition status includes recognition success and recognition failure, including: Determine the type of the candidate result according to the distribution algorithm model, the number of occurrences of the candidate result and the number of occurrences of the interference result, wherein the type of the candidate result includes a determined valid result, a determined invalid result and a to-be-determined valid result; According to the type of the candidate result, determining the identification status of the data to be labeled; The training steps of the distribution algorithm model include: Determine the total number of manual identification rounds and the number of unique participants in each round; Count the cumulative number of participants and analyze the distribution of recognition results of the training data to be labeled; Determine the estimated accuracy of each distribution; The distribution algorithm model is determined in combination with the accuracy requirement and the estimated accuracy of each distribution situation.

Images