Crowdsourcing-based word segmentation annotation quality evaluation system

Abstract

The invention discloses a word segmentation annotation quality evaluation system based on crowdsourcing, relates to the field of annotation quality evaluation, and is used for solving the problem that quality identification is unclear due to the fact that submitted annotation results and corresponding types are deviated due to the fact that subjective understanding abilities of crowdsourcing users are different. The method comprises the following steps: collecting a text word segmentation confidence score, a standard similarity between a text corresponding type and other types, a user current-round labeling quantity difference and a user historical labeling accuracy difference value to establish a data analysis model, obtaining a deviation evaluation coefficient, comparing the deviation evaluation coefficient with a deviation threshold value, obtaining and counting a comparison result, and further determining the possibility that a labeling result is wrongly submitted; and a group of fuzzy rules are formulated for fuzzy reasoning according to the similarity of the categories which can be divided and the similarity difference between the category corresponding to the most similar labeling result and the most similar category, so that a division scheme of the re-division labeling result is determined, the blank that a current system cannot capture the subjective difference of the user is filled up, and the labeling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of annotation quality assessment, and more specifically, to a word segmentation annotation quality assessment system based on crowdsourcing. Background Art

[0002] With the development of information technology and the advent of the big data era, natural language processing (NLP) is increasingly being used in various fields. Word segmentation is a basic task in natural language processing (NLP). Since there are no obvious separators between words in Chinese text, the accuracy of word segmentation directly affects subsequent applications such as semantic understanding, information retrieval, and text analysis.

[0003] The prior art has the following deficiencies:

[0004] At present, multiple participants are allowed to segment and annotate multiple texts. Although the standard regulations for single-type text annotation in the crowdsourcing mode are clearly divided, due to the different types of annotated texts, the subjective understanding ability of crowdsourcing users varies, which leads to deviations between the submitted annotation results and the corresponding types. Common systems cannot identify the specific quality of such annotation results, and transmit them to reviewers for manual inspection through technologies such as data reflow. However, this not only wastes quality inspection time but also consumes manpower, affecting annotation efficiency and quality. Therefore, a word segmentation and annotation quality assessment system based on crowdsourcing is proposed.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a word segmentation and annotation quality assessment system based on crowdsourcing, which solves the problems raised in the above-mentioned background technology by using different product inspection methods.

[0007] To achieve the above object, the present invention provides the following technical solution: a crowdsourcing-based word segmentation and annotation quality assessment system, comprising a data acquisition module, a data processing module, a deviation analysis module and an annotation division module; signal connections between the modules;

[0008] The data collection module collects text feature information and user feature information, and through data processing, obtains the text segmentation confidence score, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and sends them to the data processing module;

[0009] The data processing module is used to obtain the text segmentation confidence score, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of user historical annotations, establish a data analysis model, obtain the deviation evaluation coefficient, and send it to the deviation analysis module;

[0010] The deviation analysis module is used to obtain the deviation evaluation coefficient and compare and analyze it with the preset deviation threshold, obtain and count the comparison results, calculate the weighted ratio of the sum of the deviation evaluation coefficient values ​​greater than or equal to the deviation threshold and the sum of the deviation evaluation coefficient values ​​less than the deviation threshold, further determine the possibility of incorrect submission of the annotation result, mark it as a re-divided annotation result, and send it to the annotation division module;

[0011] The annotation division module is used to determine the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category based on the obtained re-divided annotation results, and introduce fuzzy logic to determine the division scheme of the re-divided annotation results.

[0012] In a preferred embodiment, the text feature information includes the text segmentation confidence score and the standard similarity between the corresponding type of the text and other types, and the user feature information includes the difference in the number of annotations made by the user in this round and the difference in the accuracy of the user's historical annotations;

[0013] The confidence score of the text segmentation is calculated by subtracting the confidence score of the submitted type from the confidence score of the annotation standard of each category. ; Where i is the i-th comparison annotation standard;

[0014] By obtaining the annotation standards defined for each text type, the annotation standards and the annotation standards of the corresponding types of the annotation results submitted by the user are vectorized by combining word frequency and inverse document frequency to obtain the vocabulary vectors of each annotation standard, and the Jaccard similarity is used to obtain the similarity between the corresponding type of text and the other type standards. ;

[0015] The difference between the number of annotations of the corresponding type of the annotation result of the user in this round and the number of annotations of other types is calculated to obtain the difference in the number of annotations of the user in this round. ;

[0016] The user's historical annotation accuracy difference is obtained by calculating the difference between the user's historical average accuracy of the corresponding type of annotation and the historical average accuracy of the other types of annotation. .

[0017] In a preferred embodiment, the text segmentation confidence score is obtained , the similarity between the corresponding type of text and other types of standards , the difference in the number of annotations by users in this round And the difference in the accuracy of user historical annotations , generating the deviation assessment coefficient , according to the formula:

[0018] ;

[0019] like <0, ≥0, then , on the contrary, if ≥0, When <0, ;

[0020] In the formula, is the deviation assessment coefficient, , , as well as are the preset proportional coefficients for the confidence score of the text segmentation, the standard similarity between the corresponding type of the text and the other types, the difference in the number of annotations of the user in this round, and the difference in the accuracy of the user's historical annotations, respectively. , , as well as Both are greater than 0.

[0021] In a preferred embodiment, after obtaining the deviation assessment coefficient, the deviation assessment coefficient is compared and analyzed with the continuously iterated deviation threshold;

[0022] If the deviation evaluation coefficient is greater than or equal to the deviation threshold, the current annotation result is marked as a possible erroneous submission result, and a staggered signal is generated;

[0023] If the deviation evaluation coefficient is less than the deviation threshold, the current annotation result is marked as a possible correct submission result and a correct signal is generated.

[0024] In a preferred embodiment, the values ​​of the deviation evaluation coefficients greater than or equal to the deviation threshold are counted and integrated into a value set, which is expressed as follows: , the deviation evaluation coefficient is less than the deviation threshold value and is integrated into a numerical set, and its expression is , and bring it into the weighted ratio calculation, the specific formula is expressed as:

[0025] ;

[0026] In the formula, and are the weight coefficients of the numerical set whose deviation evaluation coefficient is greater than or equal to the deviation threshold and the numerical set whose deviation evaluation coefficient is less than the deviation threshold, respectively. is the value of the first deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the second deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the Ath deviation evaluation coefficient greater than or equal to the deviation threshold, A is the total number of values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the first deviation evaluation coefficient that is less than the deviation threshold, is the value of the second deviation evaluation coefficient that is less than the deviation threshold, is the value of the Bth deviation evaluation coefficient that is less than the deviation threshold, B is the total number of values ​​of the deviation evaluation coefficient that is less than the deviation threshold, and F is the possibility of the current annotation result.

[0027] In a preferred embodiment, the possibility of the current annotation result is compared with a preset possible threshold. If it is greater than or equal to the possible threshold, the current annotation result is defined as a re-divided annotation result, and the corresponding type of the current annotation result is eliminated. Conversely, if it is less than the possible threshold, the current annotation result is defined as a correct submission, and the current annotation result is used as the annotation standard of the corresponding type.

[0028] In a preferred embodiment, by bringing the re-divided annotation results into the annotation standards of each category for quality assessment, the categories with unqualified quality of the current annotation results are preferentially excluded to obtain the remaining divisible categories, and the annotation standards of the remaining divisible categories and the current annotation results are brought into the similarity calculation to obtain the divisible category similarity;

[0029] The annotation result most similar to the current annotation result is obtained by similarity calculation, and the annotation standard in the category corresponding to the annotation result most similar to the current annotation result is found and the difference between the similarity of the annotation standard that can be divided into categories is calculated to obtain the similarity difference between the category corresponding to the most similar annotation result and the most similar category.

[0030] In a preferred embodiment, the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category are defined as input variables, and are respectively divided into different fuzzy sets;

[0031] The division result of the re-divided annotation result is defined as the output variable and divided into fuzzy sets;

[0032] Formulate fuzzy rules to describe the impact of the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category on the division result of the re-divided annotation result;

[0033] Fuzzy reasoning is performed according to fuzzy rules to determine the division scheme for re-dividing the annotation results.

[0034] Technical effects and advantages of the present invention:

[0035] 1. The present invention establishes a data analysis model by collecting text segmentation confidence scores, standard similarities between the corresponding text type and other types, the difference in the number of users' annotations in this round, and the difference in the accuracy of users' historical annotations, obtains the deviation assessment coefficient and compares it with the deviation threshold, obtains and statistically compares the results, further determines the possibility of incorrect submission of the annotation results, marks them as re-divided annotation results, improves the annotation quality and efficiency, identifies the submission deviation of the annotation results, fills the gap that the current system cannot capture the subjective differences of users, and further reduces the risk of incorrect submission.

[0036] 2. The present invention determines the similarity of the divisible categories based on the obtained re-divided and labeled results, and the similarity difference between the category corresponding to the most similar labeled result and the most similar category, formulates a set of fuzzy rules for fuzzy reasoning, determines the division scheme of the re-divided and labeled results, reduces the quality assessment time, quickly determines that the fuzzy labeling results can accurately divide the corresponding categories, reduces the risk of misjudgment, and improves labeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The module diagram of a crowdsourcing-based word segmentation and annotation quality assessment system of the present invention is shown in FIG. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The present disclosure relates to a crowdsourcing-based word segmentation and annotation quality assessment. Currently, there are differences in annotation specifications and standards for various types of texts, especially for semantic texts, pathological medicine, and big data training annotations. Different annotation specifications have clear standards. However, in a crowdsourcing environment, users have different levels of text comprehension and are highly mobile. A single quality assessment based on text type may result in user submission errors leading to quality misjudgment. Therefore, a comprehensive analysis is performed through user feature information and its corresponding text annotation information to assess whether there is a possibility of misinterpretation of the annotation results submitted by the user.

[0040] Example 1

[0041] The present invention discloses a crowdsourcing-based word segmentation and annotation quality assessment system. Figure 1 As shown, it includes a data acquisition module, a data processing module, a deviation analysis module and a marking and division module; the signal connections between the modules;

[0042] The data collection module collects text feature information and user feature information, and through data processing, obtains the text segmentation confidence score, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and sends them to the data processing module;

[0043] For the definition of text, the user's annotation results are unqualified and unqualified. Usually, the common annotation quality assessment system will check the quality of the user's annotation under this type one by one according to the corresponding annotation standard (referred to as data return), set the annotation red line as a basis, and generate a return signal for the annotation result that violates the red line and send the annotation result to the user dialogue box. This is a common technical means and will not be elaborated here;

[0044] Among them, the text feature information includes the text segmentation confidence score and the standard similarity between the corresponding type of the text and the other types. The user feature information includes the difference in the number of annotations made by the user in this round and the difference in the accuracy of the user's historical annotations.

[0045] The text segmentation confidence score refers to an indicator used to evaluate the quality of text segmentation, reflecting the accuracy and reliability of the user when segmenting the text. The logic for obtaining the confidence score of the submitted type is to subtract the confidence score of the annotation standard of each category from the confidence score of the submitted type to obtain the text segmentation confidence score. ; Where i is the i-th comparison annotation standard;

[0046] Specifically, the confidence score is calculated by weighted calculation of word frequency, semantic similarity, historical annotation quality, and text complexity;

[0047] The word frequency is calculated by counting the frequency of each word in the training data. Specifically, word w appears n times in the training set, and the total number of words is N. Then the word frequency is expressed as ;

[0048] Among them, the semantic similarity is calculated by calculating the similarity between the current annotation result and the historical annotation result. Specifically, the vector of the current annotation result is represented by A, the vector of the historical annotation result is represented by H, and the semantic similarity score is S. The semantic similarity score is obtained by the cosine similarity formula, which is expressed as:

[0049] ;

[0050] In the formula, is the magnitude of vector A, is the modulus of vector H; obtained by the following formula:

[0051] as well as

[0052] Where b is the element index in vector A, z is the dimension of vector A, m is the dimension of vector H, and j is the element index in vector H;

[0053] Text complexity is expressed through lexical diversity, using the type-token ratio to measure the richness of unique vocabulary used in the text. Specifically, it is calculated by comparing the number of different words that appear in the text to the total number of all words in the text.

[0054] The quality of historical annotations is calculated by comparing the passing and failing annotation results of the system evaluation. Specifically, the results of the system evaluation and their passing or failing are defined by the experimenters based on the annotation standards and their corresponding scoring items, which will not be elaborated here.

[0055] The confidence score of the submitted type and the confidence score of the annotation standard of each category are combined to obtain the text segmentation confidence score. The specific formula is:

[0056] ;

[0057] In the formula, Give the submission type a confidence score, Confidence scores for annotation criteria for each category;

[0058] It should be noted that if Greater than , then the text segmentation confidence score is defined as 0, which means that the annotation standard of this category does not match the annotation result, and the user will not submit it to this type;

[0059] Generally, regarding crowdsourcing annotation standards, the system will publish them to each user's visualization port for users to view. Therefore, although the submission of user annotation results may have subjective differences, errors in large categories will not be submitted (large category submission errors belong to the red line range, which can be detected by traditional systems and evaluated as unqualified annotation results, so they are not considered). For example, large categories include pathology and physical experiments, etc. Pathology is divided into multiple subcategories, such as clinical trials, pathological principles, etc. Users may submit the annotation text of pathological principles to the annotation subcategory of clinical trials, but not to physical experiments, etc., which will not be elaborated here;

[0060] The similarity between the corresponding type of text and the other types of standards refers to the similarity between the annotation standards of the corresponding type of the annotation results submitted by the user and the annotation standards of the remaining other types. The acquisition logic is to obtain the annotation standards defined for each text type, vectorize the annotation standards and the annotation standards of the corresponding type of the annotation results submitted by the user by combining word frequency and inverse document frequency, and obtain the vocabulary vectors of each annotation standard. The Jaccard similarity is used to obtain the similarity between the corresponding type of text and the other types of standards. ;

[0061] Specifically, the word frequency and inverse document frequency are calculated using the following formula, and then vectorized by combining the word frequency and inverse document frequency:

[0062] as well as

[0063] In the formula, t is the word, d is the annotation standard, is the number of times word t appears in the annotation standard d, is the total number of words in the annotation standard d, is the frequency of occurrence of word t in the annotation standard; is the total number of marking standards, is the standard number of tags containing word t, is the rarity of word t in the annotation standard set;

[0064] Specifically, the combined formula is:

[0065] ;

[0066] In the formula, is the importance score of word t in the annotation standard d;

[0067] Furthermore, the Jaccard similarity is used to calculate the similarity between the corresponding type of text and the other types of standards:

[0068] ;

[0069] In the formula, is the vocabulary set of annotation standards corresponding to the type of the current annotation result, is the annotation standard vocabulary set for the i-th comparison type;

[0070] The logic for obtaining the difference in the number of annotations of the user in this round is to calculate the difference between the number of annotations of the corresponding type of the annotation result of the user in this round and the number of annotations of other types to obtain the difference in the number of annotations of the user in this round. Specifically, the number of annotations of the corresponding type of the annotation results submitted by the user (the annotation results and annotation types have been defined above) and the number of annotations of other types are counted to perform difference calculation;

[0071] It should be noted that if the calculated difference is ≥0, it means that the number of annotations of the corresponding type is greater than or equal to the number of annotations of other types, which means that the user is more familiar with the content of this type and the possibility of incorrect submission is reduced. On the contrary, if the calculated difference is <0, it means that the number of annotations of the corresponding type is less than the number of annotations of other types, which means that the user does not understand this type well and the possibility of incorrect submission is increased.

[0072] The logic for obtaining the difference in the accuracy of a user's historical annotations is to calculate the difference between the user's historical average accuracy of the corresponding type of annotations and the historical average accuracy of the other types of annotations to obtain the difference in the accuracy of the user's historical annotations. ;

[0073] It should be noted that if the calculated difference is ≥0, it means that the user's historical performance in the corresponding type is better than other types, indicating that the user has a better understanding of the annotation of this type and the possibility of incorrect submission is lower. Conversely, if the calculated difference is <0, it means that the user's historical performance in the corresponding type is lower than other types, which may mean that the user has problems or is unfamiliar with the annotation of this type and the possibility of incorrect submission is higher;

[0074] The data processing module is used to obtain the text segmentation confidence score, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of user historical annotations, establish a data analysis model, obtain the deviation evaluation coefficient, and send it to the deviation analysis module;

[0075] Among them, the data analysis model refers to a weighted analysis model, which generates a deviation assessment coefficient through weighted calculation;

[0076] Get the confidence score of text segmentation , the similarity between the corresponding type of text and other types of standards , the difference in the number of annotations by users in this round And the difference in the accuracy of user historical annotations , generating the deviation assessment coefficient , according to the formula:

[0077]

[0078] Furthermore, if <0, ≥0, then , on the contrary, if ≥0, When <0, ;

[0079] In the formula, is the deviation assessment coefficient, , , as well as are the preset proportional coefficients for the confidence score of the text segmentation, the standard similarity between the corresponding type of the text and the other types, the difference in the number of annotations of the user in this round, and the difference in the accuracy of the user's historical annotations, respectively. , , as well as All are greater than 0;

[0080] Among them, the confidence score of the text segmentation, the standard similarity between the corresponding type of the text and the other types, the difference in the number of annotations of the user in this round, and the difference in the accuracy of the user's historical annotations are all data-based manifestations that directly express the possibility of incorrect submission of the current annotation results;

[0081] From the above formula, we can see that when the similarity between the corresponding type of the text and the standards of other types is greater, it means that there is a possibility of confusion between the types, indicating that the annotation result is too close to the standards of other types, and the deviation evaluation coefficient is larger. Conversely, the greater the text segmentation confidence score, the difference in the number of user annotations in this round, and the accuracy of the user's historical annotations, the smaller the deviation evaluation coefficient.

[0082] The deviation analysis module is used to obtain the deviation evaluation coefficient and compare and analyze it with the preset deviation threshold, obtain and count the comparison results, calculate the weighted ratio of the sum of the deviation evaluation coefficient values ​​greater than or equal to the deviation threshold and the sum of the deviation evaluation coefficient values ​​less than the deviation threshold, further determine the possibility of incorrect submission of the annotation result, mark it as a re-divided annotation result, and send it to the annotation division module;

[0083] The logic for obtaining the deviation threshold is to collect the correct set of historical annotation results, including manual submission by users and submission by system data reflow, and then divide the data set into training set and test set, set the evaluation index and clustering algorithm, and in each round of cross-validation, train the model on the training set and evaluate the model performance on the test set. Then adjust the deviation threshold according to the performance of the validation set. Therefore, the deviation threshold is constantly updated.

[0084] In the present invention, clustering algorithm is a type of unsupervised learning algorithm used to divide data points in a data set into groups or clusters with similarities; a common one is K-means clustering, which divides the data points in a data set into K clusters so that the distance between the data point of each curve and the center point (centroid) of the cluster to which it belongs is minimized, and finally the effect of the adjusted deviation threshold is measured by the Euclidean distance, so as to set the deviation threshold;

[0085] After obtaining the deviation assessment coefficient, compare and analyze the deviation assessment coefficient with the continuously iterated deviation threshold;

[0086] If the deviation evaluation coefficient is greater than or equal to the deviation threshold, the current annotation result is marked as a possible erroneous submission result, and a staggered signal is generated;

[0087] If the deviation evaluation coefficient is less than the deviation threshold, the current annotation result is marked as a possible correct submission result, and a correct signal is generated;

[0088] The values ​​of the deviation evaluation coefficients greater than or equal to the deviation threshold are counted separately and integrated into a set of values, whose expression is: , the deviation evaluation coefficient is less than the deviation threshold value and is integrated into a numerical set, and its expression is , and bring it into the weighted ratio calculation, the specific formula is expressed as:

[0089] ;

[0090] In the formula, and are the weight coefficients of the numerical set whose deviation evaluation coefficient is greater than or equal to the deviation threshold and the numerical set whose deviation evaluation coefficient is less than the deviation threshold, respectively. is the value of the first deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the second deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the Ath deviation evaluation coefficient greater than or equal to the deviation threshold, A is the total number of values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the first deviation evaluation coefficient that is less than the deviation threshold, is the value of the second deviation evaluation coefficient that is less than the deviation threshold, is the value of the Bth deviation evaluation coefficient that is less than the deviation threshold, B is the total number of values ​​of the deviation evaluation coefficient that is less than the deviation threshold, and F is the possibility of the current annotation result;

[0091] Compare the possibility of the current annotation result with the preset possible threshold. If it is greater than or equal to the possible threshold, define the current annotation result as a reclassified annotation result, and remove the corresponding type of the current annotation result. Otherwise, if it is less than the possible threshold, define the current annotation result as a correct submission, and use the current annotation result as the annotation standard of the corresponding type.

[0092] Among them, the possible threshold is obtained by calculating the deviation assessment coefficient of this round and submitting the correct set of historical standard results, which will not be described here;

[0093] It should be noted that the reclassification of the annotation results indicates that the submission of the current annotation results is wrong, and it is necessary to perform data reflow and other operation processes to further determine the classification category of the annotation results; however, when reflowing data, the type corresponding to the current annotation result should be avoided to save the reflow time, so the corresponding type of the current annotation result is eliminated; correct submission indicates that the submission of the current annotation result is correct, and as the annotation standard of the corresponding type, the dynamic adjustment of the quality assessment is completed by dynamically adjusting the annotation standard. It can be understood that since the current annotation result does not have similarities with the standards of other types and cannot be completely consistent with the annotation standards of the corresponding type, it can be used as a basis for dynamically adjusting the quality assessment standard, etc., which will not be elaborated here;

[0094] The present invention establishes a data analysis model by collecting text segmentation confidence scores, standard similarities between corresponding text types and other types, differences in the number of annotations by users in this round, and differences in the accuracy of users' historical annotations, obtains a deviation assessment coefficient and compares it with a deviation threshold, obtains and statistically compares the results, further determines the possibility of incorrect submission of annotation results, marks them as re-divided annotation results, improves annotation quality and efficiency, identifies submission deviations of annotation results, fills the gap that the current system cannot capture subjective differences among users, and further reduces the risk of incorrect submission.

[0095] Example 2

[0096] In the embodiment 1 of the present invention, an example is given to illustrate that the present invention establishes a data analysis model by collecting text segmentation confidence scores, standard similarities between the corresponding type of the text and other types, the difference in the number of annotations of the user in this round, and the difference in the accuracy of the user's historical annotations, obtains a deviation assessment coefficient and compares it with a deviation threshold, obtains and statistically compares the results, further determines the possibility of incorrect submission of the annotation results, and marks it as an operation strategy for re-dividing the annotation results; however, in the embodiment 1, only how to detect the incorrect submission of the annotation results is discussed, and how to re-divide the annotation results is not discussed in detail. Obviously, although a census quality assessment can be performed on all types through data reflux, the focus is on consuming energy and increasing the quality assessment time, which reduces the annotation efficiency; in view of the above problems, the embodiment 2 of the present invention is further refined;

[0097] Among them, if an incorrect submission of the annotation result is detected, the user can be prompted in the dialog box to submit the error, but the annotation experience for the user is relatively poor, and the accuracy of the subsequent submission work will also be reduced. Obviously, the user can submit the annotation results arbitrarily and frequently to test the annotation results that meet the type, but this will increase the operating pressure of the system and further reduce the evaluation efficiency of the annotation quality;

[0098] The annotation division module is used to determine the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category according to the obtained re-divided annotation results, and bring in fuzzy logic to determine the division scheme of the re-divided annotation results;

[0099] The expression of the re-divided annotation results is the current annotation result described in Example 1;

[0100] Among them, the similarity of the divisible category refers to the similarity that the current annotation result can be classified into the category. By bringing the re-divided annotation result into the annotation standards of each category for quality assessment, the categories with unqualified quality of the current annotation results are preferentially excluded to obtain the remaining divisible categories. The annotation standards of the remaining divisible categories and the current annotation result are brought into the similarity calculation to obtain the divisible category similarity;

[0101] Specifically, the quality assessment system compares the quality of the current annotation result with the annotation standards of each category (such as text similarity or confidence score, etc.), and generates a failure signal for the current annotation result that fails the comparison, and marks the category as a category with unqualified quality of the current annotation result;

[0102] It should be noted that the definition of unqualified quality is set by the experimenters according to the specific labeling standards and qualified quality classification, and is not limited here;

[0103] The similarity difference between the category corresponding to the most similar annotation result and the most similar category refers to obtaining the annotation result most similar to the current annotation result through similarity calculation, finding the annotation standard in the category corresponding to the annotation result most similar to the current annotation result and performing difference calculation with the similarity of the annotation standard that can be divided into categories to obtain the similarity difference between the category corresponding to the most similar annotation result and the most similar category;

[0104] The similarity-related calculation steps and formulas have been described in Example 1 and will not be repeated here.

[0105] Specifically, the unilateral similarity calculation cannot determine which specific category the current annotation result should be classified into. By further calculating the similarity difference between the annotation standard of the category corresponding to the annotation result most similar to the current annotation result (the maximum value in the similarity calculation is taken as the most similar annotation result) and the annotation standard of the classifiable category, the fuzzy logic is used to determine the division scheme of the annotation result.

[0106] For example, "High", "Low", and "Medium" are for the similarity of the divisible categories, and "Large Discrepancy", "Small Difference", and "General Difference" are for the similarity difference between the category corresponding to the most similar annotation result and the most similar category;

[0107] Formulate a set of fuzzy rules to describe the impact of different input variables on output variables. The definition of rules can be based on professional knowledge or obtained through data analysis and experiments. For example:

[0108] The similarity of the divisible categories is marked as X, the similarity difference between the category corresponding to the most similar annotation result and the most similar category is marked as U, and the division result of the divided annotation result is marked as C_results;

[0109] Then we can define:

[0110] Rule 1: IF (X is High) AND (U is Large Discrepancy) THEN (C_resultsis High)

[0111] Rule 2: IF (U is Low) AND (U is General Difference) THEN (C_resultsis Low) ...

[0113] Perform fuzzy reasoning according to fuzzy rules to determine the division scheme for re-dividing the annotation results;

[0114] It should be noted that the division of fuzzy sets can be adjusted according to actual conditions. For example, although this embodiment takes three fuzzy sets as an example, the average similarity evaluation coefficient of the classifiable documents and the average similarity evaluation coefficient of the remaining classification items can actually be divided into more than three sets to facilitate better precise adjustment according to different similarities.

[0115] Furthermore, for the judgment of the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category, a threshold can be set according to the actual situation for judgment. For example, when the similarity of the divisible categories exceeds 69%, it is marked as "High"; when the similarity difference between the category corresponding to the most similar annotation result and the most similar category is higher than 61%, it is marked as "Large Discrepancy", and so on. I will not go into details here.

[0116] The present invention determines the similarity of the divisible categories based on the obtained re-divided and labeled results, and formulates a set of fuzzy rules for fuzzy reasoning between the category corresponding to the most similar labeled result and the similarity difference between the category and the most similar category, determines the division scheme of the re-divided and labeled results, reduces the quality assessment time, quickly determines whether the fuzzy labeling results can accurately divide the corresponding categories, reduces the risk of misjudgment, and improves labeling efficiency.

[0117] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0118] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0119] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0125] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A crowdsourcing-based word segmentation and annotation quality assessment system, characterized by: It includes data acquisition module, data processing module, deviation analysis module and marking and division module; signal connection between each module; The data collection module collects text feature information and user feature information, and through data processing, obtains the text segmentation confidence score, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and sends them to the data processing module; The data processing module is used to obtain the text segmentation confidence score, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of user historical annotations, establish a data analysis model, obtain the deviation evaluation coefficient, and send it to the deviation analysis module; The deviation analysis module is used to obtain the deviation evaluation coefficient and compare and analyze it with the preset deviation threshold, obtain and count the comparison results, calculate the weighted ratio of the sum of the deviation evaluation coefficient values ​​greater than or equal to the deviation threshold and the sum of the deviation evaluation coefficient values ​​less than the deviation threshold, further determine the possibility of incorrect submission of the annotation result, mark it as a re-divided annotation result, and send it to the annotation division module; The annotation division module is used to determine the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category based on the obtained re-divided annotation results, and introduce fuzzy logic to determine the division scheme of the re-divided annotation results.

2. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 1, characterized in that: The text feature information includes the confidence score of the text segmentation and the standard similarity between the corresponding type of the text and other types. The user feature information includes the difference in the number of annotations made by the user in this round and the difference in the accuracy of the user's historical annotations. The confidence score of the text segmentation is calculated by subtracting the confidence score of the submitted type from the confidence score of the annotation standard of each category. ; Where i is the i-th comparison annotation standard; By obtaining the annotation standards defined for each text type, the annotation standards and the annotation standards of the corresponding types of the annotation results submitted by the user are vectorized by combining word frequency and inverse document frequency to obtain the vocabulary vectors of each annotation standard, and the Jaccard similarity is used to obtain the similarity between the corresponding type of text and the other type standards. ; The difference between the number of annotations of the corresponding type of the annotation result of the user in this round and the number of annotations of other types is calculated to obtain the difference in the number of annotations of the user in this round. ; The user's historical annotation accuracy difference is obtained by calculating the difference between the user's historical average accuracy of the corresponding type of annotation and the historical average accuracy of the other types of annotation. .

3. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 2, characterized in that: Get text segmentation confidence score , the similarity between the corresponding type of text and other types of standards , the difference in the number of annotations by users in this round And the difference in the accuracy of user historical annotations , generating the deviation assessment coefficient , according to the formula: ； like <0, ≥0, then , on the contrary, if ≥0, When <0, ; In the formula, is the deviation assessment coefficient, , , as well as are the preset proportional coefficients for the confidence score of the text segmentation, the standard similarity between the corresponding type of the text and the other types, the difference in the number of annotations of the user in this round, and the difference in the accuracy of the user's historical annotations, respectively. , , as well as Both are greater than 0.

4. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 3, characterized in that: After obtaining the deviation assessment coefficient, compare and analyze the deviation assessment coefficient with the continuously iterated deviation threshold; If the deviation evaluation coefficient is greater than or equal to the deviation threshold, the current annotation result is marked as a possible erroneous submission result, and a staggered signal is generated; If the deviation evaluation coefficient is less than the deviation threshold, the current annotation result is marked as a possible correct submission result and a correct signal is generated.

5. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 4, characterized in that: The values ​​of the deviation evaluation coefficients greater than or equal to the deviation threshold are counted separately and integrated into a value set, whose expression is: , the deviation evaluation coefficient is less than the deviation threshold value and is integrated into a numerical set, and its expression is , and bring it into the weighted ratio calculation, the specific formula is expressed as: ； In the formula, and are the weight coefficients of the numerical set whose deviation evaluation coefficient is greater than or equal to the deviation threshold and the numerical set whose deviation evaluation coefficient is less than the deviation threshold, respectively. is the value of the first deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the second deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the Ath deviation evaluation coefficient greater than or equal to the deviation threshold, A is the total number of values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the first deviation evaluation coefficient that is less than the deviation threshold, is the value of the second deviation evaluation coefficient that is less than the deviation threshold, is the value of the Bth deviation evaluation coefficient that is less than the deviation threshold, B is the total number of values ​​of the deviation evaluation coefficient that is less than the deviation threshold, and F is the possibility of the current annotation result.

6. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 5, characterized in that: The possibility of the current annotation result is compared with the preset possible threshold. If it is greater than or equal to the possible threshold, the current annotation result is defined as a re-divided annotation result, and the corresponding type of the current annotation result is eliminated. Otherwise, if it is less than the possible threshold, the current annotation result is defined as a correct submission, and the current annotation result is used as the annotation standard for the corresponding type.

7. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 6, characterized in that: By bringing the re-divided annotation results into the annotation standards of each category for quality assessment, the categories with unqualified quality of the current annotation results are preferentially excluded to obtain the remaining divisible categories, and the annotation standards of the remaining divisible categories and the current annotation results are brought into the similarity calculation to obtain the divisible category similarity; The annotation result most similar to the current annotation result is obtained by similarity calculation, and the annotation standard in the category corresponding to the annotation result most similar to the current annotation result is found and the difference between the similarity of the annotation standard that can be divided into categories is calculated to obtain the similarity difference between the category corresponding to the most similar annotation result and the most similar category.

8. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 7, characterized in that: The similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category are defined as input variables, and are divided into different fuzzy sets respectively; The division result of the re-divided annotation result is defined as the output variable and divided into fuzzy sets; Formulate fuzzy rules to describe the impact of the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category on the division result of the re-divided annotation result; Fuzzy reasoning is performed according to fuzzy rules to determine the division scheme for re-dividing the annotation results.

Images