A word segmentation and annotation quality assessment system based on crowdsourcing

Abstract

The present invention discloses a word segmentation and annotation quality assessment system based on crowdsourcing, which relates to the field of annotation quality assessment and is used to solve the problem that the subjective understanding ability of crowdsourcing users varies, resulting in deviations between submitted annotation results and corresponding types, making quality identification unclear. The system collects text word segmentation confidence scores, standard similarities between text corresponding types and other types, the difference in the number of users' annotations in this round, and the difference in the accuracy of users' historical annotations to establish a data analysis model, obtains a deviation assessment coefficient and compares it with a deviation threshold, obtains and statistically compares the comparison results, further determines the possibility of incorrect submission of annotation results, marks them as re-divided annotation results, formulates a set of fuzzy rules for fuzzy reasoning based on the similarity of categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category, and determines a division scheme for re-dividing annotation results, thereby filling the gap that the current system cannot capture the subjective differences of users and improving annotation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of annotation quality assessment, and more particularly to a word segmentation and annotation quality assessment system based on crowdsourcing. Background Art

[0002] With the development of information technology and the advent of the big data era, natural language processing (NLP) is increasingly being applied across various fields. Word segmentation is a fundamental task in NLP. Because Chinese text lacks clear word separators, the accuracy of word segmentation directly impacts subsequent applications such as semantic understanding, information retrieval, and text analysis.

[0003] The existing technology has the following deficiencies:

[0004] Currently, multiple participants are allowed to segment and annotate multiple texts. Although the crowdsourcing model clearly defines the standards for single-type text annotation, the subjective understanding of crowdsourcing users varies depending on the type of text being annotated. This can lead to deviations between the submitted annotation results and the corresponding types. Conventional systems are unable to identify the specific quality of such annotation results and instead transmit them to reviewers for manual inspection through technologies such as data reflow. However, this not only wastes quality inspection time and manpower, but also affects annotation efficiency and quality. Therefore, a crowdsourcing-based word segmentation and annotation quality assessment system is proposed.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a crowdsourcing-based word segmentation and annotation quality assessment system, which solves the problems raised in the above-mentioned background technology by applying different product inspection methods.

[0007] To achieve the above object, the present invention provides the following technical solution: a crowdsourcing-based word segmentation and annotation quality assessment system, comprising a data acquisition module, a data processing module, a deviation analysis module, and an annotation division module; signal connections between the modules;

[0008] The data collection module collects text feature information and user feature information, and through data processing, obtains the text segmentation confidence score, the similarity between the corresponding text type and the other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and sends it to the data processing module;

[0009] The data processing module is used to obtain the text segmentation confidence score, the similarity between the corresponding text type and the other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, establish a data analysis model, obtain the deviation evaluation coefficient, and send it to the deviation analysis module;

[0010] The deviation analysis module is used to obtain the deviation evaluation coefficient and compare it with the preset deviation threshold to obtain and statistically analyze the comparison results. The weighted ratio calculation is performed on the sum of the deviation evaluation coefficient values ​​greater than or equal to the deviation threshold and the sum of the deviation evaluation coefficient values ​​less than the deviation threshold to further determine the possibility of incorrect submission of the annotation results, mark them as re-divided annotation results, and send them to the annotation division module;

[0011] The annotation division module is used to determine the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category based on the obtained re-divided annotation results, and introduce fuzzy logic to determine the division scheme of the re-divided annotation results.

[0012] In a preferred embodiment, the text feature information includes the text segmentation confidence score and the similarity between the corresponding text type and other types of standards. The user feature information includes the difference in the number of annotations made by the user in this round and the difference in the accuracy of the user's historical annotations.

[0013] The confidence score of the text segmentation is calculated by subtracting the confidence score of the submitted type from the confidence score of the annotation standard of each category. ; Where i is the i-th comparison annotation standard;

[0014] By obtaining the annotation standards defined for each text type, the annotation standards and the annotation standards of the corresponding types of the annotation results submitted by the user are vectorized by combining word frequency and inverse document frequency to obtain the vocabulary vectors of each annotation standard, and the Jaccard similarity is used to obtain the similarity between the corresponding type of text and the other type standards. ;

[0015] The difference between the number of annotations of the corresponding type of the annotation result of the user in this round and the number of annotations of other types is calculated to obtain the difference in the number of annotations of the user in this round. ;

[0016] The user's historical annotation accuracy difference is obtained by calculating the difference between the user's historical average accuracy of the corresponding type of annotation and the historical average accuracy of the other types of annotation. .

[0017] In a preferred embodiment, the confidence score of the text segmentation is obtained , the similarity between the text type and other types of standards , the difference in the number of user annotations in this round And the difference in user historical annotation accuracy , generating the deviation assessment coefficient , according to the formula:

[0018] ;

[0019] like <0, ≥0, then , on the contrary, if ≥0, When <0, ;

[0020] Where, is the deviation assessment coefficient, 、 、 as well as The preset proportional coefficients are respectively the confidence score of the text segmentation, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and 、 、 as well as Both are greater than 0.

[0021] In a preferred embodiment, after obtaining the deviation evaluation coefficient, the deviation evaluation coefficient is compared and analyzed with the continuously iterated deviation threshold;

[0022] If the deviation evaluation coefficient is greater than or equal to the deviation threshold, the current annotation result is marked as a possible erroneous submission result and a staggered signal is generated;

[0023] If the deviation evaluation coefficient is less than the deviation threshold, the current annotation result is marked as a possible correct submission result and a correct signal is generated.

[0024] In a preferred embodiment, the values ​​of the deviation evaluation coefficients greater than or equal to the deviation threshold are counted and integrated into a value set, which is expressed as follows: , the deviation evaluation coefficient is less than the deviation threshold value and is integrated into a numerical set, which is expressed as , and bring it into the weighted ratio calculation, the specific formula is expressed as:

[0025] ;

[0026] Where, and are the weight coefficients of the numerical set whose deviation evaluation coefficient is greater than or equal to the deviation threshold and the numerical set whose deviation evaluation coefficient is less than the deviation threshold, respectively. is the value of the first deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the second deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the Ath deviation evaluation coefficient greater than or equal to the deviation threshold, A is the total number of values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the first deviation evaluation coefficient that is less than the deviation threshold, is the value of the second deviation evaluation coefficient less than the deviation threshold, is the value of the Bth deviation evaluation coefficient that is less than the deviation threshold, B is the total number of values ​​of the deviation evaluation coefficient that are less than the deviation threshold, and F is the probability of the current annotation result.

[0027] In a preferred embodiment, the possibility of the current annotation result is compared with a preset possibility threshold. If it is greater than or equal to the possibility threshold, the current annotation result is defined as a re-divided annotation result, and the corresponding type of the current annotation result is eliminated. Conversely, if it is less than the possibility threshold, the current annotation result is defined as a correct submission, and the current annotation result is used as the annotation standard for the corresponding type.

[0028] In a preferred embodiment, the reclassified annotation results are subjected to the annotation standards of each category for quality assessment, and the categories with unqualified quality of the current annotation results are preferentially excluded to obtain the remaining divisible categories. The annotation standards of the remaining divisible categories and the current annotation results are subjected to similarity calculation to obtain the divisible category similarity.

[0029] The annotation result that is most similar to the current annotation result is obtained by similarity calculation, and the annotation standard in the category corresponding to the annotation result that is most similar to the current annotation result is found and the difference between the similarity of the annotation standard that can be divided into categories is calculated to obtain the similarity difference between the category corresponding to the most similar annotation result and the most similar category.

[0030] In a preferred embodiment, the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category are defined as input variables and divided into different fuzzy sets respectively;

[0031] The division result of the re-divided annotation result is defined as the output variable and divided into fuzzy sets;

[0032] Formulate fuzzy rules to describe the impact of the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category on the division result of the re-divided annotation result;

[0033] Fuzzy reasoning is performed based on fuzzy rules to determine the division scheme for re-dividing the annotation results.

[0034] Technical effects and advantages of the present invention:

[0035] 1. The present invention establishes a data analysis model by collecting text segmentation confidence scores, standard similarities between the corresponding text type and other types, the difference in the number of annotations by users in this round, and the difference in the accuracy of users' historical annotations, obtains a deviation assessment coefficient and compares it with the deviation threshold, obtains and statistically compares the results, further determines the possibility of incorrect submission of annotation results, marks them as re-classified annotation results, improves annotation quality and efficiency, identifies submission deviations of annotation results, fills the gap that the current system cannot capture user subjective differences, and further reduces the risk of incorrect submission.

[0036] 2. The present invention determines the similarity of the divisible categories based on the obtained re-divided labeling results and the similarity difference between the category corresponding to the most similar labeling result and the most similar category, formulates a set of fuzzy rules for fuzzy reasoning, determines the division scheme of the re-divided labeling results, reduces the quality assessment time, quickly determines whether the fuzzy labeling results can accurately divide the corresponding categories, reduces the risk of misjudgment, and improves labeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a module diagram of a crowdsourcing-based word segmentation and annotation quality assessment system of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] The present disclosure relates to crowdsourcing-based word segmentation and annotation quality assessment. Currently, the annotation specifications and standards for various types of text vary depending on the text type. In particular, different annotation specifications for semantic text, pathological medicine, and big data training annotation have clear standards. However, in a crowdsourcing environment, users' text comprehension levels vary and are highly fluid. A single quality assessment based on text type may lead to misjudgment of quality due to user submission errors. Therefore, a comprehensive analysis of user feature information and its corresponding text annotation information is conducted to assess whether there is a possibility of misinterpretation in the annotation results submitted by users.

[0040] Example 1

[0041] The present invention discloses a word segmentation and annotation quality assessment system based on crowdsourcing, such as Figure 1 As shown, it includes data acquisition module, data processing module, deviation analysis module and marking and division module; signal connection between each module;

[0042] The data collection module collects text feature information and user feature information, and through data processing, obtains the text segmentation confidence score, the similarity between the corresponding text type and the other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and sends it to the data processing module;

[0043] For text, the user's annotation results are defined as unqualified or unqualified. Generally, common annotation quality assessment systems will check the quality of user annotations of this type one by one according to the corresponding annotation standards (referred to as data reflux), set annotation red lines as a basis, and generate a rejection signal for annotation results that violate the red lines and send the annotation results to the user's dialogue box. This is a common technical method and will not be elaborated here.

[0044] Among them, text feature information includes the text segmentation confidence score and the similarity between the corresponding text type and other types of standards. User feature information includes the difference in the number of annotations made by the user in this round and the difference in the accuracy of the user's historical annotations.

[0045] The text segmentation confidence score refers to an indicator used to evaluate the quality of text segmentation, reflecting the accuracy and reliability of the user's text segmentation. Its acquisition logic is to obtain the text segmentation confidence score by subtracting the confidence score of the submitted type from the confidence score of the annotation standard of each category. ; Where i is the i-th comparison annotation standard;

[0046] Specifically, the confidence score is calculated by weighted calculation of word frequency, semantic similarity, historical annotation quality, and text complexity;

[0047] The word frequency is calculated by counting the frequency of each word in the training data. Specifically, word w appears n times in the training set, and the total number of words is N. The word frequency is expressed as ;

[0048] Among them, semantic similarity is achieved by calculating the similarity between the current annotation result and the historical annotation result. Specifically, the vector of the current annotation result is represented by A, the vector of the historical annotation result is represented by H, and the semantic similarity score is S. The semantic similarity score is obtained by the cosine similarity formula, which is expressed as:

[0049] ;

[0050] Where, is the modulus of vector A, is the modulus of vector H; it is obtained by the following formula:

[0051] as well as

[0052] Where b is the element index in vector A, z is the dimension of vector A, m is the dimension of vector H, and j is the element index in vector H;

[0053] Text complexity is expressed through lexical diversity, which uses the type-token ratio to measure the richness of unique vocabulary used in the text. Specifically, it is calculated by counting the number of different words that appear in the text and the total number of all words in the text.

[0054] The quality of historical annotations is calculated by comparing the passing and failing annotation results of the system evaluation. Specifically, the results of the system evaluation and the passing or failing scores are defined by our researchers based on the annotation standards and their corresponding scoring items, which will not be detailed here.

[0055] The confidence score of the submitted type and the confidence score of the annotation standard of each category are combined to obtain the text segmentation confidence score. The specific formula is:

[0056] ;

[0057] Where, Score the confidence level of the submission type, Confidence scores for the annotation criteria for each category;

[0058] It should be noted that if Greater than , then the text segmentation confidence score is defined as 0, which means that the annotation standard of this category does not match the annotation result, and the user will not submit it to this type;

[0059] Typically, regarding crowdsourcing annotation standards, the system publishes the results to each user's visualization portal for review. Therefore, although user annotation submissions may be subjective, errors in large categories will not be submitted (large category submission errors fall within the red line range, which traditional systems can detect and evaluate as failed annotation results, so they are not considered). For example, large categories include pathology and physical experiments, and pathology is divided into multiple subcategories, such as clinical trials and pathology principles. A user may submit annotations for pathology principles to the clinical trials annotation subcategory, but not to physical experiments, etc., which will not be discussed here.

[0060] The similarity between the corresponding type of text and the rest of the standard types refers to the similarity between the standard of the corresponding type of the annotation result submitted by the user and the standard of the remaining other types. The acquisition logic is to obtain the annotation standard defined for each text type, and vectorize the annotation standard and the annotation standard of the corresponding type of the annotation result submitted by the user by combining word frequency and inverse document frequency to obtain the vocabulary vector of each annotation standard. The Jaccard similarity is used to obtain the similarity between the corresponding type of text and the rest of the standard types. ;

[0061] Specifically, the word frequency and inverse document frequency are calculated using the following formula, and then vectorized by combining the word frequency and inverse document frequency:

[0062] as well as

[0063] In the formula, t is the word, d is the annotation standard, is the number of times word t appears in the annotation standard d, is the total number of words in the annotation standard d, is the frequency of occurrence of word t in the annotation standard; is the total number of marking standards, is the number of annotation standards containing word t, is the rarity of word t in the annotation standard set;

[0064] Specifically, the combined formula is:

[0065] ;

[0066] Where, is the importance score of word t in the annotation standard d;

[0067] Furthermore, Jaccard similarity is used to calculate the similarity between the corresponding type of text and the other types of standards:

[0068] ;

[0069] Where, A vocabulary set of annotation standards corresponding to the type of the current annotation result. is the set of standard annotation vocabulary for the i-th comparison type;

[0070] The logic for obtaining the difference in the number of annotations of the user in this round is to calculate the difference between the number of annotations of the corresponding type of the annotation result of the user in this round and the number of annotations of other types. Specifically, the number of annotations of the corresponding type of annotation results submitted by users (the annotation results and annotation types have been defined above) and the number of annotations of other types are counted to perform difference calculation;

[0071] It should be noted that if the calculated difference is ≥0, it means that the number of annotations of the corresponding type is greater than or equal to the number of annotations of other types, indicating that the user is more familiar with the content of this type and the possibility of incorrect submission is reduced. Conversely, if the calculated difference is <0, it means that the number of annotations of the corresponding type is less than the number of annotations of other types, indicating that the user has insufficient understanding of this type and the possibility of incorrect submission is increased.

[0072] The logic for obtaining the difference in the accuracy of user historical annotations is to calculate the difference between the historical average accuracy of the user's annotations of the corresponding type and the historical average accuracy of the annotations of other types to obtain the difference in the accuracy of the user's historical annotations. ;

[0073] It should be noted that if the calculated difference is ≥0, it means that the user's historical performance in the corresponding type is better than other types, indicating that they have a better understanding of the annotation of this type and are less likely to submit errors. Conversely, if the calculated difference is <0, it means that the user's historical performance in the corresponding type is lower than other types, which may mean that the user has problems or is unfamiliar with the annotation of this type and is more likely to submit errors.

[0074] The data processing module is used to obtain the text segmentation confidence score, the similarity between the corresponding text type and the other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, establish a data analysis model, obtain the deviation evaluation coefficient, and send it to the deviation analysis module;

[0075] The data analysis model refers to a weighted analysis model, which generates a deviation assessment coefficient through weighted calculation;

[0076] Get text segmentation confidence score , the similarity between the text type and other types of standards , the difference in the number of user annotations in this round And the difference in user historical annotation accuracy , generating the deviation assessment coefficient , according to the formula:

[0077]

[0078] Furthermore, if <0, ≥0, then , on the contrary, if ≥0, When <0, ;

[0079] Where, is the deviation assessment coefficient, 、 、 as well as The preset proportional coefficients are respectively the confidence score of the text segmentation, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and 、 、 as well as All greater than 0;

[0080] Among them, the confidence score of the text segmentation, the standard similarity between the corresponding text type and other types, the difference in the number of annotations by the user in this round, and the difference in the accuracy of the user's historical annotations are all data-based manifestations that directly express the possibility of incorrect submission of the current annotation result;

[0081] From the above formula, we can see that when the similarity between the corresponding type of text and the standards of other types is greater, it means that there is a possibility of confusion between the types, indicating that the annotation results are too close to the standards of other types, and the deviation evaluation coefficient is larger. Conversely, the greater the text word segmentation confidence score, the difference in the number of user annotations in this round, and the accuracy of the user's historical annotations, the smaller the deviation evaluation coefficient.

[0082] The deviation analysis module is used to obtain the deviation evaluation coefficient and compare it with the preset deviation threshold to obtain and statistically analyze the comparison results. The weighted ratio calculation is performed on the sum of the deviation evaluation coefficient values ​​greater than or equal to the deviation threshold and the sum of the deviation evaluation coefficient values ​​less than the deviation threshold to further determine the possibility of incorrect submission of the annotation results, mark them as re-divided annotation results, and send them to the annotation division module;

[0083] The logic for obtaining the deviation threshold is to collect the correct set of historical annotation results, including manual submissions by users and submissions from system data reflow. The dataset is then divided into training and test sets. Evaluation indicators and clustering algorithms are set. In each round of cross-validation, the model is trained on the training set and its performance is evaluated on the test set. The deviation threshold is then adjusted based on the performance of the validation set. Therefore, the deviation threshold is continuously updated.

[0084] In the present invention, clustering algorithm is a type of unsupervised learning algorithm used to divide data points in a dataset into groups or clusters with similarities. A common example is K-means clustering, which divides the data points in a dataset into K clusters so that the distance between each curve data point and the center point (centroid) of its cluster is minimized. Finally, the effect of the adjusted deviation threshold is measured by the Euclidean distance, thereby setting the deviation threshold.

[0085] After obtaining the deviation assessment coefficient, the deviation assessment coefficient is compared and analyzed with the continuously iterated deviation threshold;

[0086] If the deviation evaluation coefficient is greater than or equal to the deviation threshold, the current annotation result is marked as a possible erroneous submission result and a staggered signal is generated;

[0087] If the deviation evaluation coefficient is less than the deviation threshold, the current annotation result is marked as a possible correct submission result and a correct signal is generated;

[0088] The values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold are counted separately and integrated into a value set, which is expressed as follows: , the deviation evaluation coefficient is less than the deviation threshold value and is integrated into a numerical set, which is expressed as , and bring it into the weighted ratio calculation, the specific formula is expressed as:

[0089] ;

[0090] Where, and are the weight coefficients of the numerical set whose deviation evaluation coefficient is greater than or equal to the deviation threshold and the numerical set whose deviation evaluation coefficient is less than the deviation threshold, respectively. is the value of the first deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the second deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the Ath deviation evaluation coefficient greater than or equal to the deviation threshold, A is the total number of values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the first deviation evaluation coefficient that is less than the deviation threshold, is the value of the second deviation evaluation coefficient less than the deviation threshold, is the value of the Bth deviation evaluation coefficient less than the deviation threshold, B is the total number of values ​​of the deviation evaluation coefficient less than the deviation threshold, and F is the probability of the current annotation result;

[0091] Compare the probability of the current annotation result with the preset possibility threshold. If it is greater than or equal to the possibility threshold, the current annotation result is defined as a reclassified annotation result, and the corresponding type of the current annotation result is eliminated. On the contrary, if it is less than the possibility threshold, the current annotation result is defined as correctly submitted, and the current annotation result is used as the annotation standard for the corresponding type.

[0092] Among them, the possible threshold is obtained by calculating the deviation assessment coefficient of this round and submitting the correct set of historical standard results, which will not be described in detail here;

[0093] It should be noted that reclassifying the annotation results means that the submission of the current annotation results is wrong, and data reflow and other operation processes are required to further determine the classification category of the annotation results; however, when reflowing data, the type corresponding to the current annotation result should be avoided to save reflow time, so the corresponding type of the current annotation result is eliminated; correct submission means that the submission of the current annotation result is correct, and it is used as the annotation standard of the corresponding type to dynamically adjust the annotation standard to complete the dynamic adjustment of the quality assessment. It can be understood that since the current annotation result has no similarities with the standards of other types and cannot be completely consistent with the annotation standard of the corresponding type, it can be used as the basis for dynamically adjusting the quality assessment standard, etc., which will not be elaborated here;

[0094] The present invention establishes a data analysis model by collecting text segmentation confidence scores, standard similarities between the corresponding text type and other types, the difference in the number of users' annotations in this round, and the difference in the accuracy of users' historical annotations, obtains a deviation assessment coefficient and compares it with the deviation threshold, obtains and statistically compares the results, further determines the possibility of incorrect submission of annotation results, marks them as re-classified annotation results, improves annotation quality and efficiency, identifies submission deviations of annotation results, fills the gap that the current system cannot capture user subjective differences, and further reduces the risk of incorrect submission.

[0095] Example 2

[0096] In Example 1 of the present invention, the present invention focuses on an example of establishing a data analysis model by collecting text segmentation confidence scores, standard similarities between the corresponding text type and other types, the difference in the number of annotations by users in this round, and the difference in the accuracy of users' historical annotations, obtaining a deviation assessment coefficient and comparing it with a deviation threshold, obtaining and statistically analyzing the comparison results, further determining the possibility of incorrect submission of annotation results, and marking it as an operational strategy for re-classifying the annotation results; however, in Example 1, only how to detect incorrect submission of annotation results is discussed, and how to re-classify the annotation results is not discussed in detail. Obviously, although a census quality assessment of all types can be performed through data reflow, the focus is on energy consumption and increased quality assessment time, which reduces annotation efficiency; to address the above issues, Example 2 of the present invention further refines the above issues;

[0097] Among them, if an incorrect annotation result is submitted, the user can be prompted in the dialog box to submit an error. However, this will provide a relatively poor annotation experience for the user and reduce the accuracy of subsequent submissions. Obviously, users can submit annotation results arbitrarily and frequently to try out annotation results that meet the type. However, this will increase the operating pressure of the system and further reduce the efficiency of annotation quality assessment.

[0098] The annotation division module is used to determine the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category based on the obtained re-divided annotation results, and to introduce fuzzy logic to determine the division scheme of the re-divided annotation results;

[0099] The expression of the re-divided annotation results is the current annotation result described in Example 1;

[0100] Among them, the similarity of the divisible category refers to the similarity of the current annotation result being able to be classified into the category. By bringing the re-divided annotation result into the annotation standards of each category for quality assessment, the categories with unqualified quality of the current annotation result are preferentially excluded to obtain the remaining divisible categories. The annotation standards of the remaining divisible categories and the current annotation result are brought into the similarity calculation to obtain the divisible category similarity;

[0101] Specifically, the quality assessment system compares the current annotation results with the annotation standards of each category (such as text similarity or confidence score). If the current annotation results fail the comparison, a failure signal is generated and the category is marked as a category with unqualified quality of the current annotation results.

[0102] It should be noted that the definition of unqualified quality is set by the experimenters based on the specific labeling standards and qualified quality classification, and is not limited here;

[0103] The similarity difference between the category corresponding to the most similar annotation result and the most similar category refers to the similarity calculation to obtain the annotation result that is most similar to the current annotation result, find the annotation standard in the category corresponding to the annotation result that is most similar to the current annotation result, and perform the difference calculation with the similarity of the annotation standard that can be divided into categories to obtain the similarity difference between the category corresponding to the most similar annotation result and the most similar category;

[0104] The similarity calculation steps and formulas have been described in Example 1 and will not be repeated here.

[0105] Specifically, unilateral similarity calculation cannot determine which specific category the current annotation result should be classified into. By further calculating the similarity difference between the annotation standard of the category corresponding to the annotation result that is most similar to the current annotation result (the maximum value in the similarity calculation is regarded as the most similar annotation result) and the annotation standard of the class that can be divided, the fuzzy logic is used to determine the division scheme of the annotation results.

[0106] For example, "High", "Low", and "Medium" are for the similarity of the divisible categories, "Large Discrepancy", "Small Difference", and "General Difference" are for the similarity difference between the category corresponding to the most similar annotation result and the most similar category;

[0107] Formulate a set of fuzzy rules to describe the impact of different input variables on the output variable. The definition of rules can be based on professional knowledge or obtained through data analysis and experiments. For example:

[0108] The similarity of the divisible categories is marked as X, the similarity difference between the category corresponding to the most similar annotation result and the most similar category is marked as U, and the division result of the divided annotation results is marked as C_results;

[0109] Then we can define:

[0110] Rule 1: IF (X is High) AND (U is Large Discrepancy) THEN (C_resultsis High)

[0111] Rule 2: IF (U is Low) AND (U is General Difference) THEN (C_resultsis Low) ...

[0113] Perform fuzzy reasoning based on fuzzy rules to determine the division scheme for re-dividing the annotation results;

[0114] It should be noted that the division of fuzzy sets can be adjusted according to actual conditions. For example, although this embodiment takes three fuzzy sets as an example, the average similarity evaluation coefficients of the classifiable documents and the average similarity evaluation coefficients of the remaining classification items can actually be divided into more than three sets to facilitate better precise adjustment according to different similarities.

[0115] Furthermore, for the judgment of the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category, a threshold can be set according to the actual situation for judgment. For example, when the similarity of the divisible categories exceeds 69%, it is marked as "High", and when the similarity difference between the category corresponding to the most similar annotation result and the most similar category is higher than 61%, it is marked as "Large Discrepancy", and so on. I will not go into details here.

[0116] The present invention determines the similarity of the divisible categories based on the obtained re-divided labeling results and the similarity difference between the category corresponding to the most similar labeling result and the most similar category, formulates a set of fuzzy rules for fuzzy reasoning, determines the division scheme of the re-divided labeling results, reduces the quality assessment time, quickly determines whether the fuzzy labeling results can accurately divide the corresponding categories, reduces the risk of misjudgment, and improves labeling efficiency.

[0117] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0118] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0119] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0125] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A crowdsourcing-based word segmentation and annotation quality assessment system, characterized by: It includes data acquisition module, data processing module, deviation analysis module and marking and division module; signal connection between each module; The data collection module collects text feature information and user feature information, and through data processing, obtains the text segmentation confidence score, the similarity between the corresponding text type and the other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and sends it to the data processing module; The data processing module is used to obtain the text segmentation confidence score, the similarity between the corresponding text type and the other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, establish a data analysis model, obtain the deviation evaluation coefficient, and send it to the deviation analysis module; The deviation analysis module is used to obtain the deviation evaluation coefficient and compare it with the preset deviation threshold to obtain and statistically analyze the comparison results. The weighted ratio calculation is performed on the sum of the deviation evaluation coefficient values ​​greater than or equal to the deviation threshold and the sum of the deviation evaluation coefficient values ​​less than the deviation threshold to further determine the possibility of incorrect submission of the annotation results, mark them as re-divided annotation results, and send them to the annotation division module; The annotation division module is used to determine the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category based on the obtained re-divided annotation results, and to introduce fuzzy logic to determine the division scheme of the re-divided annotation results; Text feature information includes the confidence score of the text segmentation and the similarity between the corresponding text type and other types. User feature information includes the difference in the number of annotations made by the user in this round and the difference in the accuracy of the user's historical annotations. The confidence score of the text segmentation is calculated by subtracting the confidence score of the submitted type from the confidence score of the annotation standard of each category. ; Where i is the i-th comparison annotation standard; By obtaining the annotation standards defined for each text type, the annotation standards and the annotation standards of the corresponding types of the annotation results submitted by the user are vectorized by combining word frequency and inverse document frequency to obtain the vocabulary vectors of each annotation standard, and the Jaccard similarity is used to obtain the similarity between the corresponding type of text and the other type standards. ; The difference between the number of annotations of the corresponding type of the annotation result of the user in this round and the number of annotations of other types is calculated to obtain the difference in the number of annotations of the user in this round. ; The user's historical annotation accuracy difference is obtained by calculating the difference between the user's historical average accuracy of the corresponding type of annotation and the historical average accuracy of the other types of annotation. ; Get text segmentation confidence score , the similarity between the text type and other types of standards , the difference in the number of user annotations in this round And the difference in user historical annotation accuracy , generating the deviation assessment coefficient , according to the formula: ； like <0, ≥0, then , on the contrary, if ≥0, When <0, ; Where, is the deviation assessment coefficient, 、 、 as well as The preset proportional coefficients are respectively the confidence score of the text segmentation, the standard similarity between the corresponding text type and other types, the difference in the number of user annotations in this round, and the difference in the accuracy of the user's historical annotations, and 、 、 as well as Both are greater than 0.

2. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 1, characterized in that: After obtaining the deviation assessment coefficient, the deviation assessment coefficient is compared and analyzed with the continuously iterated deviation threshold; If the deviation evaluation coefficient is greater than or equal to the deviation threshold, the current annotation result is marked as a possible erroneous submission result and a staggered signal is generated; If the deviation evaluation coefficient is less than the deviation threshold, the current annotation result is marked as a possible correct submission result and a correct signal is generated.

3. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 2, characterized in that: The values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold are counted separately and integrated into a value set, which is expressed as follows: , the deviation evaluation coefficient is less than the deviation threshold value and is integrated into a numerical set, which is expressed as , and bring it into the weighted ratio calculation, the specific formula is expressed as: ； Where, and are the weight coefficients of the numerical set whose deviation evaluation coefficient is greater than or equal to the deviation threshold and the numerical set whose deviation evaluation coefficient is less than the deviation threshold, respectively. is the value of the first deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the second deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the Ath deviation evaluation coefficient greater than or equal to the deviation threshold, A is the total number of values ​​of the deviation evaluation coefficient greater than or equal to the deviation threshold, is the value of the first deviation evaluation coefficient that is less than the deviation threshold, is the value of the second deviation evaluation coefficient less than the deviation threshold, is the value of the Bth deviation evaluation coefficient that is less than the deviation threshold, B is the total number of values ​​of the deviation evaluation coefficient that are less than the deviation threshold, and F is the probability of the current annotation result.

4. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 3, characterized in that: The possibility of the current annotation result is compared with the preset possible threshold. If it is greater than or equal to the possible threshold, the current annotation result is defined as a re-divided annotation result, and the corresponding type of the current annotation result is eliminated. On the contrary, if it is less than the possible threshold, the current annotation result is defined as a correct submission, and the current annotation result is used as the annotation standard for the corresponding type.

5. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 4, characterized in that: By bringing the re-divided annotation results into the annotation standards of each category for quality assessment, the categories with unqualified quality of the current annotation results are preferentially excluded to obtain the remaining divisible categories. The annotation standards of the remaining divisible categories and the current annotation results are brought into the similarity calculation to obtain the divisible category similarity; The annotation result that is most similar to the current annotation result is obtained by similarity calculation, and the annotation standard in the category corresponding to the annotation result that is most similar to the current annotation result is found and the difference between the similarity of the annotation standard that can be divided into categories is calculated to obtain the similarity difference between the category corresponding to the most similar annotation result and the most similar category.

6. The crowdsourcing-based word segmentation and annotation quality assessment system according to claim 5, characterized in that: The similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category are defined as input variables and divided into different fuzzy sets respectively; The division result of the re-divided annotation result is defined as the output variable and divided into fuzzy sets; Formulate fuzzy rules to describe the impact of the similarity of the divisible categories and the similarity difference between the category corresponding to the most similar annotation result and the most similar category on the division result of the re-divided annotation result; Fuzzy reasoning is performed based on fuzzy rules to determine the division scheme for re-dividing the annotation results.

Images