Text annotation correction method, device and system based on artificial intelligence and medium

Abstract

The invention discloses a text annotation correction method, device and system based on artificial intelligence and a medium, and the method comprises the steps: obtaining tags of a plurality of annotation personnel annotating the same text data set, and constructing a subjective prejudice detection model according to the tags; vectorizing the label of each labeling person, inputting the vectorized label into a subjective prejudice detection model, and predicting a subjective prejudice matrix of the labeling person; and subtracting the subjective bias matrix from the coding matrix converted from the label to obtain a labeling result after correcting the subjective bias of the labeling personnel. According to the method and the device, the problem that a trained artificial intelligence model is inaccurate due to the fact that manual annotation data for training supervised artificial intelligence has subjective deviation of annotation personnel can be solved, interference of the subjective deviation of the annotation personnel on an annotation result can be remarkably reduced, and the annotation result is more accurate and fair.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based text annotation correction method, device, system and medium. Background Art

[0002] In the field of artificial intelligence, many models require a large number of manually annotated data sets for training and verification. However, the subjectivity and consistency issues in the data annotation process have a significant impact on the accuracy of subsequent model training, which greatly affects the results of downstream tasks. Currently, ensuring the subjective consistency of annotators is a major difficulty in data annotation. Subjective bias means that different annotators may have different views on the data to be annotated, which will lead to artificial subjective differences in the annotation results. This phenomenon is particularly evident in data annotation for classification tasks.

[0003] There are two main solutions to this problem: the first is to use manual multiple labeling. The second is to use manual additional labeling of absolutely correct data sets. However, the labor cost of existing methods is relatively expensive, and the process of manual additional labeling will inevitably produce new wrong labels. Therefore, it is urgent to develop a method with low labor cost and can correct the subjective bias of the labeling results. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an artificial intelligence-based text annotation correction method, device, system and medium that can be applied to the field of artificial intelligence or other related fields. Its main purpose is to solve the problem that the manually labeled data used to train supervised artificial intelligence has subjective bias of the labelers, resulting in inaccurate trained artificial intelligence models.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a text annotation correction method based on artificial intelligence, comprising:

[0007] Obtain labels of the same text dataset from several annotators, and build a subjective bias detection model based on the labels;

[0008] The labels of each annotator are vectorized and input into the subjective bias detection model to predict the subjective bias matrix of the annotator;

[0009] The labeling result after correcting the subjective bias of the labeler is obtained by subtracting the subjective bias matrix from the coding matrix converted from the label.

[0010] In one embodiment, the step of obtaining labels of the same text dataset from a plurality of annotators and building a subjective bias detection model based on the labels includes:

[0011] Obtain labels of the same text data set annotated by several annotators, and convert the labels into a one-hot encoding matrix as a first detection result;

[0012] averaging the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix, and using the first unbiased label matrix as the second detection result;

[0013] Comparing the labeling results of all labelers in the first detection result with the second detection result, obtaining a subjective bias matrix of each labeler, and using it as a third detection result;

[0014] Performing a weighted average calculation on the third detection result to obtain a second unbiased label matrix as a fourth detection result;

[0015] Comparing the first detection result with the second unbiased label matrix to obtain a second subjective bias matrix as a fifth detection result;

[0016] The fifth detection result is input into a convolutional neural network to construct the subjective bias detection model.

[0017] In one embodiment, the step of obtaining labels annotated by several annotators for the same text data set and converting the labels into a one-hot encoding matrix as the first detection result includes:

[0018] Determine the label category based on the labels of the same text dataset and assign a code to each category;

[0019] For each annotated text data, the code corresponding to its annotated label is set to 1 in the vector, and the remaining positions are set to 0 to form a one-hot encoded vector;

[0020] The one-hot encoded vectors are combined into a one-hot encoding matrix as the first detection result.

[0021] In one embodiment, the step of averaging the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix and using the matrix as the second detection result includes:

[0022] The average of the labeling results of all labelers at the corresponding position of the one-hot encoding matrix is ​​calculated to obtain the average recognition of each category among all labelers;

[0023] A first unbiased label matrix is ​​generated according to the average recognition degree as a second detection result.

[0024] In one embodiment, the step of comparing the labeling results of all labelers in the first test result with the second test result to obtain the subjective bias matrix of each labeler and using it as the third test result includes:

[0025] Compare the labeling results of each labeler with the first unbiased label matrix, and calculate the difference between the labeling results and the unbiased labels on each text data;

[0026] The differences are accumulated and normalized to obtain the subjective bias matrix of each annotator as the third detection result.

[0027] In one embodiment, the step of performing weighted average calculation on the third detection result to obtain a second unbiased label matrix as the fourth detection result includes:

[0028] Comparing the results of all the annotators in the first detection result with the subjective bias matrix to obtain a preliminary unbiased label matrix for each annotator;

[0029] The difference between the preliminary unbiased label matrix and the first unbiased label matrix is ​​compared, and the preliminary unbiased label matrices of all labelers are weighted according to the weights to obtain a second unbiased label matrix; wherein the labeler with a larger absolute value of the difference has a smaller weight assigned to him.

[0030] In one embodiment, the step of subtracting the subjective bias matrix from the coding matrix converted from the labels to obtain the labeling result after correcting the subjective bias of the labeler includes:

[0031] The one-hot encoding matrix of the actual labeling results of the labelers is subtracted from the subjective bias matrix to obtain the detection model prediction result matrix;

[0032] For each label of text data, the category corresponding to the maximum value in the detection model prediction result matrix is ​​selected as the final labeling result.

[0033] A second aspect of the present invention provides a text annotation correction device based on artificial intelligence, comprising:

[0034] An acquisition module is used to obtain labels annotated by multiple annotators;

[0035] An input module, used for vectorizing the labels and inputting them into a subjective bias model to obtain a subjective bias matrix of the labeler;

[0036] A processing module, configured to obtain a labeling result after correcting the subjective bias of the labeler by subtracting the subjective bias matrix from the matrix converted from the labels;

[0037] A third aspect of the present invention provides a text annotation correction system based on artificial intelligence, the system comprising at least one processor; and

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned artificial intelligence-based text annotation correction method.

[0040] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned artificial intelligence-based text annotation correction method.

[0041] Beneficial effects: The present invention discloses a text annotation correction method, device, system and medium based on artificial intelligence. Compared with the prior art, the embodiment of the present invention obtains the labels of the same text data set annotated by several annotators, and constructs a subjective bias detection model according to the labels; the labels of each annotator are vectorized and input into the subjective bias detection model to predict the subjective bias matrix of the annotator; the coding matrix converted by the labels is subtracted from the subjective bias matrix to obtain the annotation result after correcting the subjective bias of the annotator. According to the present invention, there is no need for multiple manual annotations, that is, it can solve the problem that the artificially annotated data used to train supervised artificial intelligence has the subjective bias of the annotator, resulting in the inaccuracy of the trained artificial intelligence model, and can significantly reduce the interference of the subjective bias of the annotator on the annotation results. The annotation results are more accurate and fair, which is extremely important for improving the accuracy of subsequent model training. It can also significantly reduce the human resources for labeling. The annotators only need to jointly annotate a small amount of data, and there is no need to repeatedly annotate a large amount of data, which greatly reduces the labor cost, can improve production efficiency, fully automatically detect and correct the subjective bias of the annotators, and no manual intervention is required, eliminating the possibility of secondary introduction of subjective bias. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0043] Figure 1 A flow chart of a text annotation correction method based on artificial intelligence provided by an embodiment of the present invention;

[0044] Figure 2 A flowchart of step S100 in the text annotation correction method based on artificial intelligence provided in an embodiment of the present invention;

[0045] Figure 3 A flowchart of step S110 in the text annotation correction method based on artificial intelligence provided in an embodiment of the present invention;

[0046] Figure 4 A flowchart of step S120 in the text annotation correction method based on artificial intelligence provided in an embodiment of the present invention;

[0047] Figure 5 A flowchart of step S130 in the text annotation correction method based on artificial intelligence provided in an embodiment of the present invention;

[0048] Figure 6 A flowchart of step S140 in the text annotation correction method based on artificial intelligence provided in an embodiment of the present invention;

[0049] Figure 7 A flowchart of step S300 in the text annotation correction method based on artificial intelligence provided in an embodiment of the present invention;

[0050] Figure 8 A schematic diagram of functional modules of an artificial intelligence-based text annotation correction device provided in an embodiment of the present invention;

[0051] Fig. 9 A schematic diagram of the hardware structure of an artificial intelligence-based text annotation and correction system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0053] Subjective bias refers to the fact that different annotators may have different opinions on the data to be annotated, which will lead to artificial subjective differences in the annotation results. That is, in the data annotation process, different people hold different standards for the same data. Correction of subjective bias can solve the problem that the manually annotated data used to train supervised artificial intelligence has subjective bias of the annotators, resulting in inaccurate trained artificial intelligence models. For example, when marking subjective questions in an exam, since everyone has different opinions, it is easy to have different views on the answers to the subjective questions, which leads to different scores. Correction of subjective bias can make the scores of subjective questions more objective, that is, it can reduce the interference of the subjective bias of the annotators on the annotation results, and the annotation results are more accurate and fair.

[0054] The application scenarios of subjective bias correction are all about subjective judgment, such as the scoring of personnel actions in certain professional skills competitions and the marking of subjective questions in exams, which puts forward high requirements for subjective bias correction. However, most of them currently use multiple manual annotations and use manual additional annotations to identify absolutely correct data sets. However, the labor cost of existing methods is relatively expensive, and the process of manual additional annotation will inevitably produce new wrong labels.

[0055] In order to solve the above problems, the present invention proposes a text annotation correction method based on artificial intelligence, please refer to Figure 1 , Figure 1 A flowchart of an embodiment of the text annotation correction method based on artificial intelligence provided by the present invention. The text annotation correction method based on artificial intelligence provided in this embodiment is applied to a system consisting of a terminal device, a network and a server, wherein the network is a medium for providing a communication link between the terminal device and the server, which may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.; the operating system on the terminal device may include a handheld device operating system (iPhone operating system, iOS system), Android system or other operating system, and the terminal device is connected to the server through the network to achieve interaction, thereby performing operations such as receiving or sending data, and specifically may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, portable computers and desktop servers, etc. Figure 1 As shown, the method specifically comprises the following steps:

[0056] S100, obtaining labels of the same text data set annotated by several annotators, and building a subjective bias detection model based on the labels;

[0057] S200, vectorizing the labels of each annotator and inputting them into a subjective bias detection model to predict the subjective bias matrix of the annotator;

[0058] S300, subtracting the subjective bias matrix from the coding matrix converted from the labels to obtain a labeling result after correcting the subjective bias of the labeler.

[0059] In this embodiment, it is first necessary to obtain or create a text dataset, and then label it by several labelers. These labels will serve as the basis for model training. For example, in the construction of a user portrait label system, labels can be based on multiple dimensions such as user attributes and user behaviors. During the construction process, it is crucial to ensure the accuracy and consistency of the labels; these labels are converted into a format that can be processed by the model, such as using word embedding technology to convert the labels into vector form. Label vectorization allows the model to map text labels to numerical space, thereby performing mathematical calculations, and using vectorized labels as training data to build or train a subjective bias detection model. This model can use a variety of machine learning algorithms to evaluate the performance of the model and ensure that it can accurately predict subjective bias. This can be done by comparing the results predicted by the model with the bias of the actual labelers. Once the model is trained and verified, it can be used to predict the subjective bias matrix of each labeler. The subjective bias matrix will show the degree of bias of different labelers on different labels. The predicted bias matrix is ​​used to improve the labeling process, improve the quality of the data set, and reduce the impact of bias on the final model. The labels of the labelers are used to evaluate the bias level of each labeler using the trained subjective bias detection model. This may require a pre-trained model that can identify and quantify the bias in the annotations and convert them into a coding matrix in numerical form. This may involve the use of text vectorization techniques such as one-hot encoding, TF-IDF, word embedding models, etc. According to the output results of the bias detection model, a matrix representing the degree of bias of each annotator is constructed. The subjective bias matrix is ​​subtracted from the coding matrix to obtain the annotation results after correcting the subjective bias of the annotator. The annotation results adjust the original annotations to reduce the impact of individual bias. The corrected annotation results are analyzed in detail to verify whether the bias is effectively controlled and to ensure the accuracy and fairness of the results.

[0060] Technical effect: The technical solution of the present invention can solve the problem that the manually labeled data used to train supervised artificial intelligence carries the subjective bias of the labelers, resulting in inaccurate trained artificial intelligence models. It can significantly reduce the interference of the subjective bias of the labelers on the labeling results, and the labeling results are more accurate and fair. It plays an extremely important role in improving the accuracy of subsequent model training. It can also significantly reduce the human resources for labeling. The labelers only need to jointly label a small amount of data, and there is no need to repeatedly label a large amount of data, which greatly reduces the labor cost and can improve production efficiency. It can automatically detect and correct the subjective bias of the labelers without manual intervention, eliminating the possibility of secondary introduction of subjective bias.

[0061] In one embodiment, Figure 2As shown, the step of obtaining labels of the same text data set annotated by several annotators in step S100, and building a subjective bias detection model based on the labels, includes:

[0062] S110, obtaining labels annotated by several annotators on the same text data set, and converting the labels into a one-hot encoding matrix as a first detection result;

[0063] S120, averaging the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix, and using it as the second detection result;

[0064] S130, comparing the labeling results of all labelers in the first detection result with the second detection result, obtaining a subjective bias matrix of each labeler, and using it as a third detection result;

[0065] S140, performing weighted average calculation on the third detection result to obtain a second unbiased label matrix as a fourth detection result;

[0066] S150, comparing the first detection result with the second unbiased label matrix to obtain a second subjective bias matrix as a fifth detection result;

[0067] S160: Input the fifth detection result into a convolutional neural network to construct the subjective bias detection model.

[0068] In this embodiment, first, labels of several annotators for the same text data set are collected, and the labels of each annotator are converted into a one-hot encoding matrix. The one-hot encoding matrices of all annotators are averaged to obtain the first unbiased label matrix as the second detection result. The first unbiased label matrix represents the consensus of the labeling results of all annotators.

[0069] The one-hot encoding matrix of each annotator is compared with the first unbiased label matrix to obtain the subjective bias matrix of each annotator as the third test result. The subjective bias matrix reflects the deviation between each annotator and the consensus. The third test result is weighted averaged to obtain the second unbiased label matrix as the fourth test result. The weighted average can be based on factors such as the experience and reliability of the annotator.

[0070] The first detection result (original one-hot encoding matrix) is compared with the fourth detection result (second unbiased label matrix) to obtain the second subjective bias matrix, which further refines the subjective bias of the labeler.

[0071] The fifth test result is used as input, and a convolutional neural network (CNN) is used to build a subjective bias detection model. Since the fourth test reduces the influence of the subjective bias of the labeler's label compared to the second test result, the fifth test result calculated from the fourth test result is more accurate than the third test result calculated from the second test result. Therefore, this process can significantly reduce the interference of the subjective bias of the labeler on the labeling result, and the labeling result is more accurate and fair, which plays an extremely important role in improving the accuracy of subsequent model training.

[0072] CNN can capture patterns and features in labeled data to identify and quantify subjective bias. Use labeled data to train the CNN model and use methods such as cross-validation to evaluate the model's performance. Ensure that the model can effectively identify subjective bias, apply the trained model to actual labeled data, detect and correct subjective bias, and collect feedback information to continuously optimize the model.

[0073] Through this process, a system can be built that can not only detect subjective biases of annotators, but also provide a mechanism to correct for these biases, thereby improving the quality and fairness of data annotation.

[0074] In one embodiment, Figure 3 As shown, step S110 obtains labels of the same text data set annotated by several annotators, and converts the labels into a one-hot encoding matrix as the first detection result, including:

[0075] S111, determining label categories according to labels of the same text data set, and assigning codes to each category;

[0076] S112. For each annotated text data, the code corresponding to the annotated label is set to 1 in the vector, and the remaining positions are set to 0 to form a one-hot encoded vector.

[0077] S113: Combining the one-hot encoded vectors into a one-hot encoding matrix as a first detection result.

[0078] In this embodiment, first, all the tags in the text data set need to be classified. This step usually involves counting and analyzing the tags in the data set to determine different tag categories and assigning a unique code to each tag category. This code is usually an integer starting from 0, but it can also be a unique identifier in other forms. For each text data, one or more tags are assigned by a manual or automatic tagging system according to its content and context. For each annotated text data, a vector is created according to its tag. The length of this vector is equal to the total number of tag categories. In the vector, the encoding position corresponding to the tag annotated with the text data is set to 1, indicating that the text belongs to this category, and all other positions are set to 0, indicating that the text does not belong to these categories. Among them, One-hot encoding is a way of representing tags, in which each category is represented by a unique binary vector, in which only one position is 1 and the rest are 0. Therefore, each text data can be represented by a one-hot vector, which reflects the relationship between the text and each tag category.

[0079] Combine the one-hot encoding vectors of all text data to form a matrix. Each row of this matrix represents the one-hot encoding of a text data, and each column represents the encoding of a label category. Each text data can be represented by a row in the matrix, and the entire matrix reflects the relationship between all text data in the entire data set and each label category. One-hot encoded vectors are often used as input for machine learning models, especially in classification tasks such as text classification and sentiment analysis. One-hot encoding is simple and intuitive, easy to implement, and can clearly represent category relationships. One-hot encoding provides a method to convert text data into numerical data, so that machine learning models can process this data.

[0080] In one embodiment, Figure 4 As shown, step S120 averages the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix, and uses it as the second detection result, including:

[0081] S121, average the labeling results of all labelers at the corresponding positions of the one-hot encoding matrix to obtain the average recognition of each category among all labelers;

[0082] S122. Generate a first unbiased label matrix according to the average recognition degree as a second detection result.

[0083] In this embodiment, when multiple labelers label the same text data set, the average recognition of each category among all labelers can be obtained by calculating the average value of the corresponding position in the one-hot encoding matrix. Specifically, for each text data, each element in its one-hot encoding vector represents the labeling of a category, 1 means that the category is labeled, and 0 means that it is not labeled. Then, the one-hot encoding vectors of all labelers are element-wise averaged to obtain the average recognition of each category.

[0084] Based on these average recognitions, a first unbiased label matrix can be generated as the second test result. The first unbiased label matrix reflects the average labeling of each category among all labelers, and can be used to further analyze the labeling consistency of the data set or as an input feature for model training. This method helps to reduce the bias caused by the subjectivity of individual labelers, thereby improving the generalization ability of the model.

[0085] When processing text datasets, for each annotated text data, we can convert it into numerical data through one-hot encoding, and then use it in machine learning models. One-hot encoding is a method of converting categorical variables into a format that is easy for machine learning algorithms to handle. It eliminates possible erroneous numerical relationships between categories by assigning an independent binary vector to each category, thereby preventing the algorithm from making inaccurate predictions based on these relationships.

[0086] One-hot encoding is used to represent variables with limited categories, such as gender (male, female), color (red, green, blue), etc. Each categorical feature is represented as a binary vector with a length equal to the number of categories. For each category, One-hot encoding creates a vector with a length equal to the total number of categories, where the element representing the category is set to 1 and the remaining elements are set to 0. In programming implementation, One-hot encoding can be done through various libraries. For example, the Pandas library in Python can use the get_dummies function for One-hot encoding. One-hot encoding is often used to process categorical data, especially when there are multiple categorical features in the data set. One-hot encoding is a simple and effective method for converting categorical data into a format suitable for machine learning models. Although it may cause high-dimensionality problems in some cases, it is still an indispensable preprocessing step in many machine learning tasks.

[0087] In practical applications, although one-hot encoding can clearly distinguish different categories, when the number of categories is large, it will cause the dimension of the feature space to increase dramatically, which may cause the computational complexity of the model to increase and cause overfitting problems. Therefore, when applying one-hot encoding, it is necessary to weigh its advantages and disadvantages according to the specific situation and consider possible dimensionality reduction techniques, such as principal component analysis (PCA). At the same time, for situations with a large number of categories, other types of encoding methods, such as embeddings, can also be considered to express category information in a more compact form.

[0088] In one embodiment, Figure 5 As shown, the step of comparing the labeling results of all labelers in the first detection result with the second detection result to obtain the subjective bias matrix of each labeler as the third detection result includes:

[0089] S131, comparing the labeling results of each labeler with the first unbiased label matrix, and calculating the difference between the labeling results and the unbiased labels on each text data;

[0090] S132: The differences are accumulated and normalized to obtain the subjective bias matrix of each labeler as the third detection result.

[0091] In this embodiment, for each text data, its vector representation in the one-hot encoding matrix is ​​obtained, and the vector representation is compared with the corresponding row vector in the first unbiased label matrix. For each position in the vector, the difference between the annotation result vector and the first unbiased label matrix vector is calculated. The difference can be a simple Hamming distance (i.e., the number of different positions) or a more complex metric, such as cosine similarity, etc. The difference on each text data is accumulated to obtain the total difference between each annotation person and the first unbiased label matrix on all text data. In order to obtain the subjective bias of each annotation person, the cumulative difference needs to be normalized. Normalization can be achieved by dividing the cumulative difference of each annotation person by the maximum value of the cumulative difference of all annotation persons. Finally, the normalized difference of each annotation person will form the subjective bias matrix, and the rows of the subjective bias matrix represent different annotation persons, and the columns represent different text data. Each element in the subjective bias matrix reflects the degree of subjective bias of the corresponding annotation person on the corresponding text data.

[0092] This method can quantify the difference between each annotator and the first unbiased label matrix, and then evaluate their subjective bias. This helps to identify possible annotation bias and provide a reference for further data cleaning and model training.

[0093] In one embodiment, Figure 6 As shown, the step of performing weighted average calculation on the third detection result in step S140 to obtain a second unbiased label matrix as the fourth detection result includes:

[0094] S141, comparing the results of all the labelers in the first detection result with the subjective bias matrix to obtain a preliminary unbiased label matrix for each labeler;

[0095] S142. Compare the difference between the preliminary unbiased label matrix and the first unbiased label matrix, weight the preliminary unbiased label matrices of all labelers according to the weights, and obtain a second unbiased label matrix, wherein the labeler with a larger absolute value of the difference is assigned a smaller weight.

[0096] In this embodiment, the results of each annotator in the first detection result set are compared with the subjective bias matrix, and the deviation is calculated by a certain measurement method (such as Euclidean distance, cosine similarity, etc.), so as to obtain a preliminary unbiased label matrix for each annotator. The preliminary unbiased label matrix of each annotator is compared with the first unbiased label matrix, the difference is calculated, and the weight is assigned according to the absolute value of the difference. If the difference between the preliminary unbiased label matrix of the annotator and the first unbiased label matrix is ​​large, it indicates that the result deviation of the annotator is large, so a smaller weight is assigned; conversely, if the difference is small, a larger weight is assigned.

[0097] The preliminary unbiased label matrices of all labelers are weighted and summed according to their respective weights to obtain the second unbiased label matrix. The second unbiased label matrix integrates the opinions of all labelers and reduces the influence of individual bias. A dynamic weight adjustment mechanism is designed to dynamically adjust the weights of labelers according to their performance in continuous tasks to ensure the accuracy and fairness of the second unbiased label matrix.

[0098] This method is suitable for fields that require a large number of labelers, such as natural language processing and computer vision. In particular, in data labeling tasks, it is crucial to ensure the consistency and fairness of the labeling results. This method can significantly improve the quality of the labeling results and reduce the impact of individual bias on the final model performance, thereby providing more reliable data for the training of machine learning models. Through this refinement process, we can ensure that the labeling results are more accurate and fair, providing a solid foundation for building high-quality and reliable artificial intelligence models.

[0099] In one embodiment, as shown in Part 7, step S300, the step of obtaining the labeling result after correcting the subjective bias of the labeler by subtracting the subjective bias matrix from the coding matrix converted from the label, comprises:

[0100] S310, subtracting the subjective bias matrix from the one-hot encoding matrix of the actual labeling results of the labelers to obtain a detection model prediction result matrix;

[0101] S320: For each label of text data, select the category corresponding to the maximum value in the detection model prediction result matrix as the final labeling result.

[0102] In this embodiment, for each annotator's one-hot encoding matrix, the corresponding row vector in the subjective bias matrix is ​​subtracted to obtain the detection model prediction result matrix. This step aims to eliminate or reduce the impact of the annotation personnel's subjective bias on the annotation results, and analyze the detection model prediction result matrix, which contains the corrected annotation results and reflects the label distribution after removing individual bias. For each text data, the category with the highest probability (or score) is selected from the detection model prediction result matrix as the final annotation result of the text. This step ensures the objectivity and accuracy of the annotation results; selecting the category corresponding to the maximum value means selecting the category with the highest score after correction among all possible categories, which represents the most likely prediction of the model after removing the bias.

[0103] Verify the bias-corrected annotation results to ensure that they meet the expected quality and standards. Evaluate the effect of bias correction by comparative analysis with the original annotation results. Establish a feedback mechanism to compare the bias-corrected results with the performance in actual applications, and continuously adjust and optimize the subjective bias matrix and bias correction algorithm. This method is suitable for text data annotation tasks that require high accuracy and fairness, such as sentiment analysis, topic classification, and named entity recognition in natural language processing. Use the results of bias correction to train and guide annotators to help them understand their potential biases and improve the consistency and accuracy of annotations. Through this refinement process, we can significantly improve the quality and reliability of text data annotations and provide a more accurate and fair data foundation for the training of machine learning models.

[0104] Another embodiment of the present invention provides a text annotation correction device based on artificial intelligence, such as Figure 8 As shown, the device 1 comprises:

[0105] An acquisition module 11 is used to obtain labels of the same text data set annotated by several annotators, and to build a subjective bias detection model based on the labels;

[0106] An input module 12 is used to vectorize the labels of each annotator and input them into the subjective bias detection model to predict the subjective bias matrix of the annotator;

[0107] The processing module 13 is used to obtain a labeling result after correcting the subjective bias of the labeler by subtracting the subjective bias matrix from the coding matrix converted from the label.

[0108] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of text annotation correction based on artificial intelligence than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0109] In one embodiment, the processing module 13 includes:

[0110] An execution unit, used for subtracting the subjective bias matrix from the one-hot encoding matrix of the actual labeling results of the labelers to obtain a detection model prediction result matrix;

[0111] The screening unit is used to select the category corresponding to the maximum value in the detection model prediction result matrix as the final labeling result for each text data label.

[0112] In one embodiment, the device 1 further comprises:

[0113] A conversion module is used to obtain labels of the same text data set annotated by several annotators, and convert the labels into a one-hot encoding matrix as a first detection result;

[0114] A first weighting module is used to average the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix as the second detection result;

[0115] A first comparison module, used for comparing the labeling results of all labelers in the first detection result with the second detection result, to obtain a subjective bias matrix of each labeler as a third detection result;

[0116] A second weighting module, used for performing weighted average calculation on the third detection result to obtain a second unbiased label matrix as a fourth detection result;

[0117] A second comparison module, used for comparing the first detection result with the second unbiased label matrix to obtain a second subjective bias matrix as a fifth detection result;

[0118] A generation module is used to input the fifth detection result into a convolutional neural network to construct the subjective bias detection model.

[0119] In one embodiment, the conversion module comprises:

[0120] The coding unit is used to determine the label category based on the labels of the same text dataset and assign a code to each category;

[0121] A judgment unit is used to set the code corresponding to the annotated label of each annotated text data to 1 in the vector and set the remaining positions to 0 to form a one-hot encoded vector;

[0122] The vector reorganization unit is used to combine the one-hot encoded vectors into a one-hot encoding matrix as the first detection result.

[0123] In one embodiment, the first weighting module comprises:

[0124] The first average value calculation unit is used to average the labeling results of all labelers at the corresponding positions of the one-hot encoding matrix to obtain the average recognition of each category among all labelers;

[0125] The first synthesis unit is used to generate a first unbiased label matrix as a second detection result according to the average recognition degree.

[0126] In one embodiment, the first comparison module includes:

[0127] A first comparison unit is used to compare the labeling results of each labeler with the first unbiased label matrix, and calculate the difference between the labeling results and the unbiased labels on each text data;

[0128] The first accumulation unit is used to accumulate and normalize the differences to obtain the subjective bias matrix of each labeler as the third detection result.

[0129] In one embodiment, the second weighting module comprises:

[0130] A second average value calculation unit, used for comparing the results of all labelers in the first detection result with the subjective bias matrix to obtain a preliminary unbiased label matrix for each labeler;

[0131] The second synthesis unit is used to compare the difference between the preliminary unbiased label matrix and the first unbiased label matrix, weight the preliminary unbiased label matrices of all labelers according to the weight, and obtain a second unbiased label matrix, wherein the labeler with a larger absolute value of the difference is assigned a smaller weight.

[0132] Another embodiment of the present invention provides a text annotation correction system based on artificial intelligence, such as Fig. 9 As shown, the system 10 includes:

[0133] One or more processors 110 and memory 120, Figure 6A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0134] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0135] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the text annotation correction method based on artificial intelligence in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, realizing the text annotation correction method based on artificial intelligence in the above method embodiment.

[0136] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] One or more units are stored in the memory 120, and when executed by one or more processors 110, the text annotation correction method based on artificial intelligence in any of the above method embodiments is executed.

[0138] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the text annotation correction method based on artificial intelligence in any of the above method embodiments is executed, for example, the following steps are executed:

[0139] Obtain labels of the same text dataset from several annotators, and build a subjective bias detection model based on the labels;

[0140] The labels of each annotator are vectorized and input into the subjective bias detection model to predict the subjective bias matrix of the annotator;

[0141] The labeling result after correcting the subjective bias of the labeler is obtained by subtracting the subjective bias matrix from the coding matrix converted from the label.

[0142] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0143] In summary, the present invention discloses a method, device, system and medium for correcting text annotation based on artificial intelligence, the method includes: obtaining labels of the same text data set annotated by several annotators, and constructing a subjective bias detection model according to the labels; vectorizing the labels of each annotator and inputting them into the subjective bias detection model to predict the subjective bias matrix of the annotator; and subtracting the subjective bias matrix from the coding matrix converted by the labels to obtain the annotation result after correcting the subjective bias of the annotator. According to the present invention, it is possible to solve the problem that the artificially annotated data used for training supervised artificial intelligence has the subjective bias of the annotator, resulting in the inaccuracy of the trained artificial intelligence model, and it is possible to significantly reduce the interference of the subjective bias of the annotator on the annotation result, and the annotation result is more accurate and fair, which plays an extremely important role in improving the accuracy of subsequent model training, and can also significantly reduce the human resources for labeling. The annotators only need to jointly annotate a small amount of data, and there is no need to repeatedly annotate a large amount of data, which greatly reduces the labor cost, can improve production efficiency, fully automatically detect and correct the subjective bias of the annotators, and no human intervention is required, eliminating the possibility of introducing subjective bias twice.

[0144] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0145] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A text annotation correction method based on artificial intelligence, characterized in that: include: Obtain labels of the same text dataset from several annotators, and build a subjective bias detection model based on the labels; The labels of each annotator are vectorized and input into the subjective bias detection model to predict the subjective bias matrix of the annotator; The labeling result after correcting the subjective bias of the labeler is obtained by subtracting the subjective bias matrix from the coding matrix converted from the label.

2. The text annotation correction method based on artificial intelligence according to claim 1 is characterized in that: The step of obtaining labels annotated by a plurality of annotators for the same text data set and building a subjective bias detection model based on the labels includes: Obtain labels of the same text data set annotated by several annotators, and convert the labels into a one-hot encoding matrix as a first detection result; averaging the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix, and using the first unbiased label matrix as the second detection result; Comparing the labeling results of all labelers in the first detection result with the second detection result, obtaining a subjective bias matrix of each labeler, and using it as a third detection result; Performing a weighted average calculation on the third detection result to obtain a second unbiased label matrix as a fourth detection result; Comparing the first detection result with the second unbiased label matrix to obtain a second subjective bias matrix as a fifth detection result; The fifth detection result is input into a convolutional neural network to construct the subjective bias detection model.

3. The text annotation correction method based on artificial intelligence according to claim 2 is characterized in that: The step of obtaining labels annotated by a plurality of annotators on the same text data set and converting the labels into a one-hot encoding matrix as the first detection result comprises: Determine the label category based on the labels of the same text dataset and assign a code to each category; For each annotated text data, the code corresponding to its annotated label is set to 1 in the vector, and the remaining positions are set to 0 to form a one-hot encoded vector; The one-hot encoded vectors are combined into a one-hot encoding matrix as the first detection result.

4. The text annotation correction method based on artificial intelligence according to claim 2 is characterized in that: The step of averaging the labeling results of all labelers in the first detection result to obtain a first unbiased label matrix and using it as the second detection result includes: The average of the labeling results of all labelers at the corresponding position of the one-hot encoding matrix is ​​calculated to obtain the average recognition of each category among all labelers; A first unbiased label matrix is ​​generated according to the average recognition degree as a second detection result.

5. The text annotation correction method based on artificial intelligence according to claim 2 is characterized in that: The step of comparing the labeling results of all labelers in the first test result with the second test result to obtain the subjective bias matrix of each labeler and using it as the third test result includes: Compare the labeling results of each labeler with the first unbiased label matrix, and calculate the difference between the labeling results and the unbiased labels on each text data; The differences are accumulated and normalized to obtain the subjective bias matrix of each annotator as the third detection result.

6. The text annotation correction method based on artificial intelligence according to claim 2 is characterized in that: The step of performing weighted average calculation on the third detection result to obtain a second unbiased label matrix as the fourth detection result includes: Comparing the results of all the annotators in the first detection result with the subjective bias matrix to obtain a preliminary unbiased label matrix for each annotator; The difference between the preliminary unbiased label matrix and the first unbiased label matrix is ​​compared, and the preliminary unbiased label matrices of all labelers are weighted according to the weights to obtain a second unbiased label matrix, wherein the labeler with a larger absolute value of the difference is assigned a smaller weight.

7. The text annotation correction method based on artificial intelligence according to claim 1 is characterized in that: The step of subtracting the subjective bias matrix from the coding matrix converted from the labels to obtain the labeling result after correcting the subjective bias of the labeler comprises: The one-hot encoding matrix of the actual labeling results of the labelers is subtracted from the subjective bias matrix to obtain the detection model prediction result matrix; For each label of text data, the category corresponding to the maximum value in the detection model prediction result matrix is ​​selected as the final labeling result.

8. A text annotation correction device based on artificial intelligence, characterized in that: include: An acquisition module is used to obtain labels of the same text data set annotated by several annotators, and to build a subjective bias detection model based on the labels; An input module, used to vectorize the labels of each annotator and input them into the subjective bias detection model to predict the subjective bias matrix of the annotator; The processing module is used to obtain a labeling result after correcting the subjective bias of the labeler by subtracting the subjective bias matrix from the coding matrix converted from the label.

9. A text annotation correction system based on artificial intelligence, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the artificial intelligence-based text annotation correction method described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the one or more processors can execute the artificial intelligence-based text annotation correction method described in any one of claims 1-7.

Images