A method and apparatus for annotation quality control with active learning capabilities

Abstract

This invention relates to a method and apparatus for annotation quality control with active learning capabilities. The method includes: before executing the first annotation task, constructing and training a difficulty assessment model and a quality prediction model; during each annotation process, dynamically allocating annotation personnel based on the assessment results of the difficulty assessment model and a personnel information database; using the quality prediction model to monitor annotation quality in real time during the annotation process; and at the end of the annotation process, performing a quality assessment of the annotation results according to rules and updating the personnel information database and the annotation sample pool; after executing the first task, periodically confirming whether active learning is needed based on the most recent assessment status; if so, selecting typical records from the annotation sample pool to update the validation dataset, and upgrading the difficulty assessment model and the quality prediction model based on the updated validation dataset. This invention can improve efficiency, reduce costs, provide real-time early warning and dynamic task allocation, and proactively improve quality control.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a labeling quality control method and apparatus with active learning capabilities. Background Technology

[0002] In the context of the rapid development of artificial intelligence technology, high-quality labeled data is crucial for model training. Currently, conventional quality control solutions mostly adopt a post-labeling quality control approach, that is, quality inspectors review the labeling results one by one after the labelers have completed their labeling tasks. This conventional approach has obvious drawbacks: 1) It is time-consuming, inefficient, and incurs high labor costs; 2) Quality control feedback can only be generated after labeling is completed, and it is impossible to provide real-time early warnings and interventions for the labelers; 3) It cannot automatically identify the difficulty of the labeling tasks, and therefore cannot dynamically match tasks based on the difficulty identification results and the labelers' abilities; 4) It lacks the proactive learning ability for self-optimization and iteration. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a labeling quality control method, apparatus, electronic device, and computer-readable storage medium with active learning capabilities. Before executing the first labeling task, this invention constructs and trains a difficulty assessment model for evaluating the difficulty of the labeled files and a quality prediction model for real-time evaluation of labeling quality. During the execution of each labeling task, labeling personnel are assigned based on the difficulty assessment results of the target files using the difficulty assessment model and a personnel information database. The assigned personnel then manually label the target files. The quality prediction model monitors the labeling quality in real time during the labeling process. At the end of the labeling, the quality of the labeling results is evaluated according to rules, and the personnel information database and labeling sample pool are updated based on the labeling process data. After executing the first labeling task, the need for a round of active learning is periodically confirmed based on the recent quality assessment status. If necessary, the validation dataset is updated based on the labeling sample pool, and the difficulty assessment model and quality prediction model are upgraded and trained based on the updated validation dataset. This invention can not only shorten the cycle, improve efficiency, and reduce labor costs, but also provide real-time early warning and intervention for annotation personnel, automatically identify the difficulty of annotation tasks and perform dynamic task matching based on the identification results and personnel information, and continuously improve the predictive ability of the model through continuous updates of the validation set and active learning mechanisms.

[0004] To achieve the above objectives, a first aspect of the present invention provides a labeling quality control method with active learning capability, the method comprising:

[0005] Before performing the first annotation task, a difficulty assessment model and a quality prediction model are constructed; a training dataset is constructed through data collection; the difficulty assessment model and the quality prediction model are trained for the first time based on the training dataset; and typical data records are selected from the training dataset to form an initial validation dataset.

[0006] During the execution of each annotation task, the text file to be annotated is taken as the target file; and annotation personnel are assigned based on the difficulty assessment result of the target file according to the difficulty assessment model and the preset personnel information database; the assigned personnel manually annotate the target file; during this annotation process, the annotation quality is monitored in real time using the quality prediction model; at the end of this annotation, the quality of this annotation result is assessed according to the preset annotation quality inspection rule database; and the personnel information database and the preset annotation sample pool are updated based on the data of this annotation process.

[0007] After performing the first annotation task, the system periodically checks whether an active learning round is needed based on the quality assessment of the most recent period. If it is confirmed that an active learning round is needed, typical data records are selected from the annotation sample pool to update the validation dataset, and the difficulty assessment model and the quality prediction model are upgraded and trained once each based on the updated validation dataset.

[0008] Preferably, the difficulty assessment model is used to predict the annotation difficulty of the annotation file input to the model and output the corresponding difficulty score; the score range of the difficulty score is [0,1]; the annotation file is a text file;

[0009] The quality prediction model is used to predict the annotation quality of the annotators based on the annotator information, annotation behavior sequence and annotation content information input to the model and output the corresponding quality prediction score; the score range of the quality prediction score is [0,1].

[0010] The labeling personnel information includes personnel level, domain scope, and historical task sequence; the personnel level is a labeling level from a preset personnel level set; the domain scope consists of one or more knowledge domains from a preset knowledge domain set; the historical task sequence includes multiple historical tasks; the historical tasks include task execution time, task difficulty score, and task quality score;

[0011] The annotation behavior sequence is composed of multiple single-step annotation behaviors ordered chronologically; each single-step annotation behavior includes annotation operation type and operation processing time; the annotation operation type includes editing a new annotation, modifying an old annotation, and undoing an old annotation.

[0012] The annotation information includes the set of elements to be annotated and the set of elements already annotated;

[0013] The set of elements to be annotated includes multiple elements to be annotated; each element to be annotated corresponds to a word segment in the annotation file; each element to be annotated includes an element identifier, an element domain, an element position, an element length, and an element text; the element domain is a type of knowledge domain in the knowledge domain set; the element position is the position of the first character of the current element in the annotation file;

[0014] The labeled element set includes multiple labeled elements; each labeled element corresponds one-to-one with the element to be labeled; each labeled element includes an element identifier, an emotion type label, and an entity three-element label; the emotion type label is one type of emotion from a preset emotion type set; the entity three-element label includes an entity identifier and a three-element type; the three-element type is one type of element from a preset three-element type set, which includes subject element type, relationship element type, and object element type;

[0015] The training dataset and the validation dataset each consist of multiple first data records; each first data record includes a first training file, a first training parameter set, a first difficulty label, and a first quality label; the first training parameter set consists of a set of annotator information, annotation behavior sequences, and annotation content information; the first training file is a text file; each element to be annotated in the first training parameter set corresponds to a word segment in the first training file; the first difficulty label is a corresponding label difficulty score; the first quality label is a corresponding label quality score; the first training file and the first difficulty label are used to train or validate the difficulty assessment model, and the first training parameter set and the first quality label are used to train or validate the quality prediction model.

[0016] The personnel information database includes multiple first personnel records; each first personnel record corresponds to one annotation personnel; each first personnel record includes a first personnel identifier, a first name, a first personnel level, a first personnel status, a first domain scope, and a first task sequence; the first personnel identifier, the first name, and the first personnel level are respectively the unique identifier, real name, and annotation level of the current annotation personnel; the first personnel status includes idle status and working status; the first domain scope is the professional domain scope of the current annotation personnel, composed of one or more knowledge domains from a knowledge domain set; the first task sequence is formed by chronologically ordering multiple first tasks completed by the current annotation personnel within the most recent first time period, with each first task corresponding to a historical annotation task; the first task includes the task execution time, the task difficulty score, and the task quality score; the task difficulty score is the annotation difficulty score of the current annotation task; the task quality score is the annotation quality score of the current annotation task.

[0017] The labeled sample pool comprises multiple first sample records; each first sample record includes a first sample file and a first sample element set; the first sample file is a text file; and the first sample element set is a set of elements to be labeled.

[0018] Preferably, the difficulty assessment model includes a complexity feature recognition unit, a structural feature recognition unit, a domain feature recognition unit, a first feature fusion unit, a first hidden layer network, and a first output layer;

[0019] The complexity feature recognition unit is used to recognize the text complexity features corresponding to the labeled file according to the preset text complexity feature recognition rules, obtain the corresponding complexity feature vector H1, and send it to the first feature fusion unit; the vector length of the complexity feature vector H1 is the preset first feature dimension C1;

[0020] The structural feature recognition unit is used to recognize the text structural features of the labeled file according to the preset text structural feature recognition rules, obtain the corresponding structural feature vector H2, and send it to the first feature fusion unit; the vector length of the structural feature vector H2 is the preset second feature dimension C2;

[0021] The domain feature recognition unit is used to recognize the text knowledge domain features of the labeled file according to the preset text knowledge domain feature recognition rules, obtain the corresponding domain feature vector H3, and send it to the first feature fusion unit; the vector length of the domain feature vector H3 is the preset third feature dimension C3;

[0022] The first feature fusion unit is used to perform vector concatenation processing on the complexity feature vector H1, the structural feature vector H2 and the domain feature vector H3 to obtain the corresponding concatenated vector H4 and send it to the first hidden layer network; the vector length of the concatenated vector H4 is a preset fourth feature dimension C4, C4 = C1 + C2 + C3;

[0023] The first hidden layer network is composed of three first hidden layers connected in sequence; each first hidden layer is composed of a fully connected layer and a ReLU activation function connected in sequence; the first hidden layer network is used to perform feature extraction processing on the concatenated vector H4 to obtain the corresponding feature vector H5 and send it to the first output layer; the vector length of the feature vector H5 is a preset fifth feature dimension C5, 1 < C5 < C4.

[0024] The first output layer is used to perform regression calculation on the labeled difficulty score based on the feature vector H5 to obtain the corresponding difficulty score and output it.

[0025] in,

[0026] W1 and b1 are the weight vector parameter and offset scalar parameter of the first output layer, respectively; the vector length of the weight vector parameter W1 is C5; the tensor shape of the feature vector H5 and the weight vector parameter W1 is 1×C5; Sigmoid() is the Sigmoid activation function; S difficulty The difficulty level is rated.

[0027] Preferably, the quality prediction model includes a personnel feature extraction unit, a temporal feature extraction unit, a content feature extraction unit, a first encoder, a second encoder, a third encoder, a second feature fusion unit, a second hidden layer network, and a second output layer.

[0028] The personnel feature extraction unit is used to perform one-hot encoding on the personnel level of the labeled personnel information according to the personnel level set to obtain the corresponding level one-hot encoding vector; and to perform one-hot encoding on the domain range of the labeled personnel information according to the knowledge domain set to obtain the corresponding domain one-hot encoding vector; and to calculate the corresponding historical accuracy according to the historical task sequence of the labeled personnel information; and to send the corresponding personnel feature vector X1 composed of the historical accuracy, the level one-hot encoding vector, and the domain one-hot encoding vector to the first encoder.

[0029] The historical accuracy rate is calculated as follows:

[0030]

[0031] N hisThe total number of historical tasks in the historical task sequence, where 1 ≤ task index i ≤ N his ;l i d i s i These are, respectively, the task execution time, the task difficulty score, and the task quality score corresponding to the i-th historical task in the historical task sequence; p his The historical accuracy rate;

[0032] The temporal feature extraction unit is used to perform one-hot encoding on the annotation operation type of each single-step annotation behavior in the annotation behavior sequence according to a preset set of operation types to obtain the corresponding operation type one-hot encoding vector; and to normalize the operation processing time of each single-step annotation behavior to obtain the corresponding normalized time; and to form a corresponding single-step embedding vector by each operation type one-hot encoding vector and its corresponding normalized time; and to send the corresponding temporal feature sequence X2, which is formed by sorting all the obtained single-step embedding vectors in chronological order, to the second encoder.

[0033] The content feature extraction unit is used to perform one-hot encoding, position embedding encoding, length normalization, and word segmentation embedding encoding on the element domain, element position, element length, and element text of each element to be labeled in the labeled content information to obtain the corresponding element domain one-hot encoding vector, position embedding encoding vector, normalized length, and word segmentation embedding encoding vector; and initialize a full-zero sentiment type one-hot encoding vector for each element to be labeled based on the sentiment type set, and initialize a full-zero three-element type one-hot encoding vector for each element to be labeled based on the three-element type set, and set an entity identifier encoding initialized to zero for each element to be labeled, and then use each entity identifier encoding... The three element types corresponding to each element form a corresponding entity encoding vector; and the element domain one-hot encoding vector, the position embedding encoding vector, the normalized length, the word segmentation embedding encoding vector, the sentiment type one-hot encoding vector, and the entity encoding vector corresponding to each element to be labeled form a corresponding element encoding vector; and based on the sentiment type label and the entity three element label of each labeled element in the labeled content information, the sentiment type one-hot encoding vector and the entity encoding vector of the element encoding vector corresponding to the current element are set; finally, the content feature tensor X3 formed by all the obtained element encoding vectors is sent to the third encoder;

[0034] The first encoder is implemented based on an MLP model; the first encoder is used to perform feature encoding processing on the personnel feature vector X1 to obtain the corresponding encoded feature vector X4 and send it to the second feature fusion unit; the vector length of the encoded feature vector X4 is a preset sixth feature dimension C6;

[0035] The second encoder is implemented based on the Bi-LSTM model; the second encoder is used to perform bidirectional temporal feature encoding on the temporal feature sequence X2 to obtain the corresponding encoded feature vector X5 and send it to the second feature fusion unit; the vector length of the encoded feature vector X5 is the preset seventh feature dimension C7; C7 > C6;

[0036] The third encoder is composed of a CNN network, pooling layers, and fully connected layers connected sequentially. The third encoder uses the CNN network to perform feature encoding on the content feature tensor X3 to obtain a corresponding process encoding tensor, and performs max pooling on each feature dimension of the process encoding tensor based on the pooling layers to obtain a corresponding pooled feature vector. Finally, it performs fully connected computation on the pooled feature vector based on the fully connected layers to obtain a corresponding encoded feature vector X6, which is then sent to the second feature fusion unit. The vector length of the encoded feature vector X6 is a preset eighth feature dimension C8; C8 < C7.

[0037] The second feature fusion unit is used to perform vector concatenation processing on the encoded feature vectors X4, X5, and X6 to obtain the corresponding concatenated vector X7, which is then sent to the second hidden layer network. The vector length of the concatenated vector X7 is a preset ninth feature dimension C9, where C9 = C6 + C7 + C8.

[0038] The second hidden layer network consists of three second hidden layers connected sequentially; each second hidden layer consists of a fully connected layer and a ReLU activation function connected sequentially; the second hidden layer network is used to perform feature extraction processing on the concatenated vector X7 to obtain the corresponding feature vector X8, which is then sent to the second output layer; the vector length of the feature vector X8 is a preset tenth feature dimension C. 10 , 1 < C 10 <C9;

[0039] The second output layer is used to perform regression calculation on the labeled quality score based on the feature vector X8 to obtain the corresponding quality prediction score and output it.

[0040] in,

[0041] W2 and b2 are the weight vector parameter and offset scalar parameter of the second output layer, respectively; the vector length of the weight vector parameter W2 is C. 10The tensor shape of both the feature vector X8 and the weight vector parameter W2 is 1×C. 10 ;Sigmoid() is the Sigmoid activation function; S pre-qualify The quality prediction score is given.

[0042] Preferably, the first round of training of the difficulty assessment model and the quality prediction model based on the training dataset specifically includes:

[0043] Extract the first training file and the first difficulty label of each first data record in the training dataset to form a corresponding second data record; extract the first training parameter set and the first quality label of each first data record to form a corresponding third data record; and form a corresponding first dataset from all the obtained second data records; and form a corresponding second dataset from all the obtained third data records.

[0044] The difficulty assessment model is then trained for the first time based on the first dataset.

[0045] The quality prediction model is then trained for the first time based on the second dataset.

[0046] Preferably, the step of selecting typical data records from the training dataset to form the initial validation dataset specifically includes:

[0047] Step 61: Count the total number of records in the training dataset to obtain the corresponding total number of samples N1; and initialize an empty set as the corresponding validation dataset;

[0048] Step 62: Sort all the text elements of each of the first data records in the training dataset into a corresponding text segmentation sequence; encode each text segmentation sequence based on a preset dense vector encoder to obtain a corresponding text encoding tensor; and perform average pooling on each feature dimension of each text encoding tensor to obtain a corresponding high-dimensional feature vector z. j ;

[0049] The dense vector encoder is implemented based on a pre-trained BERT model; the high-dimensional feature vector z j The vector length is the preset feature dimension D1, where D1 > 256; 1 ≤ index j ≤ N1;

[0050] Step 63: First, based on the principal component analysis method, analyze the current N1 high-dimensional feature vectors z. j Feature reduction yields N1 low-dimensional feature vectors. Then construct a similarity matrix E of shape N1×N1, and then use a greedy strategy to filter important features based on the similarity matrix E to obtain the corresponding filter set;

[0051] The filtering set includes one or more of the low-dimensional feature vectors.

[0052] Step 64, select each of the low-dimensional feature vectors in the selected set. The corresponding first data record is added to the verification dataset.

[0053] Preferably, the step of assigning personnel based on the difficulty assessment results of the target file using the difficulty assessment model and a preset personnel information database specifically includes:

[0054] The target file is used as the corresponding annotation file and input into the difficulty assessment model for processing to obtain the corresponding difficulty score;

[0055] The target file's professional field is then identified to obtain the corresponding current field.

[0056] And in the preset correspondence between annotation difficulty and personnel level, the first score range that satisfies the first level range of the current difficulty score correspondence is taken as the corresponding current level range; wherein, the correspondence between annotation difficulty and personnel level includes multiple first correspondences; the first correspondence includes the first score range and the first level range; the first score range is an annotation difficulty score range; the first level range is composed of one or more annotation level levels in the personnel level set;

[0057] The personnel corresponding to the first personnel record in the personnel information database whose first personnel status is idle, whose first personnel level meets the current level range, and whose first domain range meets the current domain range are designated as the personnel to be assigned in this instance.

[0058] Preferably, the step of using the quality prediction model to monitor the annotation quality in real time specifically includes:

[0059] At the start of this annotation process, the first personnel level, the first domain range, and the first task sequence of the first personnel record corresponding to the currently assigned personnel are extracted from the personnel information database and used as the corresponding personnel level, the domain range, and the historical task sequence to form a corresponding annotation personnel information;

[0060] During this annotation process, each annotation action of the assigned personnel is recorded to obtain the corresponding annotation action sequence; and the corresponding set of annotated elements is refreshed based on the real-time annotation results of the assigned personnel.

[0061] During this annotation process, a new annotation content information is periodically formed from the set of elements to be annotated corresponding to the current annotation task and the latest set of annotated elements at a preset first time frequency. The current annotation personnel information, the annotation behavior sequence, and the annotation content information are input into the quality prediction model for processing, and the quality prediction score obtained from this processing is used as the corresponding current score. The model then identifies whether the current score is lower than a preset warning score threshold. If so, a real-time warning is issued to the personnel assigned in this instance.

[0062] Preferably, the step of evaluating the quality of the annotation results according to a preset annotation quality inspection rule library specifically includes:

[0063] According to the various annotation quality inspection rules in the annotation quality inspection rule library, the annotation quality score of the annotated element set obtained in this annotation is calculated to obtain the corresponding quality assessment score; and it is identified whether the quality assessment score is lower than the preset quality assessment threshold; if so, the target file and its corresponding unannotated element set and the annotated element set are combined to form a corresponding manual review data package and sent to the preset manual review interface; if not, the target file and its corresponding unannotated element set and the annotated element set are combined to form a corresponding annotation file data package and sent to the preset annotation via interface.

[0064] Preferably, updating the personnel information database and the preset annotation sample pool based on the data from this annotation process specifically includes:

[0065] The annotation behavior sequence obtained from the behavior records of the personnel assigned in this annotation process is taken as the current annotation behavior sequence; the total processing time of all operations in the current annotation behavior sequence is calculated and the result is taken as the corresponding task execution time; the difficulty score and quality assessment score corresponding to this annotation task are taken as the corresponding task difficulty score and task quality score; the task execution time, task difficulty score, and task quality score obtained in this process are combined to form a corresponding first task; the first task sequence recorded by the first personnel in the personnel information database corresponding to the personnel assigned in this process is taken as the corresponding current task sequence; the first tasks in the current task sequence that exceed the most recent first time period are deleted, and the first tasks obtained in this process are added to the current task sequence;

[0066] The target file and its corresponding set of elements to be annotated for this annotation task are used as the first sample file and the first sample element set to form a corresponding first sample record; and the current first sample record is added to the annotation sample pool.

[0067] Preferably, the step of periodically confirming whether a round of active learning is needed based on the quality assessment of the most recent period specifically includes:

[0068] The number of manual reviews in the most recent time period is periodically counted according to a preset second time frequency to obtain the corresponding first total number of reviews; and when the first total number of reviews exceeds the preset total number of reviews threshold, it is confirmed that a round of active learning is required.

[0069] Preferably, the step of selecting typical data records from the labeled sample pool to update the validation dataset specifically includes:

[0070] Step 12-1: Count the total number of records in the labeled sample pool to obtain the corresponding total number of samples N1; sort all the element texts of each first sample record in the labeled sample pool into a corresponding text segmentation sequence; encode each text segmentation sequence based on a preset dense vector encoder to obtain a corresponding text encoding tensor; and perform average pooling on each feature dimension of each text encoding tensor to obtain a corresponding high-dimensional feature vector z. j ;

[0071] The dense vector encoder is implemented based on a pre-trained BERT model; the high-dimensional feature vector z j The vector length is the preset feature dimension D1, where D1 > 256; 1 ≤ index j ≤ N1;

[0072] Step 12-2: First, based on the principal component analysis method, analyze the current N1 high-dimensional feature vectors z. j Feature reduction yields N1 low-dimensional feature vectors. Then construct a similarity matrix E of shape N1×N1, and then use a greedy strategy to filter important features based on the similarity matrix E to obtain the corresponding filter set;

[0073] The filtering set includes one or more of the low-dimensional feature vectors.

[0074] Step 12-3: The low-dimensional feature vectors in the labeled sample pool and the selected set are compared. The corresponding first sample record is designated as a typical sample record, and all the typical sample records are extracted to form a typical record set; then the labeled sample pool is cleared.

[0075] Step 12-4: Assign multiple senior annotators to each typical sample record in the typical record set, with one of them serving as the first annotator and the others as the first reviewers.

[0076] Steps 12-5: The annotation level of each first annotator is used as a corresponding personnel level. The professional knowledge domains of each first annotator are statistically analyzed, and a corresponding domain range is formed based on the statistical results. The task execution time, task difficulty score, and task quality score of each historical annotation task for each first annotator in the most recent first time period are statistically analyzed, and a corresponding historical task is formed based on the statistical results. All historical tasks in the most recent first time period are combined into a corresponding historical task sequence. Finally, the personnel level, domain range, and historical task sequence corresponding to each first annotator are combined into a corresponding annotation personnel information.

[0077] Steps 12-6 involve each of the first annotators manually annotating the first sample file of the typical sample record they are responsible for, recording each annotation action of the current annotator during the annotation process to obtain the corresponding annotation action sequence, and statistically analyzing the annotation results of the current annotator at the end of the annotation process to obtain the corresponding set of annotated elements.

[0078] Steps 12-7: Each of the first reviewers of each of the typical sample records performs annotation difficulty scoring on the first sample file of the current typical sample record according to the various annotation difficulty evaluation rules in the preset annotation difficulty evaluation rule library to obtain the corresponding third score; and calculates the average score of all the third scores corresponding to each of the first sample files and uses the calculation result as a corresponding first difficulty label.

[0079] Steps 12-8: Each of the first reviewers of each of the typical sample records performs annotation quality scoring on the first sample element set of the current typical sample record according to the annotation quality inspection rules of the annotation quality inspection rule library to obtain the corresponding fourth score; and calculates the average score of all the fourth scores corresponding to each of the first sample element sets and uses the calculation result as a corresponding first quality label.

[0080] Steps 12-9: Each first sample file is treated as a corresponding first training file; the first sample element set and the labeled element set corresponding to each first sample file are combined to form a corresponding labeled content information; the labeled personnel information, the labeled behavior sequence, and the labeled content information corresponding to each first sample file are combined to form a corresponding first training parameter set; the first training file, the first training parameter set, the first difficulty label, and the first quality label corresponding to each typical sample record in the typical record set are combined to form a corresponding first data record; and all the first data records obtained in this step are added to the verification dataset.

[0081] Preferably, the first step is to use principal component analysis to analyze the current N1 high-dimensional feature vectors z. j Feature reduction yields N1 low-dimensional feature vectors. Next, construct a similarity matrix E of shape N1×N1, and then use a greedy strategy to filter important features based on the similarity matrix E to obtain the corresponding filter set, specifically including:

[0082] Step 13-1, using N1 of the high-dimensional feature vectors z j A high-dimensional feature matrix A of shape N1×D1 is constructed; and the mean μ of each column element of the high-dimensional feature matrix A is calculated. col ; and based on each of the aforementioned mean μ col For each column of N1 matrix elements a row,col The corresponding matrix element b is obtained by centering. row,col ; and consisting of N1×D1 matrix elements b row,col The corresponding central feature matrix B is formed; the covariance matrix C of the central feature matrix B is calculated; the characteristic equation of the covariance matrix C is constructed; and the corresponding eigenvalue vector U and feature tensor V are obtained by solving the characteristic equation.

[0083] Each row of the high-dimensional feature matrix A corresponds to a high-dimensional feature vector z. j Each column corresponds to one dimension of the D1-dimensional feature; the high-dimensional feature matrix A includes N1×D1 matrix elements a. row,col 1 ≤ row index row ≤ N1, 1 ≤ column index col ≤ D1;

[0084] The mean μ col The matrix element b row,col The covariance matrix C is:

[0085] b row,col =arow,col -μ col ,

[0086] The shape of the covariance matrix C is D1×D1;

[0087] The characteristic equation of the covariance matrix C is: CV = UV;

[0088] The eigenvalue vector U has a length of D1 and consists of D1 eigenvalues ​​u. x Sort in descending order; 1 ≤ index x ≤ D1; adjacent first and last eigenvalues ​​u x u x+1 The size relationship is u x ≥u x+1 The feature tensor V has a shape of D1×D1, consisting of D1 feature vectors v of length D1. x Composition; and each of the said feature vectors v x Modulus length‖v x || = 1; and for any two eigenvectors v x v y≠x Satisfies orthogonality: 1 ≤ index y ≤ D1;

[0089] Step 13-2: Starting from 128 and going up to 256, search for the first low feature dimension k that satisfies the preset principal component screening conditions based on the feature vector U; and then use the first k feature vectors v in the feature tensor V. 1≤x≤k The extracted features are combined into a projection matrix W of shape D1×k; and based on the projection matrix W, each of the high-dimensional feature vectors z is applied. j A low-dimensional feature vector of length k is obtained by performing low-dimensional vector projection.

[0090] The principal component screening criteria are as follows:

[0091]

[0092] p hold p is the preset percentage threshold. hold ≥95%;

[0093] The low-dimensional feature vector The vector projection method is as follows:

[0094] Step 13-3, using N1 of the aforementioned low-dimensional feature vectors A sample set is constructed; the selection set is initialized to be empty; and the cosine similarity algorithm is used to analyze each pair of low-dimensional feature vectors. The vector similarity is calculated, and the similarity matrix E is composed of the N1×N1 similarities obtained.

[0095] The similarity matrix E includes N1×N1 similarity values ​​e. j,g 1 ≤ index g ≤ N1; similarity e on lines in the matrix j,g＝j If both are 1, the two symmetrical similarities of the vertical bus in the matrix are equal.

[0096] Step 13-4: Calculate N1 importance scores (imc) based on the similarity matrix E. g ; and N1 coverage cov g Initialize to 0;

[0097] Among them, the importance score imc g The calculation method is as follows:

[0098] Step 13-5: First, process each of the low-dimensional feature vectors in the sample set. Corresponding sample gain j Perform the calculation; then denote the index j corresponding to the maximum gain as j. * Then index j * The corresponding low-dimensional feature vector Transfer from the sample set to the filter set, and from index j * The corresponding N1 similarities and coverage The maximum value is selected as the new coverage. Then, based on the importance scores mentioned above, IMC g The corresponding coverage cov g The corresponding effective coverage (ecov) is obtained by weighting. g ;

[0099] in,

[0100]

[0101] Updated ( Before the update );

[0102] ecov g =imc g ×cov g ;

[0103] Step 13-6, calculate the minimum effective coverage min(ecov) gIf the effective coverage threshold is exceeded, proceed to step 13-7; otherwise, check if the sample set is empty. If it is empty, proceed to step 13-7; otherwise, return to step 13-5.

[0104] Step 13-7: Output the obtained filter set as the result of this important feature filtering.

[0105] A second aspect of the present invention provides an apparatus for implementing the annotation quality control method with active learning capability described in the first aspect above, the apparatus comprising: a quality control model preparation module, an annotation task quality control module, and an active learning module;

[0106] The quality control model preparation module is used to construct a difficulty assessment model and a quality prediction model before executing the first annotation task; to construct a training dataset through data collection; to perform the first round of training on the difficulty assessment model and the quality prediction model based on the training dataset; and to select typical data records from the training dataset to form an initial validation dataset.

[0107] The annotation task quality control module is used to, during the execution of each annotation task, take the currently unannotated text file as the target file; allocate annotation personnel based on the difficulty assessment result of the target file according to the difficulty assessment model and the preset personnel information database; have the assigned personnel manually annotate the target file; monitor the annotation quality in real time using the quality prediction model during the annotation process; evaluate the quality of the annotation results according to the preset annotation quality inspection rule library at the end of the annotation process; and update the personnel information database and the preset annotation sample pool based on the annotation process data.

[0108] The active learning module is used to periodically confirm whether an active learning round is needed based on the quality assessment status of the most recent period after the first annotation task is executed; if it is confirmed that it is needed, it selects typical data records from the annotation sample pool to update the validation dataset, and performs an upgrade training on the difficulty assessment model and the quality prediction model based on the updated validation dataset.

[0109] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0110] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0111] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0112] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0113] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for annotation quality control with active learning capabilities. As described above, before executing the first annotation task, this invention constructs and trains a difficulty assessment model for evaluating the difficulty of the annotation file and a quality prediction model for real-time evaluation of annotation quality. During the execution of each annotation task, annotation personnel are assigned based on the difficulty assessment results of the target file using the difficulty assessment model and a personnel information database. The assigned personnel then manually annotate the target file. The quality prediction model monitors the annotation quality in real time during the annotation process. At the end of the annotation, the quality of the annotation results is evaluated according to rules, and the personnel information database and annotation sample pool are updated based on the annotation process data. After executing the first annotation task, the need for a round of active learning is periodically confirmed based on the recent quality assessment status. If necessary, the validation dataset is updated based on the annotation sample pool, and the difficulty assessment model and quality prediction model are upgraded and trained based on the updated validation dataset. Through the embodiments of the present invention, not only is the cycle shortened, efficiency improved, and labor costs reduced, but the real-time performance of quality control is also improved, the flexibility of task allocation is increased, and the matching degree of task allocation is improved. In addition, a self-optimizing and iterative active learning mechanism is also provided. Attached Figure Description

[0114] Figure 1 This is a schematic diagram of a labeling quality control method with active learning capability provided in Embodiment 1 of the present invention;

[0115] Figure 2 A schematic diagram of the modules of the difficulty assessment model provided in Embodiment 1 of the present invention;

[0116] Figure 3 This is a schematic diagram of the modules of the quality prediction model provided in Embodiment 1 of the present invention;

[0117] Figure 4 This is a module structure diagram of a labeling quality control device with active learning capability provided in Embodiment 2 of the present invention;

[0118] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0119] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0120] Embodiment 1 of the present invention provides a labeling quality control method with active learning capability, such as... Figure 1 The schematic diagram of a labeling quality control method with active learning capability provided in Embodiment 1 of the present invention is shown. The method mainly includes the following steps:

[0121] Step 1: Before performing the first annotation task, construct a difficulty assessment model and a quality prediction model; construct a training dataset through data collection; conduct the first round of training on the difficulty assessment model and the quality prediction model based on the training dataset; and select typical data records from the training dataset to form the initial validation dataset.

[0122] Here, the difficulty assessment model of this invention is used to predict the annotation difficulty of the annotation file input to the model and output the corresponding difficulty score. The difficulty score ranges from [0,1]; the annotation file is a text file.

[0123] The quality prediction model in this embodiment of the invention is used to predict the annotation quality of annotation personnel based on the annotation personnel information, annotation behavior sequence, and annotation content information input to the model, and output the corresponding quality prediction score. Wherein:

[0124] 1) Personnel information includes personnel level, domain scope, and historical task sequence: a) Personnel level is a labeling level from a preset set of personnel levels; b) Domain scope consists of one or more knowledge domains from a preset set of knowledge domains; c) Historical task sequence includes multiple historical tasks; historical tasks include task execution time, task difficulty score, and task quality score.

[0125] Here, the personnel level set in this embodiment of the invention consists of multiple levels of labeled level; the level can be expressed as a numerical level or a character level, such as level 1, level 2, level 3, level 4, etc., level A, level B, level C, level D, etc.; for numerical levels, the level is determined by the sorting order of the numerical values ​​from largest to smallest or smallest to largest, and for character levels, the level is determined by the specific sorting order of the characters.

[0126] The knowledge domain set in this embodiment of the invention consists of multiple knowledge domains, such as the medical field and the computer field;

[0127] 2) The annotation behavior sequence is composed of multiple single-step annotation behaviors arranged in chronological order; the single-step annotation behavior includes the annotation operation type and the operation processing time; the annotation operation type includes editing a new annotation, modifying an old annotation, and undoing an old annotation;

[0128] 3) The annotation content information includes a set of elements to be annotated and a set of annotated elements: a) The set of elements to be annotated includes multiple elements to be annotated; each element to be annotated corresponds to a word segment in the annotation file; each element to be annotated includes an element identifier, element domain, element position, element length, and element text; the element domain is a type of knowledge domain in the knowledge domain set; the element position is the position of the first character of the current element in the annotation file; b) The set of annotated elements includes multiple annotated elements; each annotated element corresponds one-to-one with an element to be annotated; each annotated element includes an element identifier, sentiment type annotation, and entity three-element annotation; the sentiment type annotation is a type of sentiment in the preset sentiment type set; the entity three-element annotation includes an entity identifier and three-element type; the three-element type is a type of element in the preset three-element type set, which includes subject element type, relation element type, and object element type;

[0129] Here, the emotion type set in this embodiment of the invention consists of multiple emotion types, such as positive, negative, neutral, etc.

[0130] 4) The score range for the quality prediction score is [0,1].

[0131] In this embodiment of the invention, both the training dataset and the validation dataset consist of multiple first data records. Each first data record includes a first training file, a first training parameter set, a first difficulty label, and a first quality label. The first training parameter set consists of a set of annotator information, annotation behavior sequences, and annotation content information. The first training file is a text file. Each element to be annotated in the first training parameter set corresponds to a word segment in the first training file. The first difficulty label is the corresponding label difficulty score. The first quality label is the corresponding label quality score. The first training file and the first difficulty label are used to train or validate a difficulty assessment model, and the first training parameter set and the first quality label are used to train or validate a quality prediction model.

[0132] like Figure 2 As shown in the schematic diagram of the module of the difficulty assessment model provided in Embodiment 1 of the present invention, the model components of the difficulty assessment model include: a complexity feature recognition unit, a structural feature recognition unit, a domain feature recognition unit, a first feature fusion unit, a first hidden layer network, and a first output layer.

[0133] The difficulty assessment model's input is used to receive labeled files, and its output is used to output the corresponding difficulty score.

[0134] The connection relationships of the model components in the difficulty assessment model are as follows: the input ends of the complexity feature recognition unit, the structural feature recognition unit, and the domain feature recognition unit are respectively connected to the model input end, and the output ends are respectively connected to the first, second, and third input ends of the first feature fusion unit; the output end of the first feature fusion unit is connected to the input end of the first hidden layer network; the output end of the first hidden layer network is connected to the input end of the first output layer; and the output end of the first output layer is connected to the model output end.

[0135] The model component functions of the difficulty assessment model are shown below.

[0136] 1) Complexity Feature Recognition Unit:

[0137] The complexity feature recognition unit is used to recognize the text complexity features corresponding to the labeled file according to the preset text complexity feature recognition rules, and obtain the corresponding complexity feature vector H1, which is then sent to the first feature fusion unit. The vector length of the complexity feature vector H1 is a preset first feature dimension C1, for example, C1 = 64.

[0138] Here, the text complexity feature recognition rule of this embodiment of the invention requires that the complexity features should include at least the following statistical features: total number of characters, total number of words, total number of sentences, professional field range, text Shannon entropy based on n-gram algorithm, term density based on multiple types of preset knowledge dictionaries, etc.

[0139] 2) Structural Feature Recognition Unit:

[0140] The structural feature recognition unit is used to recognize the text structural features of the labeled document according to preset text structural feature recognition rules, obtain the corresponding structural feature vector H2, and send it to the first feature fusion unit. The length of the structural feature vector H2 is a preset second feature dimension C2, for example, C2 = 32.

[0141] Here, the text structure feature recognition rules of this embodiment of the invention require that the structural features should include at least the following statistical features: the document hierarchy of the text, the heading level and paragraph nesting depth, and the distribution of special object elements (such as tables, formulas, etc.) in the text.

[0142] 3) Domain Feature Recognition Unit:

[0143] The domain feature recognition unit is used to recognize the text knowledge domain features of the labeled file according to preset text knowledge domain feature recognition rules, and obtain the corresponding domain feature vector H3, which is then sent to the first feature fusion unit. The vector length of the domain feature vector H3 is a preset third feature dimension C3, for example, C3 = 32.

[0144] Here, the text knowledge domain feature recognition rule of this embodiment of the invention requires that the text content be aligned with the knowledge graphs of multiple preset domains, and that the concept coverage and association strength of each type of knowledge graph be statistically analyzed.

[0145] 4) First Feature Fusion Unit:

[0146] The first feature fusion unit is used to perform vector concatenation on the complexity feature vector H1, the structural feature vector H2, and the domain feature vector H3 to obtain the corresponding concatenated vector H4, which is then sent to the first hidden layer network. The length of the concatenated vector H4 is a preset fourth feature dimension C4, where C4 = C1 + C2 + C3, for example, C4 = 128.

[0147] 5) First hidden layer network:

[0148] The first hidden layer network consists of three first hidden layers connected sequentially; each first hidden layer consists of a fully connected layer and a ReLU activation function connected sequentially.

[0149] The first hidden layer network is used to extract features from the concatenated vector H4 to obtain the corresponding feature vector H5, which is then sent to the first output layer. The length of the feature vector H5 is a preset fifth feature dimension C5, where 1 < C5 < C4, for example, C5 = 32.

[0150] 6) First output layer:

[0151] The first output layer is used to perform regression calculation on the labeled difficulty score based on the feature vector H5 to obtain the corresponding difficulty score and output it.

[0152] Here, the difficulty score is calculated as follows:

[0153]

[0154] Among them, S difficulty For difficulty scoring, W1 and b1 are the weight vector parameter and offset scalar parameter of the first output layer, respectively; the vector length of the weight vector parameter W1 is C5; the tensor shape of the feature vector H5 and the weight vector parameter W1 is 1×C5; Sigmoid() is the Sigmoid activation function.

[0155] like Figure 3 As shown in the schematic diagram of the quality prediction model provided in Embodiment 1 of the present invention, the model components of the quality prediction model include: a personnel feature extraction unit, a temporal feature extraction unit, a content feature extraction unit, a first encoder, a second encoder, a third encoder, a second feature fusion unit, a second hidden layer network, and a second output layer.

[0156] The first input of the quality prediction model is used to receive information about the annotators, the second input is used to receive the sequence of annotation behavior, the third input is used to receive information about the annotation content, and the output is used to output the corresponding quality prediction score.

[0157] The connection relationships of the model components in the quality prediction model are as follows: the inputs of the personnel feature extraction unit, the temporal feature extraction unit, and the content feature extraction unit are connected to the corresponding first model input, second model input, and third model input, respectively, and their outputs are connected to the corresponding first, second, and third generation encoders, respectively; the outputs of the first, second, and third generation encoders are connected to the corresponding first, second, and third inputs of the second feature fusion unit, respectively; the output of the second feature fusion unit is connected to the input of the second hidden layer network; the output of the second hidden layer network is connected to the input of the second output layer; and the output of the second output layer is connected to the model output.

[0158] The functionality of the model components in the quality prediction model is shown below.

[0159] 1) Personnel Feature Extraction Unit:

[0160] The personnel feature extraction unit is used to perform one-hot encoding on the personnel level of the labeled personnel information according to the personnel level set to obtain the corresponding level one-hot encoding vector; and to perform one-hot encoding on the domain range of the labeled personnel information according to the knowledge domain set to obtain the corresponding domain one-hot encoding vector; and to calculate the corresponding historical accuracy according to the historical task sequence of the labeled personnel information; and to send the corresponding personnel feature vector X1 composed of the historical accuracy, the level one-hot encoding vector and the domain one-hot encoding vector to the first encoder.

[0161] Here, the historical accuracy rate is calculated in the embodiments of the present invention as follows:

[0162]

[0163] Where, N his The total number of historical tasks in the historical task sequence, where 1 ≤ task index i ≤ N. his ;l i d i s i These represent the task execution time, task difficulty score, and task quality score for the i-th historical task in the historical task sequence, respectively; p his For historical accuracy.

[0164] 2) Temporal Feature Extraction Unit:

[0165] The temporal feature extraction unit is used to perform one-hot encoding on the annotation operation type of each single-step annotation behavior in the annotation behavior sequence according to the preset operation type set to obtain the corresponding operation type one-hot encoding vector; and to normalize the operation processing time of each single-step annotation behavior to obtain the corresponding normalized time; and to form the corresponding single-step embedding vector by the one-hot encoding vector of each operation type and its corresponding normalized time; and to form the corresponding temporal feature sequence X2 by sorting all the obtained single-step embedding vectors in chronological order and sending it to the second encoder.

[0166] Here, the set of operation types in this embodiment of the invention consists of multiple types of annotation operation types, including at least editing new annotations, modifying old annotations, and canceling old annotations.

[0167] 3) Content Feature Extraction Unit:

[0168] The content feature extraction unit performs one-hot encoding, position embedding encoding, length normalization, and word segmentation embedding encoding on the element domain, element position, element length, and element text of each element to be labeled, respectively, to obtain the corresponding element domain one-hot encoding vector, position embedding encoding vector, normalized length, and word segmentation embedding encoding vector. It also initializes a zero-based sentiment type one-hot encoding vector for each element to be labeled based on the sentiment type set, and a zero-based three-element type one-hot encoding vector for each element to be labeled based on the three-element type set. Finally, it sets a zero-based entity identifier encoding for each element to be labeled, and then... The identifier encoding and its corresponding three-element type one-hot encoding vector form a corresponding entity encoding vector; and the element encoding vector is composed of the element domain one-hot encoding vector, position embedding encoding vector, normalized length, word segmentation embedding encoding vector, sentiment type one-hot encoding vector, and entity encoding vector corresponding to each element to be labeled; and the sentiment type one-hot encoding vector and entity encoding vector of the element encoding vector corresponding to the current element are set based on the sentiment type annotation and entity three-element annotation of each annotated element in the annotated content information; finally, the corresponding content feature tensor X3 composed of all the obtained element encoding vectors is sent to the third encoder.

[0169] 4) First encoder:

[0170] The first encoder is implemented based on the MLP model.

[0171] The first encoder performs feature encoding on the personnel feature vector X1 to obtain the corresponding encoded feature vector X4, which is then sent to the second feature fusion unit. The length of the encoded feature vector X4 is a preset sixth feature dimension C6, for example, C6 = 64.

[0172] 5) Second encoder:

[0173] The second encoder is implemented based on the Bi-LSTM model.

[0174] The second encoder performs bidirectional temporal feature encoding on the temporal feature sequence X2 to obtain the corresponding encoded feature vector X5, which is then sent to the second feature fusion unit. The length of the encoded feature vector X5 is the preset seventh feature dimension C7, where C7 > C6, for example, C7 = 256.

[0175] 6) Third encoder:

[0176] The third encoder consists of a CNN network, pooling layers, and fully connected layers connected sequentially.

[0177] The third encoder uses a CNN network to perform feature encoding on the content feature tensor X3 to obtain the corresponding process encoding tensor. It then performs max pooling on each feature dimension of the process encoding tensor based on a pooling layer to obtain the corresponding pooled feature vector. Finally, it performs fully connected computation on the pooled feature vector to obtain the corresponding encoded feature vector X6, which is sent to the second feature fusion unit. The length of the encoded feature vector X6 is a preset eighth feature dimension C8, where C8 < C7, for example, C8 = 128.

[0178] 7) Second feature fusion unit:

[0179] The second feature fusion unit is used to concatenate the encoded feature vectors X4, X5, and X6 to obtain the corresponding concatenated vector X7, which is then sent to the second hidden layer network. The length of the concatenated vector X7 is a preset ninth feature dimension C9, where C9 = C6 + C7 + C8, for example, C9 = 448.

[0180] 8) Second hidden layer network:

[0181] The second hidden layer network consists of three second hidden layers connected in sequence; each second hidden layer consists of a fully connected layer and a ReLU activation function connected in sequence.

[0182] The second hidden layer is used to extract features from the concatenated vector X7 to obtain the corresponding feature vector X8, which is then sent to the second output layer. The feature vector X8 has a length of the preset tenth feature dimension C. 10 , 1 < C 10 <C9, for example, C 10 =32.

[0183] 9) Second output layer:

[0184] The second output layer is used to perform regression calculation on the labeled quality score based on the feature vector X8 to obtain the corresponding quality prediction score and output it.

[0185] Here, the quality prediction score is calculated as follows:

[0186]

[0187] Among them, S pre-qualify For the quality prediction score, W2 and b2 are the weight vector parameter and offset scalar parameter of the second output layer, respectively; the vector length of the weight vector parameter W2 is C. 10 The tensor shape of both the eigenvector X8 and the weight vector parameter W2 is 1×C. 10 ; Sigmoid() is the Sigmoid activation function.

[0188] Step 1 involves constructing a training dataset through data collection, specifically including:

[0189] Step A1 involves collecting text files from each knowledge domain of the knowledge domain set using big data to obtain the corresponding original file set; recruiting volunteers from multiple annotation levels of the personnel level set to form the corresponding volunteer set; and recruiting multiple senior annotation personnel of the highest level to form the corresponding expert set.

[0190] Here, the original file set of this embodiment of the invention includes multiple original files; each original file is a text file; the volunteer set includes multiple volunteers; the expert set includes multiple senior annotators;

[0191] Step A2 involves assigning each volunteer's annotation level as a corresponding personnel level; statistically analyzing each volunteer's area of ​​expertise and forming a corresponding domain scope based on the statistical results; statistically analyzing the task execution time, task difficulty score, and task quality score of each volunteer's most recent first-period annotation tasks and forming a corresponding historical task based on the statistical results; and forming a corresponding historical task sequence from all the most recent first-period historical tasks; and finally, forming a corresponding annotation personnel information based on each volunteer's personnel level, domain scope, and historical task sequence.

[0192] Here, the duration of the most recent first time period in this embodiment of the invention is a preset duration parameter, such as one hour, one day, one week, or one month. Therefore, the most recent first time period is the most recent one hour, the most recent one day, the most recent one week, or the most recent one month.

[0193] Step A3 involves taking the current file corresponding to each original file in the original file set; performing word segmentation on the current file to obtain the corresponding current word segmentation sequence; removing stop words from the current word segmentation sequence; assigning a corresponding unannotated element to each word segment in the current word segmentation sequence; and forming an unannotated element set from all the obtained unannotated elements. A corresponding current annotation ratio is randomly assigned to each volunteer within a preset annotation ratio range. Each volunteer then manually annotates the current file according to the current annotation ratio. During each annotation process, the annotation actions of the current volunteer are recorded to obtain the corresponding annotation action sequence. At the end of each annotation process, the annotation results of the current volunteer are statistically analyzed to obtain the corresponding annotated element set.

[0194] Here, the annotation ratio range is a pre-set percentage range; for example, the annotation ratio range is [15%, 100%], and the current annotation ratio randomly assigned to a volunteer is 30%, then when the volunteer performs manual annotation on the current file, he only needs to annotate 30% of the elements to be annotated in the file, and the resulting set of annotated elements will only cover 30% of the corresponding set of elements to be annotated.

[0195] Step A4 involves senior annotators assigning annotation difficulty scores to each original file according to the pre-defined annotation difficulty assessment rules in the rule library, obtaining the corresponding first score; and then averaging all the first scores for each original file and using the result as a corresponding first difficulty label.

[0196] Here, the annotation difficulty assessment rule base of this invention includes a number of pre-set annotation difficulty assessment rules. Each annotation difficulty assessment rule is used to select some or all features from complexity, structure, and domain features as rule elements, and to give a series of element combinations and condition judgment rules to output a corresponding score, namely the first score.

[0197] Step A5 involves senior annotators scoring each annotated element set according to the annotation quality inspection rules in the annotation quality inspection rule library to obtain the corresponding second score; and then averaging all the second scores for each annotated element set and using the result as a corresponding first quality label.

[0198] Here, the annotation quality inspection rule library of this invention includes a number of pre-set annotation quality inspection rules. Each annotation quality inspection rule is used to take the set of annotated elements as the evaluation object and give a series of evaluation rules on content consistency and accuracy to score the annotation quality of the evaluation object to obtain the corresponding second score.

[0199] Step A6, and take each current file as a corresponding first training file; and take each set of labeled elements and its corresponding set of unlabeled elements as a corresponding set of labeled content information; and take each set of labeled elements and its corresponding set of labeled personnel information, labeling behavior sequence and labeling content information as a corresponding set of first training parameters; and take each set of first training parameters and its corresponding first training file, first difficulty label and first quality label as a corresponding first data record; and take all the first data records obtained as a corresponding training dataset.

[0200] In step 1, the difficulty assessment model and the quality prediction model are trained for the first time based on the training dataset, specifically including:

[0201] Step B1: Extract the first training file and the first difficulty label of each first data record in the training dataset to form a corresponding second data record; extract the first training parameter set and the first quality label of each first data record to form a corresponding third data record; and form the corresponding first dataset from all the obtained second data records; and form the corresponding second dataset from all the obtained third data records.

[0202] Step B2, and perform the first round of training on the difficulty assessment model based on the first dataset;

[0203] Specifically, it includes: Step B2-1, dividing the first dataset into two sub-datasets based on a preset first segmentation ratio, denoted as the corresponding first training set and first evaluation set;

[0204] Wherein, the first segmentation ratio is a pre-set ratio parameter, such as 8:2; both the first training set and the first evaluation set consist of multiple second data records; the ratio of the total number of records in the first training set to the total number of records in the first evaluation set satisfies the first segmentation ratio;

[0205] Step B2-2: Take each second data record of the first training set as the corresponding current training record; and take the first training file of the current training record as the current annotation file and input it into the difficulty assessment model for processing, and take the difficulty score obtained in this processing as the corresponding first prediction score; and take the first prediction score and the first difficulty label of the current training record as a corresponding first prediction-label pair.

[0206] Step B2-3: Input all the obtained first prediction-label pairs into the preset first model loss function to calculate the corresponding first loss value;

[0207] The first model loss function is implemented based on either the L1 loss function or the L2 loss function.

[0208] Step B2-4: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step B2-5; if it does not, based on the preset first model optimizer, perform a round of modulation on the model parameters of the difficulty assessment model in the direction of minimizing the first model loss function, and return to step B2-2 to continue training after this round of modulation.

[0209] The first loss value range is a pre-set numerical range; the first model optimizer includes the Adam optimizer and the SGD optimizer.

[0210] Steps B2-5: Take each second data record of the first evaluation set as the corresponding current evaluation record; input the first training file of the current evaluation record as the current annotation file into the difficulty evaluation model for processing, and take the difficulty score obtained in this processing as the corresponding second prediction score; and form a corresponding second prediction-label pair by the second prediction score and the first difficulty label of the current evaluation record; and input all the obtained second prediction-label pairs into the preset first model evaluation function to calculate the corresponding first evaluation value.

[0211] The first model evaluation function is implemented based on the MAE function, MSE function, or RMSE function.

[0212] Step B2-6: Identify whether the first evaluation value meets the preset first evaluation value range; if not, return to step B2-1 to continue training; if it meets the range, confirm that the first round of training of the difficulty evaluation model is over.

[0213] The first evaluation value range is a pre-set numerical range;

[0214] Step B3, and perform the first round of training on the quality prediction model based on the second dataset;

[0215] Specifically, it includes: Step B3-1, dividing the second dataset into two subsets based on a preset second segmentation ratio, denoted as the corresponding second training set and second evaluation set;

[0216] Wherein, the second segmentation ratio is a pre-set numerical range; both the second training set and the second evaluation set consist of multiple third data records; the ratio of the total number of records in the second training set to the total number of records in the second evaluation set satisfies the second segmentation ratio;

[0217] Step B3-2: Take each third data record of the second training set as the corresponding current training record; input the annotation personnel information, annotation behavior sequence and annotation content information of the first training parameter set of the current training record into the quality prediction model for processing, and take the quality prediction score obtained in this processing as the corresponding third prediction score; and form a corresponding third prediction-label pair by the third prediction score and the first quality label of the current training record.

[0218] Step B3-3: Input all the obtained third prediction-label pairs into the preset second model loss function to calculate the corresponding second loss value;

[0219] The second model loss function is implemented based on either the L1 loss function or the L2 loss function.

[0220] Step B3-4: Identify whether the second loss value meets the preset range of the second loss value; if it does, proceed to step B3-5; if it does not, based on the preset second model optimizer, perform a round of modulation on the model parameters of the quality prediction model in the direction of minimizing the second model loss function, and return to step B3-2 to continue training after this round of modulation.

[0221] The second loss value range is a pre-set numerical range; the second model optimizer includes the Adam optimizer and the SGD optimizer.

[0222] Steps B3-5: Each third data record in the second evaluation set is used as the corresponding current evaluation record; the annotation personnel information, annotation behavior sequence, and annotation content information of the first training parameter set of the current evaluation record are input into the quality prediction model for processing, and the quality prediction score obtained in this processing is used as the corresponding fourth prediction score; a corresponding fourth prediction-label pair is formed by the fourth prediction score and the first quality label of the current evaluation record; and all the obtained fourth prediction-label pairs are input into the preset second model evaluation function to calculate the corresponding second evaluation value.

[0223] The second model evaluation function is implemented based on the MAE function, MSE function, or RMSE function.

[0224] Step B3-6: Identify whether the second evaluation value meets the preset range of the second evaluation value; if it does not meet the range, return to step B3-1 to continue training; if it does meet the range, confirm that the first round of training of the quality prediction model has ended.

[0225] The second evaluation value range is a pre-set numerical range.

[0226] In step 1, typical data records are selected from the training dataset to form the initial validation dataset, specifically including:

[0227] Specifically, this includes: Step C1, counting the total number of records in the training dataset to obtain the corresponding total number of samples N1; and initializing an empty set as the corresponding validation dataset;

[0228] Step C2 involves sequentially sorting all elements of the first data record in the training dataset to form a corresponding text segmentation sequence; encoding each text segmentation sequence using a pre-defined dense vector encoder to obtain a corresponding text encoding tensor; and performing average pooling on each feature dimension of each text encoding tensor to obtain a corresponding high-dimensional feature vector z. j ;

[0229] Here, the dense vector encoder in this embodiment of the invention is implemented based on a pre-trained BERT model; the high-dimensional feature vector z j The vector length is the preset feature dimension D1, where D1 > 256, for example, D1 = 768; 1 ≤ index j ≤ N1;

[0230] Step C3: First, based on the principal component analysis method, analyze the current N1 high-dimensional eigenvectors z. j Feature reduction yields N1 low-dimensional feature vectors. Then construct a similarity matrix E of shape N1×N1, and then use a greedy strategy to filter important features based on the similarity matrix E to obtain the corresponding filter set;

[0231] The selection set includes one or more low-dimensional feature vectors.

[0232] Step C4 involves filtering each low-dimensional feature vector in the set. The corresponding first data record is added to the validation dataset.

[0233] In step C3 above, the current N1 high-dimensional eigenvectors z are first analyzed using the principal component analysis method. j Feature reduction yields N1 low-dimensional feature vectors. Next, construct a similarity matrix E of shape N1×N1, and then use a greedy strategy to filter important features based on the similarity matrix E to obtain the corresponding filter set, specifically including:

[0234] Step C3-1, using N1 high-dimensional feature vectors z j Construct a high-dimensional feature matrix A of shape N1×D1; and calculate the mean μ of each column element of the high-dimensional feature matrix A. col ; and based on each mean μ col For each column of N1 matrix elements a row,col The corresponding matrix element b is obtained by centering. row,col; and consists of N1×D1 matrix elements b row,col The corresponding central feature matrix B is formed; the covariance matrix C of the central feature matrix B is calculated; the characteristic equation of the covariance matrix C is constructed; and the corresponding eigenvalue vector U and characteristic tensor V are obtained by solving the characteristic equation.

[0235] here:

[0236] 1) The high-dimensional feature matrix A in this embodiment of the invention includes N1×D1 matrix elements a row,col 1 ≤ row index row ≤ N1, 1 ≤ column index col ≤ D1;

[0237] Specifically, each row of the high-dimensional feature matrix A corresponds to a high-dimensional feature vector z. j Each column corresponds to one dimension in the D1-dimensional feature;

[0238] 2) Mean μ col Matrix element b row,col The covariance matrix C is:

[0239]

[0240] b row,col =a row,col -μ col ,

[0241]

[0242] The shape of the covariance matrix C is D1×D1;

[0243] 3) The characteristic equation of the covariance matrix C is:

[0244] CV = UV;

[0245] The eigenvalue vector U has a length of D1, consisting of D1 eigenvalues ​​u. x Sort in descending order; 1 ≤ index x ≤ D1; adjacent first and last eigenvalues ​​u x u x+1 The size relationship is u x ≥u x+1 ;

[0246] The feature tensor V has a shape of D1×D1, consisting of D1 feature vectors v of length D1. x Composition; and each feature vector v x Modulus length‖v x || = 1; and for any two eigenvectors v x v y≠x Satisfies orthogonality: 1 ≤ index y ≤ D1;

[0247] Step C3-2: Starting from 128 and going up to 256, search for the first low feature dimension k that satisfies the preset principal component selection criteria based on the feature vector U; and then combine the first k feature vectors v in the feature tensor V. 1≤x≤k The extracted features are combined into a projection matrix W of shape D1×k; and based on the projection matrix W, each high-dimensional feature vector z is applied. j Projecting a low-dimensional vector yields a low-dimensional feature vector of length k.

[0248] here:

[0249] 1) The principal component screening conditions in this embodiment of the invention are:

[0250]

[0251] Where, p hold p is the preset percentage threshold. hold ≥95%;

[0252] 2) Low-dimensional feature vectors The vector projection method is as follows:

[0253]

[0254] Step C3-3, consisting of N1 low-dimensional feature vectors A sample set is constructed; the selection set is initialized to empty; and the cosine vector similarity algorithm is used to select each pair of low-dimensional feature vectors. The vector similarity is calculated, and the resulting N1×N1 similarity values ​​form a similarity matrix E.

[0255] Here, the similarity matrix E in this embodiment of the invention includes N1×N1 similarity values ​​e. j,g 1 ≤ index g ≤ N1; similarity e on lines in the matrix j,g＝j If both are 1, the two symmetric similarities of the vertical bus in the matrix are equal, for example, e. 2,3 =e 3,2 ;

[0256] Step C3-4: Calculate N1 importance scores (imc) based on the similarity matrix E. g ; and N1 coverage cov g Initialize to 0;

[0257] Here, the importance score of the embodiments of the present invention is imc. g The calculation method is as follows:

[0258]

[0259] Step C3-5: First, process each low-dimensional feature vector in the sample set. Corresponding sample gain j Perform the calculation; then denote the index j corresponding to the maximum gain as j. * Then index j * The corresponding low-dimensional feature vector Transfer from the sample set to the selection set, and from index j * The corresponding N1 similarities and coverage The maximum value is selected as the new coverage. Then, based on the importance scores (IMC) g Its corresponding coverage cov g The corresponding effective coverage (ecov) is obtained by weighting. g ;

[0260] here:

[0261] 1) Sample gain j The calculation method is as follows:

[0262]

[0263] 2) The coverage update method for maximum gain is as follows:

[0264] Updated ( Before the update );

[0265] 3) The effective coverage is calculated as follows:

[0266] ecov g =imc g ×cov g ;

[0267] Step C3-6, for the minimum effective coverage min(ecov) g If the effective coverage threshold is exceeded, proceed to step C3-7; otherwise, check if the sample set is empty. If it is empty, proceed to step C3-7; otherwise, return to step C3-5.

[0268] Here, the effective coverage threshold is a pre-set threshold parameter;

[0269] Step C3-7: Output the obtained filter set as the result of this important feature selection.

[0270] Step 2: During the execution of each annotation task, the text file to be annotated is taken as the target file; annotation personnel are assigned based on the difficulty assessment result of the target file according to the difficulty assessment model and the preset personnel information database; the assigned personnel manually annotate the target file; during this annotation process, the annotation quality is monitored in real time using a quality prediction model; at the end of this annotation, the annotation result is evaluated according to the preset annotation quality inspection rule library; and the personnel information database and the preset annotation sample pool are updated based on the data of this annotation process.

[0271] Specifically, this includes: Step 21, during the execution of each annotation task, the text file to be annotated is used as the target file;

[0272] Step 22, and based on the difficulty assessment model, the difficulty assessment results of the target file and the preset personnel information database are used to label and assign personnel;

[0273] The personnel information database includes multiple first-person records; each first-person record corresponds to one annotator; each first-person record includes a first-person identifier, first name, first-person level, first-person status, first domain scope, and first task sequence; the first-person identifier, first name, and first-person level are respectively the unique identifier, real name, and annotation level of the current annotator; the first-person status includes idle and working status; the first domain scope is the professional domain scope of the current annotator, consisting of one or more knowledge domains from a knowledge domain set; the first task sequence is composed of multiple first tasks completed by the current annotator in the most recent first time period, ordered chronologically, with each first task corresponding to a historical annotation task; each first task includes task execution time, task difficulty score, and task quality score; the task difficulty score is the annotation difficulty score of the current annotation task; the task quality score is the annotation quality score of the current annotation task;

[0274] Specifically, this includes: Step 221, inputting the target file as the corresponding annotation file into the difficulty assessment model for processing to obtain the corresponding difficulty score;

[0275] Step 222, and identify the professional field of the target file to obtain the corresponding current field;

[0276] Step 223, and take the first level range in the preset correspondence between difficulty level and personnel level, where the first score range satisfies the first correspondence between the current difficulty score and the first level range, as the corresponding current level range;

[0277] The correspondence between annotation difficulty and personnel level includes multiple first correspondences; each first correspondence includes a first score range and a first level range; the first score range is an annotation difficulty scoring range; and the first level range consists of one or more annotation level levels from the personnel level set.

[0278] Step 224: The personnel in the personnel information database whose first personnel status is idle, whose first personnel level meets the current level range, and whose first domain range meets the current domain range are the marked personnel for this allocation.

[0279] Step 23, and the personnel assigned in this session manually annotate the target file;

[0280] Specifically, this includes: performing word segmentation on the target file to obtain the corresponding current word segmentation sequence; performing stop word deletion on the current word segmentation sequence; setting a corresponding unannotated element for each word in the current word segmentation sequence, and forming a corresponding unannotated element set from all the obtained unannotated elements; and having the current assignment personnel manually annotate the current unannotated element set.

[0281] Step 24, and during this annotation process, use a quality prediction model to monitor the annotation quality in real time;

[0282] Specifically, it includes: Step 241, at the beginning of this annotation process, extracting the first personnel level, first domain range and first task sequence of the first personnel record corresponding to the personnel assigned in the personnel information database as the corresponding personnel level, domain range and historical task sequence to form a corresponding annotation personnel information;

[0283] Step 242: During this annotation process, record each annotation action of the assigned personnel to obtain the corresponding annotation action sequence; and refresh the corresponding set of annotated elements based on the real-time annotation results of the assigned personnel.

[0284] Step 243: During this annotation process, a new annotation content information is periodically composed of the set of elements to be annotated and the latest set of annotated elements corresponding to this annotation task, according to a preset first time frequency. The current annotation personnel information, annotation behavior sequence, and annotation content information are input into the quality prediction model for processing, and the quality prediction score obtained from this processing is used as the corresponding current score. It is also used to identify whether the current score is lower than the preset warning score threshold. If so, a real-time warning is issued to the personnel assigned in this task.

[0285] Here, the first time frequency is a preset frequency parameter; the warning score threshold is a preset threshold parameter;

[0286] Step 25, and at the end of this annotation, perform a quality assessment of the annotation results according to the preset annotation quality inspection rule library;

[0287] Specifically, this includes: scoring the labeled element set obtained from this annotation according to the various annotation quality inspection rules in the annotation quality inspection rule library to obtain the corresponding quality assessment score; identifying whether the quality assessment score is lower than the preset quality assessment threshold; if so, sending the target file and its corresponding unannotated element set and labeled element set as a corresponding manual review data package to the preset manual review interface; if not, sending the target file and its corresponding unannotated element set and labeled element set as a corresponding annotation file data package to the preset annotation through the interface.

[0288] Here, the quality assessment threshold is a pre-set threshold parameter; the manual review interface and the annotation pass interface are two types of pre-set processing interfaces; if the quality assessment score is greater than or equal to the quality assessment threshold, it means that the annotation quality meets the standard, and the next step is to send the relevant content of the annotation task, i.e., the annotation file data package, to the interface receiver for subsequent processing (such as archiving) through the annotation pass interface; if the quality assessment score is less than the quality assessment threshold, it means that the annotation quality does not meet the standard, and the next step is to send the relevant content of the annotation task, i.e., the manual review data package, to the interface receiver through the manual review interface for other personnel to re-annotate, modify, and review; and each call to the manual review interface means that a quality failure event has occurred, and the corresponding parameters such as the occurrence time of the event and the total number of events in the period are recorded, so that the monitoring of such events can be used to perform trend analysis on the quality assessment status of the most recent period; the duration of the most recent period in this embodiment of the invention is a pre-set duration parameter, such as one hour, one day, one week, one month, etc., and the corresponding most recent period is the most recent hour, the most recent day, the most recent week, and the most recent month;

[0289] Step 26, and update the personnel information database and the preset annotation sample pool based on the data from this annotation process;

[0290] The labeled sample pool consists of multiple first sample records; each first sample record includes a first sample file and a first sample element set; the first sample file is a text file; and the first sample element set is a set of elements to be labeled.

[0291] Specifically, this includes: Step 261, taking the annotation behavior sequence obtained from the behavior records of the personnel assigned in this annotation process as the current annotation behavior sequence; calculating the sum of the processing time of all operations in the current annotation behavior sequence and taking the result as the corresponding task execution time; taking the difficulty score and quality assessment score corresponding to this annotation task as the corresponding task difficulty score and task quality score; and forming a corresponding first task by combining the task execution time, task difficulty score, and task quality score obtained this time; taking the first task sequence of the first personnel record corresponding to the personnel assigned in this time in the personnel information database as the corresponding current task sequence; deleting the first task in the current task sequence that exceeds the most recent first time period and adding the first task obtained this time to the current task sequence;

[0292] Step 262: The target file of this annotation task and its corresponding set of elements to be annotated are used as the first sample file and the first sample element set to form a corresponding first sample record; and the current first sample record is added to the annotation sample pool.

[0293] Step 3: After performing the first annotation task, periodically confirm whether an active learning round is needed based on the quality assessment status of the most recent period; if it is confirmed that it is needed, select typical data records from the annotation sample pool to update the validation dataset, and perform an upgrade training on the difficulty assessment model and the quality prediction model based on the updated validation dataset.

[0294] Specifically, this includes: Step 31, after executing the first annotation task, periodically confirming whether a round of active learning is needed based on the quality assessment of the most recent period;

[0295] Specifically, this includes: periodically counting the number of manual reviews in the most recent time period according to a preset second time frequency to obtain the corresponding first total number of reviews; and confirming that a round of active learning is needed when the first total number of reviews exceeds a preset total number of reviews threshold.

[0296] Here, the second time frequency is a preset frequency parameter, and the total number of reviews threshold is a preset threshold parameter;

[0297] Step 32: If necessary, select typical data records from the labeled sample pool to update the validation dataset, and perform an upgrade training on the difficulty assessment model and the quality prediction model based on the updated validation dataset.

[0298] Specifically, this includes: Step 321, selecting typical data records from the labeled sample pool to update the validation dataset;

[0299] Specifically, this includes: step 3211, counting the total number of records in the labeled sample pool to obtain the corresponding total number of samples N1; sorting all the text elements of each first sample record in the labeled sample pool into a corresponding text segmentation sequence; encoding each text segmentation sequence based on a preset dense vector encoder to obtain the corresponding text encoding tensor; and performing average pooling on each feature dimension of each text encoding tensor to obtain a corresponding high-dimensional feature vector z. j ;

[0300] Among them, the high-dimensional feature vector z j The vector length is the preset feature dimension D1, where D1 > 256; 1 ≤ index j ≤ N1;

[0301] Step 3212: First, based on the principal component analysis method, analyze the current N1 high-dimensional eigenvectors z. j Feature reduction yields N1 low-dimensional feature vectors. Then construct a similarity matrix E of shape N1×N1, and then use a greedy strategy to filter important features based on the similarity matrix E to obtain the corresponding filter set;

[0302] The selection set includes one or more low-dimensional feature vectors.

[0303] Here, the processing procedure in the current step 3212 is completely consistent with the processing procedure in step C3 above, and will not be described in detail here;

[0304] Step 3213: The low-dimensional feature vectors from the labeled sample pool and the selected set are... The first corresponding sample record is designated as the typical sample record, and all typical sample records are extracted to form a typical record set; then the labeled sample pool is cleared.

[0305] Step 3214: Assign multiple senior annotators to each typical sample record in the typical record set, with one of them serving as the first annotator and the others as the first reviewers.

[0306] Step 3215: The annotation level of each first annotator is taken as a corresponding personnel level. The professional knowledge domain of each first annotator is statistically analyzed and a corresponding domain range is formed based on the statistical results. The task execution time, task difficulty score, and task quality score of each historical annotation task of each first annotator in the most recent first time period are statistically analyzed and a corresponding historical task is formed based on the statistical results. All historical tasks in the most recent first time period are combined into a corresponding historical task sequence. Finally, the personnel level, domain range, and historical task sequence of each first annotator are combined into a corresponding annotation personnel information.

[0307] Step 3216: Each first annotator manually annotates the first sample file of the typical sample record they are responsible for, and records each annotation action of the current annotator during the annotation process to obtain the corresponding annotation action sequence. At the end of the annotation, the annotation results of the current annotator are statistically analyzed to obtain the corresponding set of annotated elements.

[0308] Step 3217: Each first reviewer of each typical sample record scores the first sample file of the current typical sample record according to the various annotation difficulty evaluation rules in the preset annotation difficulty evaluation rule library to obtain the corresponding third score; and calculates the average score of all the third scores corresponding to each first sample file and uses the calculation result as a corresponding first difficulty label.

[0309] Step 3218: Each first reviewer of each typical sample record performs annotation quality scoring on the first sample element set of the current typical sample record according to the annotation quality inspection rule library to obtain the corresponding fourth score; and calculates the average score of all the fourth scores corresponding to each first sample element set and uses the calculation result as a corresponding first quality label.

[0310] Step 3219: Each first sample file is treated as a corresponding first training file; the first sample element set and the labeled element set corresponding to each first sample file are combined to form a corresponding labeled content information; the annotator information, annotation behavior sequence, and labeled content information corresponding to each first sample file are combined to form a corresponding first training parameter set; and the first training file, first training parameter set, first difficulty label, and first quality label corresponding to each typical sample record in the typical record set are combined to form a corresponding first data record; and all the first data records obtained in this step are added to the validation dataset.

[0311] Step 322, and upgrade the difficulty assessment model and the quality prediction model once each based on the updated validation dataset;

[0312] Specifically, this includes: step 3221, extracting the first training file and first difficulty label of each first data record in the updated validation dataset to form a corresponding fourth data record; extracting the first training parameter set and first quality label of each first data record to form a corresponding fifth data record; and forming a corresponding third dataset from all the obtained fourth data records; and forming a corresponding fourth dataset from all the obtained fifth data records.

[0313] Step 3222, and perform a round of fine-tuning training on the difficulty assessment model based on the third dataset;

[0314] Here, the fine-tuning method for the difficulty assessment model is basically the same as the first round of training method for the difficulty assessment model mentioned above. The difference is that the first round of training in the previous article modulates all the model parameters of the difficulty assessment model, while the fine-tuning training in the current step only modulates some of the model parameters of the difficulty assessment model, such as the model parameters of the last hidden layer + the first output layer of the first hidden layer network.

[0315] Step 3223, and then perform a round of fine-tuning training on the quality prediction model based on the fourth dataset.

[0316] Here, the fine-tuning method for the quality prediction model is basically the same as the first round of training method for the quality prediction model mentioned above. The difference is that the first round of training in the previous article modulates all the parameters of the quality prediction model, while the fine-tuning training in the current step only modulates some of the model parameters of the quality prediction model, such as the model parameters of the last hidden layer + the second output layer of the second hidden layer network.

[0317] Figure 4 This is a module structure diagram of a labeling quality control device with active learning capability provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 4 As shown, the device includes: a quality control model preparation module 201, a labeling task quality control module 202, and an active learning module 203.

[0318] The quality control model preparation module 201 is used to build a difficulty assessment model and a quality prediction model before performing the first annotation task; to build a training dataset through data collection; to perform the first round of training on the difficulty assessment model and the quality prediction model based on the training dataset; and to select typical data records from the training dataset to form the initial validation dataset.

[0319] The annotation task quality control module 202 is used to, during the execution of each annotation task, take the text file to be annotated as the target file; allocate annotation personnel based on the difficulty assessment result of the target file and the preset personnel information database based on the difficulty assessment model; have the assigned personnel manually annotate the target file; use the quality prediction model to monitor the annotation quality in real time during the annotation process; evaluate the quality of the annotation results according to the preset annotation quality inspection rule library at the end of the annotation process; and update the personnel information database and the preset annotation sample pool based on the data of the annotation process.

[0320] The active learning module 203 is used to periodically confirm whether an active learning round is needed based on the quality assessment status of the most recent period after the first annotation task is performed. If it is confirmed that an active learning round is needed, typical data records are selected from the annotation sample pool to update the validation dataset, and the difficulty assessment model and the quality prediction model are upgraded and trained based on the updated validation dataset.

[0321] The annotation quality control device with active learning capability provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0322] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the quality control model preparation module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0323] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0324] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0325] Figure 5 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 5 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0326] exist Figure 5The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include Non-Volatile Memory, such as at least one disk storage device.

[0327] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0328] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0329] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for annotation quality control with active learning capabilities. As described above, before executing the first annotation task, this invention constructs and trains a difficulty assessment model for evaluating the difficulty of the annotation file and a quality prediction model for real-time evaluation of annotation quality. During the execution of each annotation task, annotation personnel are assigned based on the difficulty assessment results of the target file using the difficulty assessment model and a personnel information database. The assigned personnel then manually annotate the target file. The quality prediction model monitors the annotation quality in real time during the annotation process. At the end of the annotation, the quality of the annotation results is evaluated according to rules, and the personnel information database and annotation sample pool are updated based on the annotation process data. After executing the first annotation task, the need for a round of active learning is periodically confirmed based on the recent quality assessment status. If necessary, the validation dataset is updated based on the annotation sample pool, and the difficulty assessment model and quality prediction model are upgraded and trained based on the updated validation dataset. Through the embodiments of the present invention, not only is the cycle shortened, efficiency improved, and labor costs reduced, but the real-time performance of quality control is also improved, the flexibility of task allocation is increased, and the matching degree of task allocation is improved. In addition, a self-optimizing and iterative active learning mechanism is also provided.

[0330] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0331] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for label quality control with active learning capability, characterized in that, The method comprises: Before performing the first annotation task, a difficulty evaluation model and a quality prediction model are constructed, training data sets are constructed through data collection, the difficulty evaluation model and the quality prediction model are trained for the first time based on the training data sets, and a typical data record is selected from the training data sets to form an initial verification data set; In the process of performing each annotation task, a text file to be annotated is taken as a target file, annotation personnel are assigned based on the difficulty evaluation result of the target file and a preset personnel information library through the difficulty evaluation model, the target file is manually annotated by the assigned personnel, the quality prediction model is used to monitor the annotation quality in real time in the annotation process, the annotation result is quality evaluated according to a preset annotation quality inspection rule library when the annotation process ends, and the personnel information library and a preset annotation sample pool are updated based on the annotation process data; After performing the first annotation task, it is confirmed whether a round of active learning is needed according to the quality evaluation condition of the recent period, if it is confirmed that the active learning is needed, a typical data record is selected from the annotation sample pool to update the verification data set, and the difficulty evaluation model and the quality prediction model are respectively upgraded and trained once based on the updated verification data set; The difficulty evaluation model is used for predicting the annotation difficulty of an annotation file input by the model and outputting a corresponding difficulty score, and the quality prediction model is used for predicting the annotation quality of an annotation personnel according to annotation personnel information, an annotation behavior sequence and annotation content information input by the model and outputting a corresponding quality prediction score. The difficulty evaluation model comprises a complexity feature recognition unit, a structure feature recognition unit, a domain feature recognition unit, a first feature fusion unit, a first hidden layer network and a first output layer, the complexity feature recognition unit, the structure feature recognition unit and the domain feature recognition unit are respectively used for recognizing the text complexity, the text structure and the text knowledge domain feature of the annotation file, the first feature fusion unit is used for vector splicing of the three types of features, the first hidden layer network is used for feature extraction of the spliced vector, and the first output layer is used for regression calculation of the annotation difficulty score according to the extracted features of the hidden layer network. The quality prediction model comprises a personnel feature extraction unit, a time sequence feature extraction unit, a content feature extraction unit, a first encoder, a second encoder, a third encoder, a second feature fusion unit, a second hidden layer network, and a second output layer; the personnel feature extraction unit, the time sequence feature extraction unit, and the content feature extraction unit are respectively used for performing feature extraction on a personnel feature vector of the labeled personnel information, a time sequence feature sequence of the labeled behavior sequence, and a content feature tensor of the labeled content information; the first, second, and third encoders are respectively used for performing feature encoding on the personnel feature vector, the time sequence feature sequence, and the content feature tensor; the second feature fusion unit is used for performing vector splicing on the three types of encoded features; the second hidden layer network is used for performing feature extraction on the spliced vector; and the second output layer is used for performing regression calculation on the labeled quality score according to the extracted features of the hidden layer network.

2. The active learning capable annotation quality control method according to claim 1, characterized in that, the difficulty score has a score interval of [0, 1]; the annotation file is a text file; the quality prediction score has a score interval of [0, 1]; the labeled personnel information comprises a personnel level, a domain range, and a historical task sequence; the personnel level is a type of annotation level in a preset personnel level set; the domain range comprises one or more types of knowledge domains in a preset knowledge domain set; the historical task sequence comprises a plurality of historical tasks; and the historical task comprises a task execution duration, a task difficulty score, and a task quality score; the labeled behavior sequence is sorted in chronological order by a plurality of single-step annotation behaviors; the single-step annotation behavior comprises an annotation operation type and an operation processing duration; and the annotation operation type comprises adding a new annotation, modifying an old annotation, and canceling an old annotation; the labeled content information comprises a set of elements to be annotated and a set of annotated elements; the set of elements to be annotated comprises a plurality of elements to be annotated; each element to be annotated corresponds to a segmented word in the annotation file; and the element to be annotated comprises an element identifier, an element domain, an element position, an element length, and an element text; the element domain is a type of knowledge domain in the knowledge domain set; and the element position is the position of the first character of the current element in the annotation file; the set of annotated elements comprises a plurality of annotated elements; the annotated elements correspond one-to-one to the elements to be annotated; the annotated elements comprise the element identifier, an emotion type annotation, and an entity three-element annotation; the emotion type annotation is a type of emotion in a preset emotion type set; and the entity three-element annotation comprises an entity identifier and a three-element type; the three-element type is a type of element in a preset three-element type set; and the three-element type set comprises a subject element type, a relationship element type, and an object element type; the training data set and the validation data set each comprise a plurality of first data records; and the first data record comprises a first training file, a first training parameter set, a first difficulty label, and a first quality label. The first training parameter set is composed of a set of the annotator information, the annotation behavior sequence, and the annotation content information; the first training file is a text file; each of the to-be-annotated elements in the first training parameter set corresponds to a segmented word of the first training file; the first difficulty label is a corresponding label difficulty score; the first quality label is a corresponding label quality score; the first training file and the first difficulty label are used for training or verifying the difficulty evaluation model, and the first training parameter set and the first quality label are used for training or verifying the quality prediction model; The personnel information library includes a plurality of first personnel records; each of the first personnel records corresponds to an annotator; the first personnel record includes a first personnel identifier, a first name, a first personnel level, a first personnel state, a first domain range, and a first task sequence; the first personnel identifier, the first name, and the first personnel level are respectively a unique identity, a real name, and an annotation level of a current annotator; the first personnel state includes an idle state and a working state; the first domain range is a professional domain range of a current annotator, which is composed of one or more knowledge domains of a set of knowledge domains; the first task sequence is sorted in chronological order by a plurality of first tasks completed by a current annotator in a first time period; each of the first tasks corresponds to a historical annotation task; the first task includes a task execution duration, a task difficulty score, and a task quality score; the task difficulty score is an annotation difficulty score of a current annotation task; and the task quality score is an annotation quality score of a current annotation task. The annotation sample pool includes a plurality of first sample records; each of the first sample records includes a first sample file and a first sample element set; the first sample file is a text file; and the first sample element set is a set of to-be-annotated elements.

3. The annotation quality control method with active learning capability according to claim 2, wherein The complexity feature recognition unit is configured to recognize the text complexity features corresponding to the annotation file according to a preset text complexity feature recognition rule to obtain a corresponding complexity feature vector H1, and send the complexity feature vector H1 to the first feature fusion unit; the vector length of the complexity feature vector H1 is a preset first feature dimension C1. The structure feature recognition unit is configured to recognize the text structure features of the annotation file according to a preset text structure feature recognition rule to obtain a corresponding structure feature vector H2, and send the structure feature vector H2 to the first feature fusion unit; the vector length of the structure feature vector H2 is a preset second feature dimension C2. The domain feature recognition unit is configured to recognize the text knowledge domain features of the annotation file according to a preset text knowledge domain feature recognition rule to obtain a corresponding domain feature vector H3, and send the domain feature vector H3 to the first feature fusion unit; the vector length of the domain feature vector H3 is a preset third feature dimension C3. The first feature fusion unit is configured to perform vector splicing processing on the complexity feature vector H1, the structure feature vector H2, and the field feature vector H3 to obtain a corresponding splicing vector H4, and send the splicing vector H4 to the first hidden layer network; the vector length of the splicing vector H4 is a preset fourth feature dimension C4, C4=C1+C2+C3; The first hidden layer network is sequentially connected by three first hidden layers; each first hidden layer is sequentially connected by a full connection layer and a ReLU activation function; the first hidden layer network is configured to perform feature extraction processing on the splicing vector H4 to obtain a corresponding feature vector H5, and send the feature vector H5 to the first output layer; the vector length of the feature vector H5 is a preset fifth feature dimension C5, 1 The first output layer is configured to perform regression calculation on the labeled difficulty score according to the feature vector H5 to obtain a corresponding difficulty score and output the difficulty score; wherein, W1, b1 are a weight vector parameter and an offset scalar parameter of the first output layer respectively; the vector length of the weight vector parameter W1 is C5; the tensor shape of the feature vector H5 and the weight vector parameter W1 are both 1xC5; Sigmoid() is a Sigmoid activation function; S difficulty is the difficulty score.

4. The labeling quality control method with active learning capability according to claim 2, characterized in that, The personnel feature extraction unit is configured to perform one-hot encoding on the personnel level of the labeling personnel information according to the personnel level set to obtain a corresponding level one-hot encoding vector, perform one-hot encoding on the field range of the labeling personnel information according to the knowledge field set to obtain a corresponding field one-hot encoding vector, calculate a corresponding historical accuracy rate according to the historical task sequence of the labeling personnel information, and compose a corresponding personnel feature vector X1 from the historical accuracy rate, the level one-hot encoding vector, and the field one-hot encoding vector, and send the personnel feature vector X1 to the first encoder; The calculation method of the historical accuracy rate is as follows: N his N is the total number of historical tasks in the historical task sequence, 1≤task index i≤N his ; l i , d i , s i are the task execution duration, the task difficulty score and the task quality score of the i-th historical task in the historical task sequence, respectively; p his is the historical accuracy rate; The time sequence feature extraction unit is configured to perform one-hot encoding on the labeling operation type of each single-step labeling behavior of the labeling behavior sequence according to a preset operation type set to obtain a corresponding operation type one-hot encoding vector, perform normalization processing on the operation processing time length of each single-step labeling behavior to obtain a corresponding normalized time length, compose a corresponding single-step embedding vector from each operation type one-hot encoding vector and the corresponding normalized time length, and compose a corresponding time sequence feature sequence X2 from all the obtained single-step embedding vectors in chronological order, and send the time sequence feature sequence X2 to the second encoder; The content feature extraction unit is configured to respectively perform one-hot encoding, position embedding encoding, length normalization, and token embedding encoding on the element field, the element position, the element length, and the element text of each to-be-labeled element of the labeled content information to obtain a corresponding element field one-hot encoding vector, a position embedding encoding vector, a normalized length, and a token embedding encoding vector; initialize a full-zero sentiment type one-hot encoding vector for each to-be-labeled element based on the set of sentiment types, initialize a full-zero three-element type one-hot encoding vector for each to-be-labeled element based on the set of three-element types, set an entity identification code initialized as zero for each to-be-labeled element, and form a corresponding entity encoding vector from each entity identification code and the three-element type one-hot encoding vector corresponding thereto; form a corresponding element encoding vector from the element field one-hot encoding vector, the position embedding encoding vector, the normalized length, the token embedding encoding vector, the sentiment type one-hot encoding vector, and the entity encoding vector of each to-be-labeled element; set the sentiment type one-hot encoding vector and the entity encoding vector of the element encoding vector corresponding to the current element based on the sentiment type label and the entity three-element label of each labeled element in the labeled content information; and finally, send the corresponding content feature tensor X3 obtained from all the element encoding vectors to the third encoder. The first encoder is implemented based on an MLP model; the first encoder is configured to perform feature encoding processing on the personnel feature vector X1 to obtain a corresponding encoding feature vector X4, and send the encoding feature vector X4 to the second feature fusion unit; the vector length of the encoding feature vector X4 is a preset sixth feature dimension C6; The second encoder is implemented based on a Bi-LSTM model; the second encoder is configured to perform bidirectional time sequence feature encoding on the time sequence feature sequence X2 to obtain a corresponding encoding feature vector X5, and send the encoding feature vector X5 to the second feature fusion unit; the vector length of the encoding feature vector X5 is a preset seventh feature dimension C7; C7>C6; The third encoder is sequentially connected by a CNN network, a pooling layer, and a full connection layer; the third encoder is configured to perform feature encoding processing on the content feature tensor X3 using the CNN network to obtain a corresponding process encoding tensor, perform maximum pooling processing on each feature dimension of the process encoding tensor based on the pooling layer to obtain a corresponding pooling feature vector, and perform full connection calculation on the pooling feature vector based on the full connection layer to obtain a corresponding encoding feature vector X6, and send the encoding feature vector X6 to the second feature fusion unit; the vector length of the encoding feature vector X6 is a preset eighth feature dimension C8; C8 The second feature fusion unit is configured to perform vector splicing processing on the encoded feature vectors X4, X5 and X6 to obtain a corresponding splicing vector X7, and send the splicing vector X7 to the second hidden layer network; the vector length of the splicing vector X7 is a preset ninth feature dimension C9, C9=C6+C7+C8. The second hidden layer network is sequentially connected by three second hidden layers; each of the second hidden layers is sequentially connected by a full connection layer and a ReLU activation function; the second hidden layer network is used for feature extraction processing on the spliced vector X7 to obtain a corresponding feature vector X8 which is sent to the second output layer; the vector length of the feature vector X8 is a preset tenth feature dimension C 10 , 1 < C 10 < C9; The second output layer is configured to perform regression calculation on the label quality score according to the feature vector X8 to obtain a corresponding quality prediction score and output the quality prediction score. wherein, W2, b2 are respectively a weight vector parameter and an offset scalar parameter of the second output layer; the weight vector parameter W2 has a vector length of C 10 ; the tensor shape of the feature vector X8 and the weight vector parameter W2 are both 1xC 10 ; Sigmoid() is a Sigmoid activation function; S pre-qualify is the quality prediction score.

5. The active learning capable labeling quality control method according to claim 2, characterized by, The first round of training of the difficulty evaluation model and the quality prediction model based on the training data set specifically includes: The first training file and the first difficulty label of each first data record of the training data set are extracted to form a corresponding second data record; the first training parameter set and the first quality label of each first data record are extracted to form a corresponding third data record; all the second data records obtained are used to form a corresponding first data set; and all the third data records obtained are used to form a corresponding second data set; The first round of training of the difficulty evaluation model based on the first data set; The first round of training of the quality prediction model based on the second data set.

6. The active learning capable labeling quality control method according to claim 2, characterized by, The selection of typical data records from the training data set to form an initial verification data set specifically includes: Step 61, the total number of records of the training data set is counted to obtain a corresponding sample total number N1; and an empty set is initialized as a corresponding verification data set; Step 62, and sequentially sort all the element texts of each first data record of the training data set to form a corresponding text word segmentation sequence; and encode each text word segmentation sequence based on a preset dense vector encoder to obtain a corresponding text encoding tensor; and perform average pooling processing on each feature dimension of each text encoding tensor to obtain a corresponding high-dimensional feature vector z j ; The dense vector encoder is implemented based on a BERT model of a type that has been pre-trained; the high-dimensional feature vector z j has a preset feature dimension D1, D1>256; 1≤index j≤N1; Step 63, first, based on the principal component analysis method, the current N1 high-dimensional feature vectors z j N1 low-dimensional feature vectors are obtained by dimension reduction Then, a similarity matrix E of N1xN1 is constructed, and the important feature screening set is obtained according to the similarity matrix E by the greedy strategy. wherein the screening set comprises one or more of the low-dimensional feature vectors Step 64, each of the low-dimensional feature vectors in the screening set is classified The corresponding first data record is added to the verification data set.

7. The active learning capable labeling quality control method according to claim 2, characterized by, The difficulty evaluation result of the target file based on the difficulty evaluation model and the preset personnel information library are used to perform label personnel allocation, specifically including: The target file is input as a corresponding label file into the difficulty evaluation model to obtain a corresponding difficulty score; The professional field of the target file is identified to obtain a corresponding current field; In the preset label difficulty-personnel level correspondence, the first level range corresponding to the first correspondence in which the first score range satisfies the current difficulty score is taken as a corresponding current level range; wherein the label difficulty-personnel level correspondence includes a plurality of the first correspondences; the first correspondence includes the first score range and the first level range; the first score range is a label difficulty score range; the first level range is composed of one or more annotation level grades in the personnel level set; The first personnel record in the personnel information library, whose first personnel state is an idle state, whose first personnel level satisfies the current level range, and whose first field range satisfies the current field, is taken as the current allocation personnel.

8. The active learning capable labeling quality control method according to claim 2, characterized by, The use of the quality prediction model to perform real-time monitoring of the annotation quality specifically includes: At the starting moment of the current labeling process, the first personnel level, the first field range and the first task sequence of the first personnel record corresponding to the current assigned personnel in the personnel information library are extracted as the corresponding personnel level, field range and historical task sequence to form a corresponding labeling personnel information; During the current labeling process, the labeling behavior of the current assigned personnel at each step is recorded to obtain a corresponding labeling behavior sequence; and the real-time labeling result of the current assigned personnel is used to update the set of labeled elements; During the current labeling process, a new labeling content information is periodically formed by the set of elements to be labeled and the latest set of labeled elements corresponding to the current labeling task at a preset first time frequency; the current labeling personnel information, labeling behavior sequence and labeling content information are input into the quality prediction model for processing, and the quality prediction score obtained by the current processing is used as a corresponding current score; and it is identified whether the current score is lower than a preset warning score threshold; if yes, real-time warning is given to the current assigned personnel.

9. The active learning capable labeling quality control method according to claim 2, characterized by, The current labeling result is evaluated in quality according to the preset labeling quality inspection rule library, specifically including: According to each labeling quality inspection rule in the labeling quality inspection rule library, the set of labeled elements obtained by the current labeling is scored in labeling quality to obtain a corresponding quality evaluation score; and it is identified whether the quality evaluation score is lower than a preset quality evaluation threshold; if yes, a corresponding artificial review data package composed of the target file and the set of elements to be labeled and the set of labeled elements is sent to a preset artificial review interface; if not, a corresponding labeling file data package composed of the target file and the set of elements to be labeled and the set of labeled elements is sent to a preset labeling passing interface.

10. The active learning capable annotation quality control method according to claim 2, characterized by, The personnel information library and the preset labeling sample pool are updated based on the current labeling process data, specifically including: The labeling behavior sequence obtained by recording the behavior of the current assigned personnel during the current labeling process is used as a current labeling behavior sequence; the sum of the operation processing time lengths of the current labeling behavior sequence is calculated and the calculation result is used as a corresponding task execution time length; the difficulty score and quality evaluation score of the current labeling task are used as a corresponding task difficulty score and task quality score; a corresponding first task is formed by the task execution time length, task difficulty score and task quality score obtained this time; the first task sequence of the first personnel record corresponding to the current assigned personnel in the personnel information library is used as a corresponding current task sequence; the first task in the current task sequence that exceeds the nearest first time period is deleted, and the first task obtained this time is added to the current task sequence; And the target file of this labeling task and its corresponding set of elements to be labeled are taken as a corresponding first sample file and a set of first sample elements to form a corresponding first sample record, and the current first sample record is added to the labeling sample pool.

11. The active learning capable annotation quality control method according to claim 2, characterized by, The quality evaluation condition of the latest period is periodically determined according to the quality evaluation condition of the latest period, and whether a round of active learning is needed is determined. The number of artificial reviews of the latest period is periodically counted according to the second preset time frequency to obtain a corresponding first review total, and when the first review total exceeds a preset review total threshold, it is determined that a round of active learning is needed.

12. The active learning capable labeling quality control method according to claim 2, characterized by, The verification data set is updated by selecting typical data records from the labeling sample pool, specifically including: Step 12-1, count the total number of records of the labeled sample pool to obtain the corresponding total number of samples N1; and sequentially sort all the element texts of each first sample record of the labeled sample pool to form a corresponding text segmentation sequence; and encode each text segmentation sequence based on a preset dense vector encoder to obtain a corresponding text encoding tensor; and perform average pooling processing on each feature dimension of each text encoding tensor to obtain a corresponding high-dimensional feature vector z j ; The dense vector encoder is implemented based on a BERT model of a type that has been pre-trained; the high-dimensional feature vector z j has a preset feature dimension D1, D1>256; 1≤index j≤N1; Step 12-2, first, based on the principal component analysis method, the current N1 high-dimensional feature vectors z j N1 low-dimensional feature vectors are obtained by performing feature dimension reduction Then, a similarity matrix E with a shape of N1 x N1 is constructed, and important feature screening is performed according to the similarity matrix E to obtain a corresponding screening set according to a greedy strategy. wherein the screening set comprises one or more of the low-dimensional feature vectors Step 12-3, in the labeled sample pool, each low-dimensional feature vector of the screening set is matched with a corresponding sample record in the labeled sample pool, and the sample record with the highest matching score is selected as the first sample record corresponding to the low-dimensional feature vector. The corresponding first sample record is recorded as a typical sample record, and all typical sample records are extracted to form a typical record set; and the labeled sample pool is emptied. Step 12-4, each of the typical sample records of the typical record set is assigned a plurality of corresponding senior labeling personnel, and one of them is taken as a corresponding first labeling personnel, and the rest are taken as corresponding first review personnel; Step 12-5, the labeling level of each first labeling personnel is taken as a corresponding personnel level, the professional knowledge field of each first labeling personnel is counted and a corresponding field range is formed based on the statistical results, and the task execution time, task difficulty score and task quality score of each historical labeling task of each first labeling personnel in the latest first period are counted and a corresponding historical task is formed based on the statistical results, and all the historical tasks in the latest first period form a corresponding historical task sequence; and the personnel level, field range and historical task sequence of each first labeling personnel form a corresponding labeling personnel information; Step 12-6, each first labeling personnel manually labels the first sample file of the typical sample record responsible for it, records each labeling behavior of the current labeling personnel during the labeling process to obtain a corresponding labeling behavior sequence, and counts the labeling result of the current labeling personnel at the end of the labeling to obtain a corresponding labeled element set; Step 12-7, each first review personnel of each typical sample record scores the first sample file of the current typical sample record according to each labeling difficulty evaluation rule of the preset labeling difficulty evaluation rule library to obtain a corresponding third score; and all the third scores corresponding to each first sample file are averaged to obtain a corresponding first difficulty label; Step 12-8, each first review personnel of each typical sample record scores the first sample element set of the current typical sample record according to each labeling quality inspection rule of the labeling quality inspection rule library to obtain a corresponding fourth score; and all the fourth scores corresponding to each first sample element set are averaged to obtain a corresponding first quality label; Step 12-9, and each of the first sample file as a corresponding first training file; and each of the first sample file corresponding to the first sample element set and the labeled element set consisting of a corresponding labeled content information; and each of the first sample file corresponding to the labeling personnel information, the labeling behavior sequence and the labeling content information consisting of a corresponding first training parameter set; and each of the first training file, the first training parameter set, the first difficulty label, the first quality label corresponding to the first data record consisting of a corresponding first data record of the typical sample record of the typical record set; and all the first data records obtained this time are added to the verification data set.

13. The active learning capable labeling quality control method according to any one of claims 6, 12, wherein The principal component analysis method is used to reduce the dimension of the features to obtain N1 low-dimensional feature vectors j The principal component analysis method is used to reduce the dimension of the features to obtain N1 low-dimensional feature vectors A similarity matrix E of N1xN1 is constructed, and important feature screening is performed according to the similarity matrix E in a greedy strategy to obtain a corresponding screening set, specifically including: Step 13-1, by N1 high-dimensional feature vectors z j A high-dimensional feature matrix A of N1 x D1 is formed, and the mean value μ of each column matrix element of the high-dimensional feature matrix A is calculated col ; and based on each of the mean values μ col , the N1 matrix elements a row,col of each column are centered to obtain the corresponding matrix elements b row,col ; and N1 x D1 matrix elements b row,col form the corresponding center feature matrix B; the covariance matrix C of the center feature matrix B is calculated; the eigen equation of the covariance matrix C is constructed; and the corresponding eigenvalue vector U and eigen tensor V are obtained by solving the eigen equation; wherein each row of the high-dimensional feature matrix A corresponds to one of the high-dimensional feature vectors z j , each column corresponds to one dimension of the D1 dimensions of the high-dimensional feature; the high-dimensional feature matrix A comprises N1 x D1 matrix elements a row,col , 1 ≤ row index row ≤ N1, 1 ≤ column index col ≤ D1. said mean μ col said matrix elements b row,col and said covariance matrix C are: The covariance matrix C has a shape of D1x D1; The eigen equation of the covariance matrix C is CV=UV; The vector length of the feature value vector U is D1, which is composed of D1 feature values u x sorted in descending order; 1≤index x≤D1; the adjacent front and rear two feature values u x x+1 have the size relationship u x ≥u x+1 ; the shape of the feature tensor V is D1×D1, which is composed of D1 feature vectors v x with a vector length of D1; and the modulus of each feature vector v x ‖v x ‖=1; and any two feature vectors v x , v y≠x satisfy the orthogonal relationship: 1≤index y≤D1;​ Step 13-2, from 128 to 256, based on the feature vector U search the first to meet the preset principal component screening conditions of low feature dimension k; and the feature tensor V in the first k feature vector v 1≤x≤k extracted to form a shape as D1xk projection matrix W; and based on the projection matrix W for each of the high-dimensional feature vector z j low-dimensional vector projection to get a vector length of k low-dimensional feature vector Wherein, the principal component screening condition is: p hold p is a predetermined percentage threshold, p hold ≥ 95%. The low-dimensional feature vector The vector projection mode is: Step 13-3, by N1 low-dimensional eigenvectors composing a sample set; and initializing the screening set as empty; and calculating the vector similarity of each two low-dimensional eigenvectors according to the cosine vector similarity algorithm; and composing the similarity matrix E by the obtained N1 x N1 similarities; composing a sample set; and initializing the screening set as empty; and calculating the vector similarity of each two low-dimensional eigenvectors according to the cosine vector similarity algorithm; and composing the similarity matrix E by the obtained N1 x N1 similarities; Wherein, the similarity matrix E includes N1xN1 similarity e j,g , 1≤index g≤N1; the similarity e j,g＝j in the matrix on the line is 1, and the two symmetrical similarities of the vertical bus in the matrix are equal. Step 13-4, compute N1 importance scores imc based on the similarity matrix E g ; and initialize N1 coverage degrees cov g to 0; wherein the importance score imc g is calculated as: Step 13-5: First, process each of the low-dimensional feature vectors in the sample set. Corresponding sample gain j Perform the calculation; then denote the index j corresponding to the maximum gain as j. * Then index j * The corresponding low-dimensional feature vector Transfer from the sample set to the filter set, and from index j * The corresponding N1 similarities and coverage The maximum value is selected as the new coverage. Then, based on the importance scores mentioned above, IMC g The corresponding coverage cov g The corresponding effective coverage (ecov) is obtained by weighting. g ; Wherein, updated ecov g = imc g x cov g ; Step 13-6, identify whether the minimum effective coverage min(ecov g ) exceeds a preset effective coverage threshold; if yes, go to step 13-7; if no, identify whether the sample set is empty; if empty, go to step 13-7; if not empty, return to step 13-5; Step 13-7, the obtained screening set is output as the important feature screening result this time.

14. An apparatus for performing the active learning capable label quality control method of any one of claims 1-13, characterized in that, The device comprises a quality control model preparation module, an annotation task quality control module, and an active learning module. The quality control model preparation module is used to construct a difficulty evaluation model and a quality prediction model before executing a first annotation task; construct a training data set through data acquisition; perform a first round of training on the difficulty evaluation model and the quality prediction model based on the training data set; and select typical data records from the training data set to form an initial verification data set; The annotation task quality control module is used to take a current text file to be annotated as a target file during the execution of each annotation task; assign an annotator based on the difficulty evaluation result of the target file and a preset personnel information library; manually annotate the target file by the assigned annotator; use the quality prediction model to monitor the annotation quality in real time during the annotation process; perform quality evaluation on the annotation result at the end of the annotation according to a preset annotation quality inspection rule library; and update the personnel information library and a preset annotation sample pool based on the annotation process data; The active learning module is used to periodically confirm whether a round of active learning is needed according to the quality evaluation status of the recent period after executing the first annotation task; if it is confirmed that a round of active learning is needed, update the verification data set by selecting typical data records from the annotation sample pool, and perform an upgrade training on the difficulty evaluation model and the quality prediction model based on the updated verification data set; The difficulty evaluation model is used to predict the annotation difficulty of the annotation file input by the model and output the corresponding difficulty score; the quality prediction model is used to predict the annotation quality of the annotator according to the annotation personnel information, the annotation behavior sequence and the annotation content information input by the model and output the corresponding quality prediction score. The difficulty assessment model comprises a complexity feature recognition unit, a structure feature recognition unit, a domain feature recognition unit, a first feature fusion unit, a first hidden layer network, and a first output layer. The complexity feature recognition unit, the structure feature recognition unit, and the domain feature recognition unit are respectively used for recognizing the text complexity, the text structure, and the text knowledge domain feature of the annotation file. The first feature fusion unit performs vector splicing on the three types of features. The first hidden layer network performs feature extraction on the spliced vector. The first output layer performs regression calculation on the annotation difficulty score according to the extracted features of the hidden layer network. The quality prediction model comprises a personnel feature extraction unit, a time sequence feature extraction unit, a content feature extraction unit, a first encoder, a second encoder, a third encoder, a second feature fusion unit, a second hidden layer network, and a second output layer. The personnel feature extraction unit, the time sequence feature extraction unit, and the content feature extraction unit are respectively used for extracting the personnel feature vector of the annotation personnel information, the time sequence feature sequence of the annotation behavior sequence, and the content feature tensor of the annotation content information. The first encoder, the second encoder, and the third encoder are respectively used for encoding the personnel feature vector, the time sequence feature sequence, and the content feature tensor. The second feature fusion unit performs vector splicing on the three types of encoded features. The second hidden layer network performs feature extraction on the spliced vector. The second output layer performs regression calculation on the annotation quality score according to the extracted features of the hidden layer network.

15. An electronic device, comprising: comprise: a memory, a processor, and a transceiver; the processor is configured to read and execute instructions in the memory to implement the method of any one of claims 1-13; the transceiver is coupled with the processor and is controlled by the processor to perform message transmission and reception.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer executes the method of any one of claims 1-13.

Images