Data annotation system

Abstract

The embodiment of the invention relates to a data annotation system. The system comprises an annotation scheduling module, a machine annotation module, an intelligent annotation model, an annotation knowledge base, an annotation interpretation module and a manual annotation feedback module, the labeling scheduling module is respectively connected with the machine labeling module and the manual labeling feedback module; the machine labeling module is respectively connected with the intelligent labeling model and the labeling interpretation module; the intelligent labeling model is connected with the labeling knowledge base; the annotation interpretation module is connected with the intelligent annotation model; the manual annotation feedback module is connected with the intelligent annotation model and the annotation knowledge base. According to the method and the device, the labeling accuracy of the model can be continuously improved while the labeling efficiency is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data labeling system. Background Art

[0002] In the field of artificial intelligence technology, the performance of a model is related to the quality of data annotation of its corresponding training data set. There are two main traditional data annotation methods: pure manual annotation and pure model annotation. Pure manual annotation methods have high annotation accuracy but low annotation efficiency; pure model annotation has high annotation efficiency, but the annotation accuracy of most models is difficult to improve after reaching a certain level. Summary of the invention

[0003] The purpose of the present invention is to provide a data annotation system in view of the defects of the prior art, which includes: an annotation scheduling module, a machine annotation module, an intelligent annotation model, an annotation knowledge base, an annotation explanation module and a manual annotation feedback module. Among them, the intelligent annotation model is an artificial intelligence model for data annotation, which automatically annotates the model input data with reference to the annotation knowledge base and outputs the corresponding annotation data and confidence score; the machine annotation module is used to call the intelligent annotation model to automatically annotate batch data, and the annotation explanation module is used to reason and explain the batch annotation data output by the machine annotation module; the manual annotation feedback module sends the data with a low confidence score to the manual annotation interface for processing, and fine-tunes the intelligent annotation model based on the manual annotation feedback and enhances the annotation knowledge base. The human-machine collaborative annotation mechanism provided by the present invention realizes the processing closed loop of "annotation-explanation-feedback-continuous optimization". Through the present invention, the annotation accuracy of the model can be continuously improved while ensuring the annotation efficiency.

[0004] To achieve the above-mentioned purpose, an embodiment of the present invention provides a data annotation system, which includes: an annotation scheduling module, a machine annotation module, an intelligent annotation model, an annotation knowledge base, an annotation interpretation module and a manual annotation feedback module;

[0005] The annotation scheduling module is connected to the machine annotation module and the manual annotation feedback module respectively; the machine annotation module is connected to the intelligent annotation model and the annotation interpretation module respectively; the intelligent annotation model is connected to the annotation knowledge base; the annotation interpretation module is connected to the intelligent annotation model; the manual annotation feedback module is connected to the intelligent annotation model and the annotation knowledge base respectively;

[0006] The annotation scheduling module is used to receive the original data set input by the first client and send it to the machine annotation module as the corresponding first data set; the original data set is composed of multiple first original records; the first original record includes a first data identifier and first original data;

[0007] The annotation scheduling module is also used to cache the first machine record set received each time the first manual record set and the first machine record set sent back by the machine annotation module are received; and identify whether the first manual record set received this time is empty; if the current first manual record set is not empty, send the current first manual record set to the manual annotation feedback module, and cache the current first manual behavior report after receiving the first manual behavior report sent back by the manual annotation feedback module, and form a new first data set from all the first original records in the original data set corresponding to the current first manual record set and send it to the machine annotation module; if the current first manual record set is empty, merge all the cached first machine record sets to obtain the merged record set as the corresponding annotated data set, and perform human efficiency analysis according to all the first cached human behavior reports to obtain the corresponding human efficiency analysis report, and delete all the first cached machine record sets and all the first human behavior reports, and send the annotated data set and the human efficiency analysis report back to the first client; the first human record set includes multiple first human records; the first human record consists of the corresponding first data identifier, the first original data and the first explanation item set; the first machine record set includes multiple first machine records; the first machine record includes the first data identifier, the first original data, the first annotated data and the first explanation item set; the first explanation item set includes key segment explanation items, key feature explanation items, and counterfactual segment explanation items;

[0008] The machine labeling module is used to input each of the first original data of the current first data set into the intelligent labeling model for labeling data prediction and labeling confidence prediction processing to obtain the corresponding first labeling data and first confidence; and mark the first data identifier corresponding to each first confidence lower than a preset confidence threshold as a low confidence identifier; and according to the preset process data extraction rules, extract part of the process data generated by the intelligent labeling model in the process of processing each of the first original data of the current first data set to form a corresponding first process data set; and each of the first original data and the corresponding first data identifier, the first labeling data and the first process data set form a corresponding record to be explained; and the obtained All the records to be interpreted that are obtained are composed of a corresponding set of records to be interpreted and sent to the annotation interpretation module; and when the first machine record set sent back by the annotation interpretation module is received, the first machine record corresponding to each of the low-trusted identifiers in the first machine record set is recorded as a low-trusted record, and the first data identifier, the first original data and the first interpretation item set of each of the low-trusted records form a corresponding first artificial record, and all the first artificial records obtained form a corresponding first artificial record set; and after obtaining the first artificial record set, all the low-trusted records in the first machine record set are deleted; and after the record deletion is completed, the first artificial record set and the first machine record set obtained this time are sent back to the annotation scheduling module;

[0009] The intelligent annotation model is an artificial intelligence model that has completed model training; the intelligent annotation model is used to perform annotation data prediction and annotation confidence prediction processing on the first original data input into the model with reference to the annotation knowledge base and output the corresponding first annotation data and the first confidence;

[0010] The annotation knowledge base includes a plurality of annotation knowledge records; the annotation knowledge records include knowledge sample data and sample annotation feature vectors; the vector space of the sample annotation feature vectors is recorded as the annotation feature vector space; the feature dimension of the annotation feature vector space is recorded as R dimension, where R is a positive integer;

[0011] The annotation interpretation module is used for performing an annotation interpretation process according to the current record set to be interpreted and the intelligent annotation model each time a record set to be interpreted is received, and obtaining a corresponding first machine record set to be sent back to the machine annotation module;

[0012] The manual annotation feedback module is used to call the preset knowledge query interface to retrieve the background knowledge and references of each of the first original data of the current first manual record set each time the first manual record set is received, to obtain the corresponding search background knowledge and search document catalog to form a corresponding first background knowledge set; and the first data identifier, the first original data, the first interpretation item set and the first background knowledge set corresponding to each of the first manual records in the current first manual record set form a corresponding second manual record; and all the obtained second manual records form a corresponding second manual record set; and the second manual record set is group-sent to P preset manual annotation interfaces; and the third manual record set fed back by each manual annotation interface is received; and the manual behavior analysis processing and model training data set synthesis processing are performed on the P third manual record sets to obtain the corresponding first manual behavior report and first training data set; and the intelligent annotation model is fine-tuned according to the first training data set; and when the model fine-tuning processing is completed, according to the The first training data set performs knowledge enhancement processing on the annotation knowledge base; and when the knowledge enhancement processing is completed, the first manual behavior report is sent back to the annotation scheduling module; the total number of manual annotation interfaces P is a positive integer, which defaults to an odd number greater than 1; each of the manual annotation interfaces corresponds to a manual identity identifier; the second manual record set includes multiple second manual records; the second manual record includes the first data identifier, the first original data and the first background knowledge set; the third manual record set includes multiple third manual records; the second and third manual records correspond one to one; the third manual record includes the first data identifier, the first original data, the first manual annotation data and the first manual annotation processing time; the first training data set includes multiple first training records; the first training record includes the first training data and the first label annotation data; the first manual behavior report includes P first manual behavior records; the first manual behavior record includes the first identity identifier, the first fastest annotation time, the first slowest annotation time, the first average annotation time and the first annotation accuracy.

[0013] Preferably, the annotation scheduling module is specifically configured to, when performing human efficiency analysis according to all the first human behavior reports in the cache to obtain a corresponding human efficiency analysis report:

[0014] Step 21, forming a corresponding first record set from the first manual behavior records of all the first manual behavior reports;

[0015] Step 22, grouping the first artificial behavior records with the same first identity identifier in the first record set into one category to form a corresponding first category record subset;

[0016] Step 23, taking each of the first-category record subsets as the corresponding current record subset; extracting the first fastest annotation time with the shortest time in the current record subset as the corresponding fastest annotation speed, and extracting the first slowest annotation time with the longest time as the corresponding slowest annotation speed; and performing mean calculation on all the first average annotation times of the current record subset to obtain the corresponding average annotation speed; and performing mean calculation on all the first annotation accuracy rates of the current record subset to obtain the corresponding average accuracy rate; and forming a corresponding manual efficiency analysis record by the first identity identifier, the fastest annotation speed, the slowest annotation speed, the average annotation speed and the average accuracy rate corresponding to the current record subset;

[0017] Step 24: compose the corresponding labor efficiency analysis report from the obtained P labor efficiency analysis records.

[0018] Preferably, the model input end of the intelligent annotation model is used to receive the first original data, the first model output end is used to output the corresponding first annotation data, and the second model output end is used to output the corresponding first confidence level;

[0019] The intelligent annotation model is composed of a feature encoding unit, a knowledge retrieval unit, a feature fusion unit, an annotation feature prediction unit, an annotation data generation unit and a confidence prediction unit;

[0020] The input end of the feature encoding unit is connected to the input end of the model, and the output end is connected to the first input end of the feature fusion unit; the input end of the knowledge retrieval unit is connected to the input end of the model, and the output end is connected to the second input end of the feature fusion unit, and is also connected to the annotation knowledge base; the output end of the feature fusion unit is connected to the input end of the annotation feature prediction unit; the output end of the annotation feature prediction unit is respectively connected to the input ends of the annotation data generation unit and the confidence prediction unit; the output end of the annotation data generation unit is connected to the output end of the first model; the output end of the confidence prediction unit is connected to the output end of the second model;

[0021] The feature encoding unit is used to perform data segmentation processing on the first original data input by the model according to a preset data segmentation rule to obtain a corresponding plurality of first segment data; and perform feature encoding processing on each of the first segment data to obtain a corresponding first segment feature vector; and form a corresponding first segment feature vector sequence composed of all the obtained first segment feature vectors and send it to the feature fusion unit; the vector space of the first feature encoding vector is recorded as the corresponding data feature vector space, and the feature dimension of the data feature vector space is recorded as D dimension, where D is a positive integer; each feature dimension of the data feature vector space corresponds to a data feature name;

[0022] The knowledge retrieval unit is used to calculate the approximation between the first original data and the knowledge sample data of each annotated knowledge record in the annotated knowledge base to obtain a corresponding first approximation; and sort all the obtained first approximations in order of approximation from high to low to obtain a corresponding first approximation sequence; and extract the subsequence part exceeding the preset approximation threshold in the first approximation sequence as the corresponding first subsequence; and identify whether the first subsequence is empty; if the first subsequence is empty, extract the sample annotation feature vector of the annotated knowledge record corresponding to the first first approximation of the first approximation sequence as the corresponding first reference vector; if the first subsequence is not empty, calculate the mean vector of the sample annotation feature vectors of all the annotated knowledge records corresponding to the first subsequence and use the obtained mean vector as the corresponding first reference vector; and send the obtained first reference vector to the feature fusion unit;

[0023] The feature fusion unit is used to perform feature vector mapping processing from the data feature vector space to the annotation feature vector space on each of the first segment feature vectors of the first segment feature vector sequence to obtain a corresponding second segment feature vector; and perform feature fusion processing on each of the second segment feature vectors and the first reference vector to obtain a corresponding first fused feature vector; and form a corresponding first fused feature tensor composed of all the obtained first fused feature vectors to send to the annotation feature prediction unit;

[0024] The annotation feature prediction unit is used to perform annotation feature recognition processing according to the first fused feature tensor to obtain a corresponding first annotation feature vector and send it to the annotation data generation unit and the confidence prediction unit; the vector space of the first annotation feature vector is the annotation feature vector space;

[0025] The annotation data generating unit is used for performing annotation data generation processing according to the first annotation feature vector to obtain the corresponding first annotation data and output it;

[0026] The confidence prediction unit is used to perform annotation confidence prediction processing according to the first annotation feature vector to obtain the corresponding first confidence and output it.

[0027] Preferably, the first process data set corresponds one-to-one to the first original data;

[0028] The first process data set consists of a corresponding first segment data sequence, a first segment feature vector sequence, and a second segment feature vector sequence; the first segment data sequence consists of all the first segment data corresponding to the current first original data; the second segment feature vector sequence consists of all the second segment feature vectors corresponding to the current first original data; in the first segment data sequence, the first segment feature vector sequence, and the second segment feature vector sequence, the first segment data, the first segment feature vector, and the second segment feature vector correspond to each other one by one and the sorting order is consistent;

[0029] The first set of explanation entries corresponds one-to-one to the first set of process data;

[0030] The key segment interpretation item of the first interpretation item set includes a plurality of first segment arrays; the first segment arrays correspond one-to-one to the first segment data in the corresponding first segment data sequence; the first segment arrays are composed of the corresponding first segment data and the first contribution;

[0031] The key feature explanation items of the first explanation item set include D first feature arrays; the first feature arrays correspond one-to-one to the feature dimensions of the data feature vector space; the first feature arrays consist of corresponding first data feature names and first SHAP values;

[0032] The counterfactual fragment interpretation entry of the first interpretation entry set consists of fragment data before replacement, fragment data after replacement, annotated data before replacement and annotated data after replacement; the fragment data before replacement is one of the first fragment data in the corresponding first fragment data sequence; the fragment data after replacement is another fragment data with the same data type but different semantics as the fragment data before replacement; the annotated data before replacement is the first annotated data corresponding to the corresponding first process data set; the annotated data after replacement is another annotated data with an annotated type different from that of the first annotated data.

[0033] Preferably, the annotation interpretation module is specifically used for, when performing an annotation interpretation process according to the current record set to be interpreted and the intelligent annotation model to obtain a corresponding first machine record set and sending it back to the machine annotation module:

[0034] Step 51, setting an empty record as the corresponding first machine record for each record to be explained in the set of records to be explained; adding the first data identifier, the first original data, and the first annotation data of each record to be explained to the corresponding first machine record; and adding an empty set as the corresponding first explanation item set to each first machine record;

[0035] Step 52, taking each of the to-be-explained records in the to-be-explained record set as the corresponding current record to be explained; taking the first explanation item set of the first machine record corresponding to the current record to be explained as the corresponding current explanation item set; taking the first annotation data, the first segment data sequence and the second segment feature vector sequence of the current record to be explained as the corresponding current annotation data, the current segment data sequence and the current feature vector sequence; and calculating the contribution of each of the first segment data in the current segment data sequence to the current annotation data according to the current feature vector sequence in a multi-head attention mechanism to obtain the corresponding first contribution; and forming a corresponding first segment array from each of the first segment data in the current segment data sequence and the corresponding first contribution; and forming a corresponding key segment explanation item from all of the first segment arrays corresponding to the current feature vector sequence; and adding the current key segment explanation item to the current explanation item set;

[0036] Step 53, taking each of the first segment feature vector sequences of the record set to be interpreted as the corresponding current feature vector sequence, and performing mean vector calculation on all of the first segment feature vectors of the current feature vector sequence to obtain the corresponding mean feature vector; and forming a corresponding mean feature tensor from all of the mean feature vectors corresponding to the record set to be interpreted; and calculating the contribution of each feature dimension of the data feature vector space to the labeled data according to the mean feature tensor according to a preset SHAP algorithm to obtain the corresponding first SHAP value; and taking the data feature name corresponding to each feature dimension of the data feature vector space as the corresponding first data feature name; and forming a corresponding first feature array from each of the first data feature names and the corresponding first SHAP value; and forming a corresponding key feature explanation item from all of the obtained first feature arrays; and adding the currently obtained key feature explanation item to the first explanation item set of all the first machine records corresponding to the current record set to be interpreted;

[0037] Step 54, taking each of the to-be-interpreted records in the to-be-interpreted record set as the corresponding current record to be interpreted; taking the first original data, the first fragment data sequence and the first annotated data of the current record to be interpreted as the corresponding current original data, the current fragment data sequence and the current annotated data; and performing a round of traversal on all the first fragment data of the current fragment data sequence; and during this round of traversal, taking the currently traversed first fragment data as the corresponding current fragment data; and using the preset fragment data generation tool according to the preset fragment data replacement rule to generate a fragment data with the same data type as the current fragment data but different semantics as the corresponding current replacement data; and replacing part of the data corresponding to the current fragment data in the current original data based on the current replacement data to obtain the corresponding new original data; and inputting the new original data into the intelligent annotation module The model performs annotation data prediction and annotation confidence prediction processing and uses the annotation data output by the model as the corresponding new annotation data; and confirms whether the annotation type of the new annotation data has changed relative to the current annotation data; if it is confirmed that the annotation type has changed, terminate this round of traversal and use the current segment data, the current replacement data, the current annotation data and the new annotation data as the corresponding pre-replacement segment data, the post-replacement segment data, the pre-replacement annotation data and the post-replacement annotation data to form a corresponding counterfactual segment explanation entry; if it is confirmed that the annotation type has not changed, turn to the next first segment data for traversal until the last first segment data traversal is completed; and when this round of traversal is terminated, add the obtained counterfactual segment explanation entry to the first explanation entry set of the first machine record corresponding to the current record to be explained;

[0038] Step 55: All the first machine records obtained constitute a corresponding first machine record set and send it back to the machine labeling module.

[0039] Preferably, the manual annotation feedback module is specifically used for, when performing manual behavior analysis processing and model training data set synthesis processing according to the P third manual record sets to obtain the corresponding first manual behavior report and first training data set:

[0040] Step 61, taking each of the third manual record sets as the corresponding current record set; taking the manual identity identifier corresponding to the current record set as the corresponding first identity identifier; taking the shortest first manual labeling processing time in the current record set as the corresponding first fastest labeling time; taking the longest first manual labeling processing time in the current record set as the corresponding first slowest labeling time; and calculating the average of all the first manual labeling processing times in the current record set and taking the calculated result as the corresponding first average labeling time;

[0041] Step 62, set two counters initialized to 0 for each of the third artificial record sets, recorded as the corresponding correct counter and the error counter; and form a corresponding second record set from the third artificial records of all the third artificial record sets; and group the P third artificial records with the same first data identifier in the second record set into one category to form a corresponding second category record subset; and perform a round of traversal on all the obtained second category record subsets; and in this round of traversal, use the currently traversed second category record subset as the corresponding current subset; and select correctly labeled data and incorrectly labeled data from the P first artificially labeled data based on a preset voting rule; and add 1 to the correct counter corresponding to the first artificially labeled data that is the correctly labeled data in the current subset, and add 1 to the error counter corresponding to the first artificially labeled data that is the incorrectly labeled data in the current subset; and use the first original data of the current subset as the corresponding first training data, and use each of the first artificially labeled data that is the correctly labeled data in the current subset as a corresponding first label labeled data, and form a corresponding first training record from the obtained first training data and each of the first label labeled data;

[0042] Step 63, after the current round of traversal of all the second-category record subsets is completed, the corresponding first labeling accuracy is calculated based on the correct and error counters corresponding to each of the third artificial record sets, and the first labeling accuracy = correct counter / (correct counter+error counter); and the first identity identifier, the first fastest labeling time, the first slowest labeling time, the first average labeling time and the first labeling accuracy corresponding to each of the third artificial record sets form a corresponding first artificial behavior record; and the obtained P first artificial behavior records form the corresponding first artificial behavior report; and all the obtained first training records form the corresponding first training data set.

[0043] Preferably, the manual labeling feedback module is specifically used to input each of the first training data sets into the intelligent labeling model for labeling data prediction and labeling confidence prediction when fine-tuning the intelligent labeling model according to the first training data set, and use the labeling data output by the model as the corresponding first predicted labeling data; and form the first predicted labeling data and the corresponding first label labeling data into a corresponding first prediction-label pair; and bring all the obtained first prediction-label pairs into a preset first loss function; and based on the preset first model optimizer, perform a round of fine-tuning on the labeling data generation unit and the model parameters of the labeling data generation unit of the intelligent labeling model in the direction of minimizing the first loss function; and at the end of this round of fine-tuning, confirm that the model fine-tuning process is completed; the first loss function includes at least L1 loss function, L2 loss function, and cross entropy loss function; the first model optimizer includes at least SGD optimizer and Adam optimizer.

[0044] Preferably, the manual annotation feedback module is specifically used to use each of the first training data sets as the corresponding knowledge sample data when the annotation knowledge base is subjected to knowledge enhancement processing according to the first training data set, and input each of the knowledge sample data into the intelligent annotation model for annotation data prediction and annotation confidence prediction processing, and extract the first annotation feature vector generated by the model during the processing as the corresponding sample annotation feature vector, and form a corresponding annotation knowledge record composed of each of the knowledge sample data and the corresponding sample annotation feature vector and add it to the annotation knowledge base.

[0045] The embodiment of the present invention provides a data annotation system, including: an annotation scheduling module, a machine annotation module, an intelligent annotation model, an annotation knowledge base, an annotation explanation module and a manual annotation feedback module. Among them, the intelligent annotation model is an artificial intelligence model for data annotation, which automatically annotates the model input data with reference to the annotation knowledge base and outputs the corresponding annotation data and confidence score; the machine annotation module is used to call the intelligent annotation model to automatically annotate batch data, and the annotation explanation module is used to reason and explain the batch annotation data output by the machine annotation module; the manual annotation feedback module sends the data with a low confidence score to the manual annotation interface for processing, and fine-tunes the intelligent annotation model based on the manual annotation feedback and enhances the annotation knowledge base. The human-machine collaborative annotation mechanism provided in the embodiment of the present invention realizes the processing closed loop of "annotation-explanation-feedback-continuous optimization"; through the embodiment of the present invention, on the one hand, the annotation efficiency can be improved, and on the other hand, the annotation accuracy of the model can be continuously improved while ensuring the annotation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A module structure diagram of a data annotation system provided by an embodiment of the present invention;

[0047] Figure 2 A module structure diagram of the intelligent annotation model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] like Figure 1 As shown in the module structure diagram of a data annotation system provided by an embodiment of the present invention, the data annotation system 1 provided by an embodiment of the present invention mainly includes: an annotation scheduling module 11, a machine annotation module 12, an intelligent annotation model 13, an annotation knowledge base 14, an annotation interpretation module 15 and a manual annotation feedback module 16. The connection relationship of each system component is: the annotation scheduling module 11 is connected to the machine annotation module 12 and the manual annotation feedback module 16 respectively; the machine annotation module 12 is connected to the intelligent annotation model 13 and the annotation interpretation module 15 respectively; the intelligent annotation model 13 is connected to the annotation knowledge base 14; the annotation interpretation module 15 is connected to the intelligent annotation model 13; the manual annotation feedback module 16 is connected to the intelligent annotation model 13 and the annotation knowledge base 14 respectively.

[0050] (I) Labeling and scheduling module 11:

[0051] The annotation scheduling module 11 of the embodiment of the present invention is used to receive the original data set input by the first client 2 and send it as the corresponding first data set to the machine annotation module 12. The original data set is composed of multiple first original records; the first original record includes a first data identifier and first original data.

[0052] The annotation scheduling module 11 is also used to cache the first machine record set received each time the first manual record set and the first machine record set sent back by the machine annotation module 12 are received; and to identify whether the first manual record set received this time is empty; if the current first manual record set is not empty, the current first manual record set is sent to the manual annotation feedback module 16, and after receiving the first manual behavior report sent back by the manual annotation feedback module 16, the current first manual behavior report is cached, and a new first data set is composed of all the first original records corresponding to the current first manual record set in the original data set and sent to the machine annotation module 12; if the current first manual record set is empty, all cached first machine record sets are merged and the merged record set is used as the corresponding annotation data set, and manual efficiency analysis is performed based on all cached first manual behavior reports to obtain the corresponding manual efficiency analysis report, and all cached first machine record sets and all first manual behavior reports are deleted, and the annotation data set and the manual efficiency analysis report are sent back to the first client 2.

[0053] Among them, the first manual record set of the embodiment of the present invention includes multiple first manual records; the first manual record is composed of the corresponding first data identifier, the first original data and the first explanation item set; the first machine record set includes multiple first machine records; the first machine record includes the first data identifier, the first original data, the first annotated data and the first explanation item set; the first explanation item set includes key fragment explanation items, key feature explanation items, and counterfactual fragment explanation items.

[0054] It should be noted that the data type of the first original data in the embodiment of the present invention is a pre-set data type, such as text, image, audio, video or any type of structured data. The intelligent annotation model 13 should correspond to the data type of the first original data, and perform data annotation on a set type of original data according to a set of pre-set annotation rules.

[0055] It should also be noted that, each time the labeling scheduling module 11 of the embodiment of the present invention receives a batch of data to be labeled, that is, the original data set, it will first send it as the first data set to the machine labeling module 12 for batch automatic labeling and interpretation; on the machine labeling module 12 side, after completing each batch of automatic labeling and interpretation, the labeling and interpretation results with lower confidence will be selected from the automatic labeling results of the current time to form a first manual record set, and the labeling and interpretation results with higher confidence will be formed into a first machine record set, and the first manual record set + the first machine record set will be returned to the labeling scheduling module 11; when the first manual record set is not empty, it means that it is necessary to optimize the intelligent labeling model 13 through manual labeling feedback, so at this time the labeling scheduling module 11 will send the first manual record set to the manual labeling feedback module 16 for processing, and cache the first machine record set received this time; the manual labeling feedback module 16 side manually labels the first manual record set received this time, and optimizes the intelligent labeling model 13 based on the manual labeling results, and enhances the labeling knowledge base 14, and after completing the model optimization and knowledge base enhancement, the statistical manual efficiency report, i.e., the first manual behavior report, is sent back to the annotation scheduling module 11; after caching the first manual behavior report received at that time, the annotation scheduling module 11 continues to extract the sub-dataset corresponding to the current first manual record set in the original data set as a new first data set and sends it to the machine annotation module 12 for batch automatic annotation and interpretation, and then receives the first manual record set + the first machine record set sent by the machine annotation module 12 again, and caches the first machine record set again; if the first manual record set is empty this time, it means that there is no need to optimize the model again. At this time, all the saved first machine record sets can be merged to obtain the final standard + interpretation data set, i.e., the annotation data set, and output it; if the first manual record set is not empty this time, then the model optimization and knowledge base enhancement are performed again through the manual annotation feedback module 16 in the above manner, and this cycle is repeated until the first manual record set obtained is empty.

[0056] It should also be noted that, after completing the human-machine collaborative annotation of a batch of original data sets, the annotation scheduling module 11 of the embodiment of the present invention will also inform the external management system or personnel of the efficiency (including speed and accuracy) analysis information related to this manual annotation by outputting a manual efficiency analysis report to the outside. After obtaining the manual efficiency analysis report, the external management system or personnel can further optimize the annotation level and efficiency of the manual annotation interface according to the content of the report.

[0057] In a specific implementation of the embodiment of the present invention, the annotation scheduling module 11 is specifically configured to, when performing human efficiency analysis according to all cached first human behavior reports to obtain a corresponding human efficiency analysis report:

[0058] Step A1, forming a corresponding first record set from the first manual action records of all first manual action reports;

[0059] Here, it can be seen from the following that the first manual behavior report includes P first manual behavior records; the first manual behavior record includes the first identity, the first fastest annotation time, the first slowest annotation time, the first average annotation time and the first annotation accuracy; the total number of manual annotation interfaces P is a positive integer, which defaults to an odd number greater than 1;

[0060] Step A2, grouping the first artificial behavior records with the same first identity in the first record set into one category to form a corresponding first category record subset;

[0061] Step A3, taking each first-category record subset as the corresponding current record subset; extracting the shortest first fastest annotation time in the current record subset as the corresponding fastest annotation speed, and extracting the longest first slowest annotation time as the corresponding slowest annotation speed; and calculating the average of all first average annotation times in the current record subset to obtain the corresponding average annotation speed; and calculating the average of all first annotation accuracy rates in the current record subset to obtain the corresponding average accuracy rate; and forming a corresponding manual efficiency analysis record by the first identity identifier, the fastest annotation speed, the slowest annotation speed, the average annotation speed and the average accuracy rate corresponding to the current record subset;

[0062] Step A4: compose a corresponding labor efficiency analysis report from the obtained P labor efficiency analysis records.

[0063] (II) Machine marking module 12:

[0064] The machine annotation module 12 of the embodiment of the present invention is used to input each first original data of the current first data set into the intelligent annotation model 13 for annotation data prediction and annotation confidence prediction processing to obtain corresponding first annotation data and first confidence; and mark the first data identifier corresponding to each first confidence lower than the preset confidence threshold as a low confidence identifier; and extract part of the process data generated by the intelligent annotation model 13 in the process of processing each first original data of the current first data set according to the preset process data extraction rules to form a corresponding first process data set; and each first original data and the corresponding first data identifier, the first annotation data and the first process data set form a corresponding Records to be interpreted; and the corresponding set of records to be interpreted composed of all the obtained records to be interpreted is sent to the annotation interpretation module 15; and when the first machine record set sent back by the annotation interpretation module 15 is received, the first machine record corresponding to each low-trust mark in the first machine record set is recorded as a low-trust record, and the first data mark, the first original data and the first interpretation item set of each low-trust record are used to form a corresponding first artificial record, and all the obtained first artificial records are used to form a corresponding first artificial record set; and after the first artificial record set is obtained, all the low-trust records in the first machine record set are deleted; and after the record deletion is completed, the first artificial record set and the first machine record set obtained this time are sent back to the annotation scheduling module 11.

[0065] Here, the preset confidence threshold is a preset threshold parameter. The process data extraction rule of the embodiment of the present invention is actually to extract all the first fragment data, all the first fragment feature vectors and all the second fragment feature vectors generated in the model reasoning process to form a first process data set. The first process data set corresponds one-to-one to the first original data. The data content of the first process data set consists of the corresponding first fragment data sequence, the first fragment feature vector sequence, and the second fragment feature vector sequence; wherein the first fragment data sequence is composed of all the first fragment data corresponding to the current first original data; the second fragment feature vector sequence is composed of all the second fragment feature vectors corresponding to the current first original data; in the first fragment data sequence, the first fragment feature vector sequence and the second fragment feature vector sequence, the first fragment data, the first fragment feature vector and the second fragment feature vector correspond to each other one-to-one and the sorting order is consistent. The output positions of the first fragment data, the first fragment feature vector and the second fragment feature vector can be understood through the model description below.

[0066] In addition, the first interpretation item set of the embodiment of the present invention corresponds one-to-one with the first process data set; 1) the key segment interpretation item of the first interpretation item set includes multiple first segment arrays; the first segment array corresponds one-to-one with the first segment data in the corresponding first segment data sequence; the first segment array consists of the corresponding first segment data and the first contribution. 2) the key feature interpretation item of the first interpretation item set includes D first feature arrays; the first feature array corresponds one-to-one with the feature dimension of the data feature vector space; the first feature array consists of the corresponding first data feature name and the first SHAP value. 3) the counterfactual segment interpretation item of the first interpretation item set consists of pre-replacement segment data, post-replacement segment data, pre-replacement annotated data and post-replacement annotated data; wherein the pre-replacement segment data is a first segment data in the corresponding first segment data sequence; the post-replacement segment data is another segment data with the same data type as the pre-replacement segment data but different semantics; the pre-replacement annotated data is the first annotated data corresponding to the corresponding first process data set; the post-replacement annotated data is another annotated data with a different annotation type from the first annotated data. The specific construction method of the three interpretation items of the first interpretation item set can be understood through the description of the annotation interpretation module 15 below.

[0067] (III) Intelligent labeling model 13:

[0068] The intelligent annotation model 13 of the embodiment of the present invention is an artificial intelligence model that has completed model training. The intelligent annotation model 13 is used to perform annotation data prediction and annotation confidence prediction processing on the first original data input by the model with reference to the annotation knowledge base 14 and output the corresponding first annotation data and first confidence.

[0069] The model input end of the intelligent annotation model 13 of the embodiment of the present invention is used to receive first original data, the first model output end is used to output corresponding first annotation data, and the second model output end is used to output corresponding first confidence.

[0070] like Figure 2As shown in the module structure diagram of the intelligent annotation model provided by the embodiment of the present invention, the intelligent annotation model 13 is composed of a feature encoding unit 131, a knowledge retrieval unit 132, a feature fusion unit 133, an annotation feature prediction unit 134, an annotation data generation unit 135 and a confidence prediction unit 136. The connection relationship of each component of the model is as follows: the input end of the feature encoding unit 131 is connected to the model input end, and the output end is connected to the first input end of the feature fusion unit 133; the input end of the knowledge retrieval unit 132 is connected to the model input end, and the output end is connected to the second input end of the feature fusion unit 133, and is also connected to the annotation knowledge base 14; the output end of the feature fusion unit 133 is connected to the input end of the annotation feature prediction unit 134; the output end of the annotation feature prediction unit 134 is respectively connected to the input ends of the annotation data generation unit 135 and the confidence prediction unit 136; the output end of the annotation data generation unit 135 is connected to the first model output end; the output end of the confidence prediction unit 136 is connected to the second model output end.

[0071] The feature encoding unit 131 is used to perform data segmentation processing on the first original data input into the model according to a preset data segmentation rule to obtain corresponding multiple first segment data; and perform feature encoding processing on each first segment data to obtain a corresponding first segment feature vector; and a corresponding first segment feature vector sequence is composed of all the obtained first segment feature vectors and sent to the feature fusion unit 133.

[0072] Here, the data segmentation rule is a pre-set segmentation / slicing rule, which can be customized according to the actual data type and / or application requirements. The vector space of the first feature coding vector is recorded as the corresponding data feature vector space, and the feature dimension of the data feature vector space is recorded as D dimension, D is a positive integer, and D is generally not equal to the feature dimension R of the labeled feature vector space mentioned in the previous article. Each feature dimension of the data feature vector space corresponds to a data feature name; for example, if the original data is of text type, then the feature dimension of the data feature vector space may include semantics, part of speech, language, etc., and the corresponding data feature name is the corresponding semantics, part of speech, language, etc.; if the original data is of image type, then the feature dimension of the data feature vector space may include color, grayscale, texture, etc., and the corresponding data feature name is the corresponding color, grayscale, texture, etc.; if the original data is of audio type, then the feature dimension of the data feature vector space may include sound frequency, sound amplitude, sound intensity, etc., and the corresponding data feature name is the corresponding sound frequency, sound amplitude, sound intensity, etc.

[0073] The knowledge retrieval unit 132 is used to calculate the approximation between the first original data and the knowledge sample data of each annotated knowledge record in the annotated knowledge base 14 to obtain the corresponding first approximation; and sort all the obtained first approximations in order of approximation from high to low to obtain the corresponding first approximation sequence; and extract the subsequence part that exceeds the preset approximation threshold in the first approximation sequence as the corresponding first subsequence; and identify whether the first subsequence is empty; if the first subsequence is empty, extract the sample annotation feature vector of the annotated knowledge record corresponding to the first first approximation of the first approximation sequence as the corresponding first reference vector; if the first subsequence is not empty, calculate the mean vector of the sample annotation feature vectors of all the annotated knowledge records corresponding to the first subsequence and use the obtained mean vector as the corresponding first reference vector; and send the obtained first reference vector to the feature fusion unit 133.

[0074] Here, the preset similarity threshold is a preset threshold parameter.

[0075] The feature fusion unit 133 is used to perform feature vector mapping processing on each first-fragment feature vector of the first-fragment feature vector sequence from the data feature vector space to the annotation feature vector space to obtain a corresponding second-fragment feature vector; and perform feature fusion processing on each second-fragment feature vector and the first reference vector to obtain a corresponding first fused feature vector; and a corresponding first fused feature tensor composed of all the obtained first fused feature vectors is sent to the annotation feature prediction unit 134.

[0076] The annotation feature prediction unit 134 is used to perform annotation feature recognition processing according to the first fused feature tensor to obtain the corresponding first annotation feature vector and send it to the annotation data generation unit 135 and the confidence prediction unit 136; the vector space of the first annotation feature vector is the annotation feature vector space.

[0077] The annotation data generation unit 135 is used to perform annotation data generation processing according to the first annotation feature vector to obtain the corresponding first annotation data and output it; the confidence prediction unit 136 is used to perform annotation confidence prediction processing according to the first annotation feature vector to obtain the corresponding first confidence and output it.

[0078] (IV) Annotation knowledge base 14:

[0079] The annotation knowledge base 14 of the embodiment of the present invention includes multiple annotation knowledge records. Each annotation knowledge record includes knowledge sample data and sample annotation feature vectors. The vector space of the sample annotation feature vectors here is recorded as the annotation feature vector space; the feature dimension of the annotation feature vector space is recorded as R dimension, where R is a positive integer.

[0080] (V) Annotation interpretation module 15:

[0081] The annotation interpretation module 15 of the embodiment of the present invention is used to perform an annotation interpretation process based on the current record set to be interpreted and the intelligent annotation model 13 every time a record set to be interpreted is received, and obtain a corresponding first machine record set to send back to the machine annotation module 12.

[0082] In another specific implementation of the embodiment of the present invention, the annotation interpretation module 15 is specifically used to perform an annotation interpretation process according to the current record set to be interpreted and the intelligent annotation model 13 to obtain a corresponding first machine record set and send it back to the machine annotation module 12:

[0083] Step B1, setting an empty record as the corresponding first machine record for each record to be interpreted in the set of records to be interpreted; adding the first data identifier, the first original data, and the first annotated data of each record to be interpreted to the corresponding first machine record; and adding an empty set as the corresponding first interpretation item set to each first machine record;

[0084] Step B2, taking each record to be interpreted in the set of records to be interpreted as the corresponding current record to be interpreted; taking the first interpretation item set of the first machine record corresponding to the current record to be interpreted as the corresponding current interpretation item set; taking the first annotation data, the first fragment data sequence, and the second fragment feature vector sequence of the current record to be interpreted as the corresponding current annotation data, the current fragment data sequence, and the current feature vector sequence; and according to the multi-head attention mechanism calculation method, calculating the contribution of each first fragment data in the current fragment data sequence to the current annotation data according to the current feature vector sequence to obtain the corresponding first contribution; and forming a corresponding first fragment array from each first fragment data in the current fragment data sequence and the corresponding first contribution; and forming a corresponding key fragment interpretation item from all first fragment arrays corresponding to the current feature vector sequence; and adding the current key fragment interpretation item to the current interpretation item set;

[0085] Here, calculating the contribution of each fragment data to the overall annotation result through the attention mechanism or multi-head attention mechanism is also a conventional fragment-overall weight analysis method, which will not be further explained here;

[0086] Step B3, taking each first segment feature vector sequence of the record set to be interpreted as the corresponding current feature vector sequence, and performing mean vector calculation on all first segment feature vectors of the current feature vector sequence to obtain the corresponding mean feature vector; and forming a corresponding mean feature tensor from all mean feature vectors corresponding to the record set to be interpreted; and calculating the contribution of each feature dimension of the data feature vector space to the labeled data according to the mean feature tensor according to the preset SHAP algorithm to obtain the corresponding first SHAP value; and taking the data feature name corresponding to each feature dimension of the data feature vector space as the corresponding first data feature name; and forming a corresponding first feature array from each first data feature name and the corresponding first SHAP value; and forming a corresponding key feature explanation item from all the obtained first feature arrays; and adding the currently obtained key feature explanation item to the first explanation item set of all first machine records corresponding to the current record set to be interpreted;

[0087] Here, using the SHAP algorithm to calculate the contribution of data features to the annotation results is a conventional feature sharing and extraction method. There are many corresponding SHAP algorithms, which can be set according to actual application requirements, and will not be further explained here;

[0088] Step B4, taking each record to be interpreted in the set of records to be interpreted as the corresponding current record to be interpreted; taking the first original data, the first fragment data sequence and the first annotation data of the current record to be interpreted as the corresponding current original data, the current fragment data sequence and the current annotation data; and performing a round of traversal on all the first fragment data of the current fragment data sequence; and in this round of traversal, taking the first fragment data currently traversed as the corresponding current fragment data; and using the preset fragment data generation tool according to the preset fragment data replacement rule to generate a fragment data with the same data type as the current fragment data but different semantics as the corresponding current replacement data; and based on the current replacement data, replacing part of the data corresponding to the current fragment data in the current original data to obtain the corresponding new original data; and inputting the new original data into the intelligent annotation Model 13 performs annotation data prediction and annotation confidence prediction processing and uses the annotation data output by the model as the corresponding new annotation data; and confirms whether the annotation type of the new annotation data has changed relative to the current annotation data; if it is confirmed that the annotation type has changed, then terminate this round of traversal and use the current segment data, current replacement data, current annotation data and new annotation data as the corresponding pre-replacement segment data, post-replacement segment data, pre-replacement annotation data and post-replacement annotation data to form a corresponding counterfactual segment explanation entry; if it is confirmed that the annotation type has not changed, then turn to the next first segment data for traversal until the last first segment data traversal is completed; and when this round of traversal is terminated, add the obtained counterfactual segment explanation entry to the first explanation entry set of the first machine record corresponding to the current record to be explained;

[0089] Here, the current step is actually to explain the impact of the fragment on the overall effect by establishing a counterfactual sample;

[0090] Step B5: All the first machine records obtained form a corresponding first machine record set and send it back to the machine labeling module 12.

[0091] (VI) Manual annotation feedback module 16:

[0092] The manual annotation feedback module 16 of the embodiment of the present invention is used to call the preset knowledge query interface to retrieve the background knowledge and references of each first original data of the current first manual record set every time a first manual record set is received to obtain the corresponding search background knowledge and search document catalog to form a corresponding first background knowledge set; and the first data identifier, first original data, first interpretation item set and first background knowledge set corresponding to each first manual record of the current first manual record set are used to form a corresponding second manual record; and all the obtained second manual records are used to form a corresponding second manual record set; and the second manual record set is sent to a preset P manual annotation interfaces in a group; and the third manual record set fed back by each manual annotation interface is received; and the manual behavior analysis processing and model training data set synthesis processing are performed based on the P third manual record sets to obtain the corresponding first manual behavior report and the first training data set; and the intelligent annotation model 13 is fine-tuned according to the first training data set; and when the model fine-tuning processing is completed, the annotation knowledge base 14 is subjected to knowledge enhancement processing according to the first training data set; and when the knowledge enhancement processing is completed, the first manual behavior report is sent back to the annotation scheduling module 11.

[0093] Among them, the total number of manual annotation interfaces P is a positive integer, which defaults to an odd number greater than 1; each manual annotation interface corresponds to a manual identity identifier; the second manual record set includes multiple second manual records; the second manual record includes a first data identifier, a first original data and a first background knowledge set; the third manual record set includes multiple third manual records; the second and third manual records correspond one to one; the third manual record includes a first data identifier, a first original data, a first manual annotation data and a first manual annotation processing time; the first training data set includes multiple first training records; the first training record includes first training data and first label annotation data; the first manual behavior report includes P first manual behavior records; the first manual behavior record includes a first identity identifier, a first fastest annotation time, a first slowest annotation time, a first average annotation time and a first annotation accuracy rate.

[0094] In another specific implementation of the embodiment of the present invention, the manual annotation feedback module 16 is specifically used to perform manual behavior analysis processing and model training data set synthesis processing according to the P third manual record sets to obtain the corresponding first manual behavior report and the first training data set:

[0095] Step C1, taking each third manual record set as the corresponding current record set; taking the manual identity identifier corresponding to the current record set as the corresponding first identity identifier; taking the shortest first manual labeling processing time in the current record set as the corresponding first fastest labeling time; taking the longest first manual labeling processing time in the current record set as the corresponding first slowest labeling time; and calculating the average of all first manual labeling processing times in the current record set and taking the calculated result as the corresponding first average labeling time;

[0096] Step C2, set two counters initialized to 0 for each third artificial record set, recorded as the corresponding correct counter and error counter; and form a corresponding second record set from the third artificial records of all third artificial record sets; and group P third artificial records with the same first data identifier in the second record set into one category to form a corresponding second category record subset; and perform a round of traversal on all the obtained second category record subsets; and in this round of traversal, use the currently traversed second category record subset as the corresponding current subset; and select correctly labeled data and incorrectly labeled data from P first artificially labeled data based on a preset voting rule; and add 1 to the correct counter corresponding to the first artificially labeled data that is the correctly labeled data in the current subset, and add 1 to the error counter corresponding to the first artificially labeled data that is the incorrectly labeled data in the current subset; and use the first original data of the current subset as the corresponding first training data, and use each first artificially labeled data that is the correctly labeled data in the current subset as a corresponding first label labeled data, and form a corresponding first training record from the obtained first training data and each first label labeled data;

[0097] Step C3, after this round of traversal of all second-category record subsets is completed, the corresponding first labeling accuracy is calculated based on the correct and error counters corresponding to each third artificial record set, and the first labeling accuracy = correct counter / (correct counter+error counter); and the first identity identifier, the first fastest labeling time, the first slowest labeling time, the first average labeling time and the first labeling accuracy corresponding to each third artificial record set constitute a corresponding first artificial behavior record; and the P first artificial behavior records obtained constitute a corresponding first artificial behavior report; and all the first training records obtained constitute a corresponding first training data set.

[0098] In another specific implementation method of an embodiment of the present invention, the manual annotation feedback module 16 is specifically used to input each first training data of the first training data set into the intelligent annotation model 13 for annotation data prediction and annotation confidence prediction processing when fine-tuning the intelligent annotation model 13 according to the first training data set, and use the annotation data output by the model as the corresponding first predicted annotation data; and form the first predicted annotation data and the corresponding first label annotation data into a corresponding first prediction-label pair; and bring all the obtained first prediction-label pairs into a preset first loss function; and based on the preset first model optimizer, perform a round of fine-tuning on the annotation data generation unit 135 of the intelligent annotation model 13 and the model parameters of the annotation data generation unit 135 in the direction of minimizing the first loss function; and at the end of this round of fine-tuning, confirm that the model fine-tuning process is completed.

[0099] Here, the first loss function includes at least an L1 loss function, an L2 loss function, and a cross entropy loss function; the first model optimizer includes at least an SGD optimizer and an Adam optimizer.

[0100] In another specific implementation method of an embodiment of the present invention, the manual annotation feedback module 16 is specifically used to use each first training data set as corresponding knowledge sample data when performing knowledge enhancement processing on the annotation knowledge base 14 according to the first training data set, and input each knowledge sample data into the intelligent annotation model 13 for annotation data prediction and annotation confidence prediction processing, and extract the first annotation feature vector generated by the model during the processing as the corresponding sample annotation feature vector, and form a corresponding annotation knowledge record composed of each knowledge sample data and the corresponding sample annotation feature vector and add it to the annotation knowledge base 14.

[0101] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the marking scheduling module can be a separately established processing element, or it can be integrated in a chip of the above-mentioned device. In addition, it can also be stored in the memory of the above-mentioned device in the form of program code, and called by a processing element of the above-mentioned system to perform the functions of the above-mentioned determined module. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each method step of the aforementioned method or each module processing step of the aforementioned system can be completed by an integrated logic circuit of hardware in the processor element or instructions in software form.

[0102] For example, the modules of the above system may be one or more integrated circuits configured to implement the aforementioned method, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module of the above system is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0103] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above method embodiments are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0104] The embodiment of the present invention provides a data annotation system, including: an annotation scheduling module, a machine annotation module, an intelligent annotation model, an annotation knowledge base, an annotation explanation module and a manual annotation feedback module. Among them, the intelligent annotation model is an artificial intelligence model for data annotation, which automatically annotates the model input data with reference to the annotation knowledge base and outputs the corresponding annotation data and confidence score; the machine annotation module is used to call the intelligent annotation model to automatically annotate batch data, and the annotation explanation module is used to reason and explain the batch annotation data output by the machine annotation module; the manual annotation feedback module sends the data with a low confidence score to the manual annotation interface for processing, and fine-tunes the intelligent annotation model based on the manual annotation feedback and enhances the annotation knowledge base. The human-machine collaborative annotation mechanism provided in the embodiment of the present invention realizes the processing closed loop of "annotation-explanation-feedback-continuous optimization"; through the embodiment of the present invention, on the one hand, the annotation efficiency can be improved, and on the other hand, the annotation accuracy of the model can be continuously improved while ensuring the annotation efficiency.

[0105] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0106] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0107] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data annotation system, characterized in that: The system includes: a labeling scheduling module, a machine labeling module, an intelligent labeling model, a labeling knowledge base, a labeling explanation module and a manual labeling feedback module; The annotation scheduling module is connected to the machine annotation module and the manual annotation feedback module respectively; the machine annotation module is connected to the intelligent annotation model and the annotation interpretation module respectively; the intelligent annotation model is connected to the annotation knowledge base; the annotation interpretation module is connected to the intelligent annotation model; the manual annotation feedback module is connected to the intelligent annotation model and the annotation knowledge base respectively; The annotation scheduling module is used to receive the original data set input by the first client and send it to the machine annotation module as the corresponding first data set; the original data set is composed of multiple first original records; the first original record includes a first data identifier and first original data; The annotation scheduling module is also used to cache the first machine record set received each time the first manual record set and the first machine record set sent back by the machine annotation module are received; and identify whether the first manual record set received this time is empty; if the current first manual record set is not empty, send the current first manual record set to the manual annotation feedback module, and cache the current first manual behavior report after receiving the first manual behavior report sent back by the manual annotation feedback module, and form a new first data set from all the first original records in the original data set corresponding to the current first manual record set and send it to the machine annotation module; if the current first manual record set is empty, merge all the cached first machine record sets to obtain the merged record set as the corresponding annotated data set, and perform human efficiency analysis according to all the first cached human behavior reports to obtain the corresponding human efficiency analysis report, and delete all the first cached machine record sets and all the first human behavior reports, and send the annotated data set and the human efficiency analysis report back to the first client; the first human record set includes multiple first human records; the first human record consists of the corresponding first data identifier, the first original data and the first explanation item set; the first machine record set includes multiple first machine records; the first machine record includes the first data identifier, the first original data, the first annotated data and the first explanation item set; the first explanation item set includes key segment explanation items, key feature explanation items, and counterfactual segment explanation items; The machine labeling module is used to input each of the first original data of the current first data set into the intelligent labeling model for labeling data prediction and labeling confidence prediction processing to obtain the corresponding first labeling data and first confidence; and mark the first data identifier corresponding to each first confidence lower than a preset confidence threshold as a low confidence identifier; and according to the preset process data extraction rules, extract part of the process data generated by the intelligent labeling model in the process of processing each of the first original data of the current first data set to form a corresponding first process data set; and each of the first original data and the corresponding first data identifier, the first labeling data and the first process data set form a corresponding record to be explained; and the obtained All the records to be interpreted that are obtained are composed of a corresponding set of records to be interpreted and sent to the annotation interpretation module; and when the first machine record set sent back by the annotation interpretation module is received, the first machine record corresponding to each of the low-trusted identifiers in the first machine record set is recorded as a low-trusted record, and the first data identifier, the first original data and the first interpretation item set of each of the low-trusted records form a corresponding first artificial record, and all the first artificial records obtained form a corresponding first artificial record set; and after obtaining the first artificial record set, all the low-trusted records in the first machine record set are deleted; and after the record deletion is completed, the first artificial record set and the first machine record set obtained this time are sent back to the annotation scheduling module; The intelligent annotation model is an artificial intelligence model that has completed model training; the intelligent annotation model is used to perform annotation data prediction and annotation confidence prediction processing on the first original data input into the model with reference to the annotation knowledge base and output the corresponding first annotation data and the first confidence; The annotation knowledge base includes a plurality of annotation knowledge records; the annotation knowledge records include knowledge sample data and sample annotation feature vectors; the vector space of the sample annotation feature vectors is recorded as the annotation feature vector space; the feature dimension of the annotation feature vector space is recorded as R dimension, where R is a positive integer; The annotation interpretation module is used for performing an annotation interpretation process according to the current record set to be interpreted and the intelligent annotation model each time a record set to be interpreted is received, and obtaining a corresponding first machine record set to be sent back to the machine annotation module; The manual annotation feedback module is used to call the preset knowledge query interface to retrieve the background knowledge and references of each of the first original data of the current first manual record set each time the first manual record set is received, to obtain the corresponding search background knowledge and search document catalog to form a corresponding first background knowledge set; and the first data identifier, the first original data, the first interpretation item set and the first background knowledge set corresponding to each of the first manual records in the current first manual record set form a corresponding second manual record; and all the obtained second manual records form a corresponding second manual record set; and the second manual record set is group-sent to P preset manual annotation interfaces; and the third manual record set fed back by each manual annotation interface is received; and the manual behavior analysis processing and model training data set synthesis processing are performed on the P third manual record sets to obtain the corresponding first manual behavior report and first training data set; and the intelligent annotation model is fine-tuned according to the first training data set; and when the model fine-tuning processing is completed, according to the The first training data set performs knowledge enhancement processing on the annotation knowledge base; and when the knowledge enhancement processing is completed, the first manual behavior report is sent back to the annotation scheduling module; the total number of manual annotation interfaces P is a positive integer, which defaults to an odd number greater than 1; each of the manual annotation interfaces corresponds to a manual identity identifier; the second manual record set includes multiple second manual records; the second manual record includes the first data identifier, the first original data and the first background knowledge set; the third manual record set includes multiple third manual records; the second and third manual records correspond one to one; the third manual record includes the first data identifier, the first original data, the first manual annotation data and the first manual annotation processing time; the first training data set includes multiple first training records; the first training record includes the first training data and the first label annotation data; the first manual behavior report includes P first manual behavior records; the first manual behavior record includes the first identity identifier, the first fastest annotation time, the first slowest annotation time, the first average annotation time and the first annotation accuracy.

2. The data annotation system according to claim 1, characterized in that: The annotation scheduling module is specifically used for, when performing human efficiency analysis according to all the first human behavior reports in the cache to obtain a corresponding human efficiency analysis report: Step 21, forming a corresponding first record set from the first manual behavior records of all the first manual behavior reports; Step 22, grouping the first artificial behavior records with the same first identity identifier in the first record set into one category to form a corresponding first category record subset; Step 23, taking each of the first record subsets as the corresponding current record subset; extracting the first fastest annotation duration with the shortest time in the current record subset as the corresponding fastest annotation speed, and extracting the first slowest annotation duration with the longest time as the corresponding slowest annotation speed; and calculating the average of all the first average annotation durations in the current record subset to obtain the corresponding average annotation speed; and calculating the mean of all the first annotation accuracy rates of the current record subset to obtain a corresponding average accuracy rate; and forming a corresponding labor efficiency analysis record by the first identity identifier, the fastest annotation speed, the slowest annotation speed, the average annotation speed and the average accuracy rate corresponding to the current record subset; Step 24: compose the corresponding labor efficiency analysis report from the obtained P labor efficiency analysis records.

3. The data annotation system according to claim 1, characterized in that: The model input end of the intelligent annotation model is used to receive the first original data, the first model output end is used to output the corresponding first annotation data, and the second model output end is used to output the corresponding first confidence level; The intelligent annotation model is composed of a feature encoding unit, a knowledge retrieval unit, a feature fusion unit, an annotation feature prediction unit, an annotation data generation unit and a confidence prediction unit; The input end of the feature encoding unit is connected to the input end of the model, and the output end is connected to the first input end of the feature fusion unit; the input end of the knowledge retrieval unit is connected to the input end of the model, and the output end is connected to the second input end of the feature fusion unit, and is also connected to the annotation knowledge base; the output end of the feature fusion unit is connected to the input end of the annotation feature prediction unit; the output end of the annotation feature prediction unit is respectively connected to the input ends of the annotation data generation unit and the confidence prediction unit; the output end of the annotation data generation unit is connected to the output end of the first model; the output end of the confidence prediction unit is connected to the output end of the second model; The feature encoding unit is used to perform data segmentation processing on the first original data input by the model according to a preset data segmentation rule to obtain a corresponding plurality of first segment data; and perform feature encoding processing on each of the first segment data to obtain a corresponding first segment feature vector; and form a corresponding first segment feature vector sequence composed of all the obtained first segment feature vectors and send it to the feature fusion unit; the vector space of the first feature encoding vector is recorded as the corresponding data feature vector space, and the feature dimension of the data feature vector space is recorded as D dimension, where D is a positive integer; each feature dimension of the data feature vector space corresponds to a data feature name; The knowledge retrieval unit is used to calculate the approximation between the first original data and the knowledge sample data of each annotated knowledge record in the annotated knowledge base to obtain a corresponding first approximation; and sort all the obtained first approximations in order of approximation from high to low to obtain a corresponding first approximation sequence; and extract the subsequence part exceeding the preset approximation threshold in the first approximation sequence as the corresponding first subsequence; and identify whether the first subsequence is empty; if the first subsequence is empty, extract the sample annotation feature vector of the annotated knowledge record corresponding to the first first approximation of the first approximation sequence as the corresponding first reference vector; if the first subsequence is not empty, calculate the mean vector of the sample annotation feature vectors of all the annotated knowledge records corresponding to the first subsequence and use the obtained mean vector as the corresponding first reference vector; and send the obtained first reference vector to the feature fusion unit; The feature fusion unit is used to perform feature vector mapping processing from the data feature vector space to the annotation feature vector space on each of the first segment feature vectors of the first segment feature vector sequence to obtain a corresponding second segment feature vector; and perform feature fusion processing on each of the second segment feature vectors and the first reference vector to obtain a corresponding first fused feature vector; and form a corresponding first fused feature tensor composed of all the obtained first fused feature vectors to send to the annotation feature prediction unit; The annotation feature prediction unit is used to perform annotation feature recognition processing according to the first fused feature tensor to obtain a corresponding first annotation feature vector and send it to the annotation data generation unit and the confidence prediction unit; the vector space of the first annotation feature vector is the annotation feature vector space; The annotation data generating unit is used for performing annotation data generation processing according to the first annotation feature vector to obtain the corresponding first annotation data and output it; The confidence prediction unit is used to perform annotation confidence prediction processing according to the first annotation feature vector to obtain the corresponding first confidence and output it.

4. The data annotation system according to claim 3, characterized in that: The first process data set corresponds one-to-one to the first original data; The first process data set consists of a corresponding first segment data sequence, a first segment feature vector sequence, and a second segment feature vector sequence; the first segment data sequence consists of all the first segment data corresponding to the current first original data; the second segment feature vector sequence consists of all the second segment feature vectors corresponding to the current first original data; in the first segment data sequence, the first segment feature vector sequence, and the second segment feature vector sequence, the first segment data, the first segment feature vector, and the second segment feature vector correspond to each other one by one and the sorting order is consistent; The first set of explanation entries corresponds one-to-one to the first set of process data; The key segment interpretation items of the first interpretation item set include a plurality of first segment arrays; the first segment arrays correspond one-to-one to the first segment data in the corresponding first segment data sequence; the first segment arrays are composed of the corresponding first segment data and the first contribution; The key feature explanation items of the first explanation item set include D first feature arrays; the first feature arrays correspond one-to-one to the feature dimensions of the data feature vector space; the first feature arrays consist of corresponding first data feature names and first SHAP values; The counterfactual fragment interpretation entry of the first interpretation entry set consists of fragment data before replacement, fragment data after replacement, annotated data before replacement and annotated data after replacement; the fragment data before replacement is one of the first fragment data in the corresponding first fragment data sequence; the fragment data after replacement is another fragment data with the same data type but different semantics as the fragment data before replacement; the annotated data before replacement is the first annotated data corresponding to the corresponding first process data set; the annotated data after replacement is another annotated data with an annotated type different from that of the first annotated data.

5. The data annotation system according to claim 4, characterized in that: The annotation interpretation module is specifically used for, when a corresponding first machine record set is obtained by performing an annotation interpretation process according to the current record set to be interpreted and the intelligent annotation model and sending it back to the machine annotation module: Step 51, setting an empty record as the corresponding first machine record for each record to be explained in the record set to be explained; and adding the first data identifier, the first original data, and the first annotated data of each record to be explained to the corresponding first machine record; and adding an empty set to each of the first machine records as the corresponding first interpretation entry set; Step 52, taking each of the to-be-explained records in the to-be-explained record set as the corresponding current record to be explained; taking the first explanation item set of the first machine record corresponding to the current record to be explained as the corresponding current explanation item set; taking the first annotation data, the first segment data sequence and the second segment feature vector sequence of the current record to be explained as the corresponding current annotation data, the current segment data sequence and the current feature vector sequence; and calculating the contribution of each of the first segment data in the current segment data sequence to the current annotation data according to the current feature vector sequence in a multi-head attention mechanism to obtain the corresponding first contribution; and forming a corresponding first segment array from each of the first segment data in the current segment data sequence and the corresponding first contribution; and forming a corresponding key segment explanation item from all of the first segment arrays corresponding to the current feature vector sequence; and adding the current key segment explanation item to the current explanation item set; Step 53, taking each of the first segment feature vector sequences of the record set to be interpreted as the corresponding current feature vector sequence, and performing mean vector calculation on all of the first segment feature vectors of the current feature vector sequence to obtain a corresponding mean feature vector; and forming a corresponding mean feature tensor from all of the mean feature vectors corresponding to the record set to be interpreted; and calculating the contribution of each feature dimension of the data feature vector space to the labeled data according to the mean feature tensor according to a preset SHAP algorithm to obtain the corresponding first SHAP value; and taking the data feature name corresponding to each feature dimension of the data feature vector space as the corresponding first data feature name; and forming a corresponding first feature array from each of the first data feature names and the corresponding first SHAP value; and forming a corresponding key feature explanation entry from all of the obtained first feature arrays; and adding the currently obtained key feature explanation item to the first explanation item set of all the first machine records corresponding to the current record set to be explained; Step 54, taking each of the to-be-interpreted records in the to-be-interpreted record set as the corresponding current to-be-interpreted record; taking the first original data, the first fragment data sequence and the first annotated data of the current to-be-interpreted record as the corresponding current original data, the current fragment data sequence and the current annotated data; and performing a round of traversal on all the first fragment data of the current fragment data sequence; and during this round of traversal, taking the currently traversed first fragment data as the corresponding current fragment data; and using a preset fragment data generation tool according to a preset fragment data replacement rule to generate a fragment data with the same data type as the current fragment data but different semantics as the corresponding current replacement data; and replacing part of the data corresponding to the current fragment data in the current original data based on the current replacement data The corresponding new original data is obtained in exchange; the new original data is input into the intelligent annotation model for annotation data prediction and annotation confidence prediction processing, and the annotation data output by the model is used as the corresponding new annotation data; and it is confirmed whether the annotation type of the new annotation data has changed relative to the current annotation data; if it is confirmed that the annotation type has changed, the current round of traversal is terminated and the current segment data, the current replacement data, the current annotation data and the new annotation data are used as the corresponding pre-replacement segment data, the post-replacement segment data, the pre-replacement annotation data and the post-replacement annotation data to form a corresponding counterfactual segment explanation entry; if it is confirmed that the annotation type has not changed, the next first segment data is turned to for traversal until the last first segment data traversal is completed; When this round of traversal ends, the obtained counterfactual fragment explanation entry is added to the first explanation entry set of the first machine record corresponding to the current record to be explained; Step 55: All the first machine records obtained form a corresponding first machine record set and send it back to the machine labeling module.

6. The data annotation system according to claim 1, characterized in that: The manual annotation feedback module is specifically used for obtaining the first manual behavior report and the first training data set by performing manual behavior analysis and model training data set synthesis according to the P third manual record sets: Step 61, taking each of the third manual record sets as the corresponding current record set; taking the manual identity identifier corresponding to the current record set as the corresponding first identity identifier; and taking the first manual annotation processing time with the shortest time in the current record set as the corresponding first fastest annotation time; and taking the first manual annotation processing time that is the longest in the current record set as the corresponding first slowest annotation time; and calculating the average of all the first manual annotation processing times in the current record set and using the calculation result as the corresponding first average annotation time; Step 62, setting two counters initialized to 0 for each of the third artificial record sets, recorded as the corresponding correct counter and the wrong counter; and forming a corresponding second record set from the third artificial records of all the third artificial record sets; and grouping the P third artificial records with the same first data identifier in the second record set into one category to form a corresponding second category record subset; and performing a round of traversal on all the obtained second category record subsets; and in this round of traversal, taking the currently traversed second category record subset as the corresponding current subset; and selecting the correctly labeled data and the incorrectly labeled data from the P first artificially labeled data based on a preset voting rule; The correct counter corresponding to the first manually labeled data that is the correctly labeled data in the current subset is incremented by 1, and the error counter corresponding to the first manually labeled data that is the incorrectly labeled data in the current subset is incremented by 1; the first original data in the current subset is used as the corresponding first training data, and each of the first manually labeled data in the correctly labeled data in the current subset is used as a corresponding first label labeled data, and the obtained first training data and each of the first label labeled data form a corresponding first training record; Step 63, after the current round of traversal of all the second-category record subsets is completed, the first labeling accuracy rate is calculated based on the correct and error counters corresponding to each of the third artificial record sets, and the first labeling accuracy rate = correct counter / (correct counter+error counter); and the first identity identifier, the first fastest labeling time, the first slowest labeling time, the first average labeling time and the first labeling accuracy rate corresponding to each of the third artificial record sets form a corresponding first artificial behavior record; The obtained P first artificial behavior records form the corresponding first artificial behavior report; and all the obtained first training records form the corresponding first training data set.

7. The data annotation system according to claim 1, characterized in that: The manual annotation feedback module is specifically used to input each of the first training data sets into the intelligent annotation model to perform annotation data prediction and annotation confidence prediction processing when the intelligent annotation model is fine-tuned according to the first training data set, and use the annotation data output by the model as the corresponding first predicted annotation data; The first predicted annotation data and the corresponding first label annotation data are combined into a corresponding first prediction-label pair; all the obtained first prediction-label pairs are brought into a preset first loss function; and based on the preset first model optimizer, a round of fine-tuning is performed on the annotation data generation unit of the intelligent annotation model and the model parameters of the annotation data generation unit in the direction of minimizing the first loss function; and at the end of this round of fine-tuning, it is confirmed that the model fine-tuning process is completed; the first loss function includes at least L1 loss function, L2 loss function, and cross entropy loss function; the first model optimizer includes at least SGD optimizer and Adam optimizer.

8. The data annotation system according to claim 1, characterized in that: The manual annotation feedback module is specifically used to use each of the first training data sets as the corresponding knowledge sample data when the annotation knowledge base is subjected to knowledge enhancement processing according to the first training data set, and input each of the knowledge sample data into the intelligent annotation model for annotation data prediction and annotation confidence prediction processing, and extract the first annotation feature vector generated by the model during the processing as the corresponding sample annotation feature vector, and form a corresponding annotation knowledge record composed of each of the knowledge sample data and the corresponding sample annotation feature vector to add to the annotation knowledge base.

Images