A multi-source data annotation system and method based on artificial intelligence

Abstract

The present invention discloses a multi-source data annotation system and method based on artificial intelligence, and relates to the technical field of automatic data annotation. The present invention collects historical records of inaccurate data annotation results output by an automatic data annotation platform, monitors the identification and annotation blind spots existing in the automatic data annotation platform during the automatic annotation of data transmitted by a corresponding data acquisition terminal, that is, a type of data for which the data annotation results output by the automatic data annotation platform based on an existing annotation operation mechanism are inaccurate; through the clustering situation presented by this type of data as the automatic data annotation platform runs, a judgment analysis is carried out to determine whether a trend of development is present, and an evaluation calculation of an updated warning value of the automatic data annotation platform is implemented according to the judgment and analysis results, so as to assist in the performance monitoring of the automatic data annotation platform and improve the accuracy and efficiency of automatic data annotation.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic data annotation, and in particular to a multi-source data annotation system and method based on artificial intelligence. Background Art

[0002] The implementation principle of automatic data annotation mainly relies on technologies such as deep learning, computer vision and natural language processing. Through training and learning of large amounts of data, automatic data annotation technology can automatically identify and classify data, thereby realizing automatic data annotation. In terms of image annotation, AI automatic annotation technology usually uses technologies such as target detection and image segmentation to identify and annotate objects in images. In terms of voice annotation, it mainly uses technologies such as speech recognition and natural language processing to realize automatic transcription and annotation of voice data. These technologies can simulate the human understanding and annotation process of data, greatly improving the efficiency and accuracy of data annotation.

[0003] In addition, the implementation of automatic data labeling also involves algorithm optimization and model training. By using active learning methods, the algorithm can select samples that are more valuable for training AI models, and users label those selected samples. In this way, the AI ​​model is trained with a portion of the data that has been labeled by the user, and then the model is used to label the remaining data, and the data that is more difficult to label the model is screened out for manual labeling, and finally the data is used for model optimization. After several rounds, the AI ​​model used for data labeling will have higher accuracy and better data labeling. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-source data annotation system and method based on artificial intelligence to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a multi-source data annotation method based on artificial intelligence, the method comprising:

[0006] Step S1: extracting the historical data annotation records with human intervention factors as the final output data annotation results from the automated data annotation platform, and setting them as target records; wherein the data annotation results with human intervention factors refer to the data annotation results obtained after manual verification and adjustment of the data annotation results planned to be output by the automated data annotation platform;

[0007] Step S2: In each target record, based on the difference in data annotation results before and after manual verification and adjustment, extract the characteristic data segments showing anomalies in annotation, and divide and process the characteristic data segments based on the difference in data collection terminals;

[0008] Step S3: Based on each target record, the feature adjustment value presented by the automated data annotation platform under the human intervention factor is calculated and evaluated; the target record whose corresponding feature adjustment value is greater than the threshold is set as a feature record;

[0009] Step S4: pre-build an anomaly recognition database for each data collection terminal connected to the automated data annotation platform; extract each feature data set corresponding to each target adjustment terminal from each feature record, and store it into the anomaly recognition database of each target adjustment terminal; perform trend analysis on the data clustering phenomenon presented in each anomaly recognition database, and mark the corresponding feature cluster;

[0010] Step S5: sort out the distribution of labels of corresponding feature clusters in each anomaly recognition database, evaluate and calculate the updated warning value of the automated data annotation platform, and feedback to the management terminal when the updated warning value is greater than the warning threshold.

[0011] Furthermore, the step S2 includes:

[0012] Step S2-1: In each target record, the data labeling result that the automated data labeling platform plans to output based on the data transmitted by the corresponding data collection terminal before manual verification and adjustment is performed is set as the first feature labeling result, and the data labeling result obtained after manual verification and adjustment of the first feature labeling result is set as the second feature labeling result;

[0013] Step S2-2: for each target record, in the corresponding first feature annotation result, extract the data segments whose corresponding annotation information is different from that presented in the corresponding second feature annotation result; set the data segments as the feature data segments presenting anomalies in the target record; the annotation information includes the annotation style and the data range selected by the annotation;

[0014] Step S2-3: Divide the data constituting each characteristic data segment according to the corresponding data collection terminals, and obtain a plurality of characteristic data sets corresponding to each data segment, wherein one characteristic data set corresponds to one data collection terminal.

[0015] Furthermore, the step S3 includes:

[0016] Step S3-1: In each target record, the data collection end with the feature data set is input to the automated data annotation platform and set as the target adjustment end in the corresponding target record;

[0017] Step S3-2: In each target record, accumulate the total number N of data collection terminals that input data into the automated data labeling platform, and the total number m of target adjustment terminals among the N data collection terminals; simultaneously obtain the similarity α between the corresponding first feature labeling result and the corresponding second feature labeling result; calculate the feature adjustment value β = (m / N) × (1 / α) for each target record.

[0018] Among them, if the ratio of m / N is higher, it means that the range of data collection terminals involved in the data that are manually adjusted for annotation information is larger, which reflects that the error rate of the automatic annotation mechanism in the automated data annotation platform is larger; the larger the ratio of α is, the smaller the content of manual adjustment is, and it is likely that only slight adjustments are made, which reflects that the error rate of the automatic annotation mechanism in the automated data annotation platform is smaller.

[0019] Further, the step S4 includes:

[0020] Step S4-1: In each anomaly recognition database, feature data sets that satisfy a similarity greater than a similarity threshold are respectively collected to obtain a plurality of feature clusters; in each feature cluster, all feature data sets are sorted according to the time sequence of the corresponding storage into the anomaly recognition database to generate a feature data sequence corresponding to each feature cluster, wherein every two adjacent feature data sets constitute a storage node in the corresponding feature data sequence;

[0021] Each feature cluster obtained based on the above method represents a type of blind spot in the automatic data annotation platform's process of automatically annotating the data transmitted by the corresponding data collection end, that is, a type of data that makes the data annotation results output by the automatic data annotation platform inaccurate; if the storage interval between each two feature data sets in a feature cluster is relatively short, it means that the frequency of occurrence of this type of blind spot is higher, and the possibility of developing into a fast trend is higher;

[0022] Step S4-3: In the feature data sequence of each feature cluster, extract the interval time between the two feature data sets stored in the anomaly recognition database for each storage node in turn, and obtain the average interval time of all storage nodes;

[0023] Step S4-4: When, in a feature data sequence of a certain feature cluster, the average interval duration is less than a duration threshold, or the proportion of storage nodes whose corresponding interval duration is less than the average interval duration in all storage nodes contained in the feature data sequence is greater than the proportion threshold, the certain feature cluster is marked.

[0024] Further, the step S4 includes:

[0025] Step S5-1: Obtain the total number F of data collection terminals connected to the automated data annotation platform, set the data collection terminal with at least one marked feature cluster among the F data collection terminals as the warning terminal, and accumulate the total number R of warning terminals. For each warning terminal, accumulate the total number K of feature clusters formed in the corresponding anomaly recognition database, and the total number D of marked feature clusters among the K feature clusters;

[0026] Step S5-2: Evaluate and calculate the updated warning value of the automated data annotation platform ξ=(R / F)×[D1 / K1+D2 / K2+...+D R / K R ]; where K1, K2, ..., K R They represent the total number of feature clusters formed in the anomaly recognition database of the 1st, 2nd, ..., Rth warning terminals respectively; D1, D2, ..., D R They respectively represent the total number of marked feature clusters in the anomaly recognition database of the 1st, 2nd, ..., Rth warning terminals.

[0027] In order to better implement the above method, a multi-source data annotation system is also proposed, which includes: a target record extraction module, a feature data segment processing module, a feature record extraction module, an anomaly recognition database management module, and an update warning value evaluation module;

[0028] The target record extraction module is used to extract, from the automated data annotation platform, historical data annotation records whose final output data annotation results are human intervention factors, and set them as target records; wherein the data annotation results with human intervention factors refer to the data annotation results obtained after manual verification and adjustment of the data annotation results planned to be output by the automated data annotation platform;

[0029] The characteristic data segment processing module is used to extract characteristic data segments with abnormal annotations in each target record based on the difference in data annotation results before and after manual verification and adjustment, and divide and process the characteristic data segments based on the difference in data collection terminals;

[0030] The feature record extraction module: calculates and evaluates the feature adjustment value presented by the automated data annotation platform under the human intervention factor based on each target record; sets the target record whose corresponding feature adjustment value is greater than the threshold as the feature record;

[0031] The anomaly recognition database management module is used to pre-build an anomaly recognition database for each data collection terminal connected to the automated data annotation platform; extract each feature data set corresponding to each target adjustment terminal from each feature record, and store it into the anomaly recognition database of each target adjustment terminal; perform trend analysis on the data clustering phenomenon presented in each anomaly recognition database, and mark the corresponding feature cluster;

[0032] The update warning value evaluation module is used to sort out the label distribution of the corresponding feature clusters in each abnormal recognition database, evaluate and calculate the update warning value of the automatic data annotation platform, and feedback to the management terminal when the update warning value is greater than the warning threshold.

[0033] Furthermore, the characteristic data segment processing module includes a characteristic data segment identification and extraction unit and a data division management unit;

[0034] The characteristic data segment identification and extraction unit is used to extract characteristic data segments showing abnormal annotations in each target record based on the difference in data annotation results before and after manual verification and adjustment;

[0035] The data division management unit is used to divide the characteristic data segments based on the differences in data collection terminals.

[0036] Further, the update warning value evaluation module includes a feature clustering management unit and an update warning value calculation unit;

[0037] The feature clustering management unit is used to sort out the distribution of labels of corresponding feature clusters in each anomaly recognition database;

[0038] The update warning value calculation unit is used to evaluate and calculate the update warning value of the automatic data annotation platform, and feedback to the management terminal when the update warning value is greater than the warning threshold.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention collects historical records of inaccurate data annotation results output by the automated data annotation platform, monitors the identification and annotation blind spots existing in the automated data annotation platform during the automated annotation process of the data transmitted by the corresponding data acquisition terminal, that is, a type of data for which the data annotation results output by the automated data annotation platform based on the existing annotation operation mechanism are inaccurate; through the clustering of this type of data as the automated data annotation platform runs, a judgment analysis is conducted on whether there is a trend of development, and based on the judgment and analysis results, an evaluation and calculation of the updated warning value of the automated data annotation platform is implemented, thereby assisting in the performance monitoring of the automated data annotation platform and improving the accuracy and efficiency of automated data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a multi-source data annotation method based on artificial intelligence according to the present invention;

[0041] Figure 2 It is a structural schematic diagram of a multi-source data annotation system based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0042] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0043] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, a multi-source data labeling method based on artificial intelligence, the method comprising:

[0044] Step S1: extracting the historical data annotation records with human intervention factors as the final output data annotation results from the automated data annotation platform, and setting them as target records; wherein the data annotation results with human intervention factors refer to the data annotation results obtained after manual verification and adjustment of the data annotation results planned to be output by the automated data annotation platform;

[0045] Step S2: In each target record, based on the difference in data annotation results before and after manual verification and adjustment, extract the characteristic data segments showing anomalies in annotation, and divide and process the characteristic data segments based on the differences in data collection terminals;

[0046] Step S3: Based on each target record, the feature adjustment value presented by the automated data annotation platform under the human intervention factor is calculated and evaluated; the target record whose corresponding feature adjustment value is greater than the threshold is set as a feature record;

[0047] Step S4: pre-build an anomaly recognition database for each data collection terminal connected to the automated data annotation platform; extract each feature data set corresponding to each target adjustment terminal from each feature record, and store it into the anomaly recognition database of each target adjustment terminal; perform trend analysis on the data clustering phenomenon presented in each anomaly recognition database, and mark the corresponding feature clusters;

[0048] Step S5: Sort out the distribution of labels of corresponding feature clusters in each anomaly recognition database, evaluate and calculate the updated warning value for the automated data annotation platform, and feedback to the management terminal when the updated warning value is greater than the warning threshold.

[0049] Wherein, step S2 comprises:

[0050] Step S2-1: In each target record, the data labeling result that the automated data labeling platform plans to output based on the data transmitted by the corresponding data collection terminal before manual verification and adjustment is performed is set as the first feature labeling result, and the data labeling result obtained after manual verification and adjustment of the first feature labeling result is set as the second feature labeling result;

[0051] Step S2-2: for each target record, in the corresponding first feature annotation result, extract the corresponding data segments whose annotation information is different from that presented in the corresponding second feature annotation result; set each data segment as a feature data segment presenting an anomaly in each target record; the annotation information includes the annotation style and the data range selected by the annotation;

[0052] The data annotation styles mainly include the following:

[0053] Rectangular box annotation: Draw a rectangular box around the target object for target detection. It is often used to annotate people, cars, objects, etc. and is suitable for fields such as autonomous driving and human-computer interaction. Polygonal annotation: Use polygonal boxes to annotate irregular target objects. Compared with rectangular box annotation, it can frame the target more accurately and is suitable for annotating irregular objects. Semantic segmentation: According to the attributes of the object, complex irregular images are divided into regions and the corresponding attributes are annotated. It is often used in fields such as autonomous driving and virtual reality. Key point annotation: Annotate specific key points or landmarks, such as facial feature points, to determine the functions of people in the image.

[0054] For example, in the first feature annotation result of a target record, the annotation style corresponding to the data range Q is a rectangular frame annotation, but in the second feature annotation result, the annotation style corresponding to the data range Q is a polygonal annotation. Therefore, it can be known that the feature data segment with the anomaly is the data range Q.

[0055] For example, in the first feature annotation result of a target record, the annotation style corresponding to the data range Q is a rectangular frame annotation, but in the second feature annotation result, the annotation style corresponding to the data range Q' is a rectangular frame annotation, and the data range Q = data range Q' + data range F; therefore, it can be known that the feature data segment with an anomaly in the annotation is the data range F;

[0056] Step S2-3: Divide the data constituting each characteristic data segment according to the corresponding data collection end, and obtain a plurality of characteristic data sets corresponding to each data segment, wherein one characteristic data set corresponds to one data collection end.

[0057] Wherein, step S3 comprises:

[0058] Step S3-1: In each target record, the data collection end with the feature data set is input to the automated data annotation platform and set as the target adjustment end in the corresponding target record;

[0059] Step S3-2: In each target record, accumulate the total number N of data collection terminals that input data into the automated data labeling platform, and the total number m of target adjustment terminals among the N data collection terminals; simultaneously obtain the similarity α between the corresponding first feature labeling result and the corresponding second feature labeling result; calculate the feature adjustment value β = (m / N) × (1 / α) for each target record.

[0060] Wherein, step S4 comprises:

[0061] Step S4-1: In each anomaly recognition database, feature data sets that satisfy a similarity greater than a similarity threshold are respectively collected to obtain a plurality of feature clusters; in each feature cluster, all feature data sets are sorted according to the time sequence of the corresponding storage into the anomaly recognition database to generate a feature data sequence corresponding to each feature cluster, wherein every two adjacent feature data sets constitute a storage node in the corresponding feature data sequence;

[0062] Step S4-3: In the feature data sequence of each feature cluster, extract the interval time between the two feature data sets stored in the anomaly recognition database for each storage node in turn, and obtain the average interval time of all storage nodes;

[0063] Step S4-4: When, in a feature data sequence of a feature cluster, the average interval duration is less than the duration threshold, or the proportion of storage nodes whose corresponding interval duration is less than the average interval duration in all storage nodes contained in the feature data sequence is greater than the proportion threshold, the feature cluster is marked.

[0064] Wherein, step S4 comprises:

[0065] Step S5-1: Obtain the total number F of data collection terminals connected to the automated data annotation platform, set the data collection terminal with at least one marked feature cluster among the F data collection terminals as the warning terminal, and accumulate the total number R of warning terminals. For each warning terminal, accumulate the total number K of feature clusters formed in the corresponding anomaly recognition database, and the total number D of marked feature clusters among the K feature clusters;

[0066] Step S5-2: Evaluate and calculate the updated warning value of the automated data annotation platform ξ=(R / F)×[D1 / K1+D2 / K2+...+D R / K R ]; where K1, K2, ..., K RThey represent the total number of feature clusters formed in the anomaly recognition database of the 1st, 2nd, ..., Rth warning terminals respectively; D1, D2, ..., D R They respectively represent the total number of marked feature clusters in the anomaly recognition database of the 1st, 2nd, ..., Rth warning terminals.

[0067] In order to better implement the above method, a multi-source data annotation system is also proposed, which includes: a target record extraction module, a feature data segment processing module, a feature record extraction module, an anomaly recognition database management module, and an update warning value evaluation module;

[0068] The target record extraction module is used to extract the historical data annotation records with human intervention factors as the final output data annotation results from the automated data annotation platform, and set them as target records; wherein the data annotation results with human intervention factors refer to the data annotation results obtained after manual verification and adjustment of the data annotation results planned to be output by the automated data annotation platform;

[0069] A characteristic data segment processing module is used to extract characteristic data segments with abnormal annotations in each target record based on the difference in data annotation results before and after manual verification and adjustment, and to divide and process the characteristic data segments based on the differences in data collection terminals;

[0070] Feature record extraction module: Based on each target record, the feature adjustment value presented by the automated data annotation platform under human intervention factors is calculated and evaluated; the target record whose corresponding feature adjustment value is greater than the threshold is set as a feature record;

[0071] The anomaly recognition database management module is used to pre-build anomaly recognition databases for each data collection terminal connected to the automated data annotation platform; extract each feature data set corresponding to each target adjustment terminal from each feature record, and store it into the anomaly recognition database of each target adjustment terminal; perform trend analysis on the data clustering phenomenon presented in each anomaly recognition database, and mark the corresponding feature clusters;

[0072] The updated warning value evaluation module is used to sort out the label distribution of the corresponding feature clusters in each anomaly recognition database, evaluate and calculate the updated warning value of the automated data annotation platform, and feedback to the management terminal when the updated warning value is greater than the warning threshold.

[0073] Among them, the characteristic data segment processing module includes a characteristic data segment identification and extraction unit and a data division management unit;

[0074] A characteristic data segment identification and extraction unit is used to extract characteristic data segments showing abnormal annotations in each target record based on the difference in data annotation results before and after manual verification and adjustment;

[0075] The data division management unit is used to divide and process the characteristic data segments based on the differences in data collection terminals.

[0076] Among them, the update warning value evaluation module includes a feature clustering management unit and an update warning value calculation unit;

[0077] A feature clustering management unit, used to sort out the distribution of labels of corresponding feature clusters in each anomaly recognition database;

[0078] The update warning value calculation unit is used to evaluate and calculate the update warning value of the automatic data annotation platform, and feedback to the management terminal when the update warning value is greater than the warning threshold.

[0079] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-source data annotation method based on artificial intelligence, characterized by: The method comprises: Step S1: extracting the historical data annotation records with human intervention factors as the final output data annotation results from the automated data annotation platform, and setting them as target records; wherein the data annotation results with human intervention factors refer to the data annotation results obtained after manual verification and adjustment of the data annotation results planned to be output by the automated data annotation platform; Step S2: In each target record, based on the difference in data annotation results before and after manual verification and adjustment, extract the characteristic data segments showing anomalies in annotation, and divide and process the characteristic data segments based on the difference in data collection terminals; Step S3: Based on each target record, the feature adjustment value presented by the automated data annotation platform under the human intervention factor is calculated and evaluated; the target record whose corresponding feature adjustment value is greater than the threshold is set as a feature record; Step S4: pre-build an anomaly recognition database for each data collection terminal connected to the automated data annotation platform; extract each feature data set corresponding to each target adjustment terminal from each feature record, and store it into the anomaly recognition database of each target adjustment terminal; perform trend analysis on the data clustering phenomenon presented in each anomaly recognition database, and mark the corresponding feature cluster; Step S5: sort out the distribution of labels of corresponding feature clusters in each anomaly recognition database, evaluate and calculate the updated warning value of the automated data annotation platform, and feedback to the management terminal when the updated warning value is greater than the warning threshold.

2. The multi-source data annotation method based on artificial intelligence according to claim 1, characterized in that: The step S2 comprises: Step S2-1: In each target record, the data labeling result that the automated data labeling platform plans to output based on the data transmitted by the corresponding data collection terminal before manual verification and adjustment is performed is set as the first feature labeling result, and the data labeling result obtained after manual verification and adjustment of the first feature labeling result is set as the second feature labeling result; Step S2-2: for each target record, in the corresponding first feature annotation result, extract the data segments whose corresponding annotation information is different from that presented in the corresponding second feature annotation result; set the data segments as the feature data segments presenting anomalies in the target record; the annotation information includes the annotation style and the data range selected by the annotation; Step S2-3: Divide the data constituting each characteristic data segment according to the corresponding data collection terminals, and obtain a plurality of characteristic data sets corresponding to each data segment, wherein one characteristic data set corresponds to one data collection terminal.

3. The multi-source data annotation method based on artificial intelligence according to claim 2 is characterized in that: The step S3 comprises: Step S3-1: In each target record, the data collection end with the feature data set is input to the automated data annotation platform and set as the target adjustment end in the corresponding target record; Step S3-2: In each target record, accumulate the total number N of data collection terminals that input data into the automated data labeling platform, and the total number m of target adjustment terminals among the N data collection terminals; simultaneously obtain the similarity α between the corresponding first feature labeling result and the corresponding second feature labeling result; calculate the feature adjustment value β = (m / N) × (1 / α) for each target record.

4. The multi-source data annotation method based on artificial intelligence according to claim 3 is characterized in that: The step S4 comprises: Step S4-1: In each anomaly recognition database, feature data sets that satisfy a similarity greater than a similarity threshold are respectively collected to obtain a plurality of feature clusters; in each feature cluster, all feature data sets are sorted according to the time sequence of the corresponding storage into the anomaly recognition database to generate a feature data sequence corresponding to each feature cluster, wherein every two adjacent feature data sets constitute a storage node in the corresponding feature data sequence; Step S4-3: In the feature data sequence of each feature cluster, extract the interval time between the two feature data sets stored in the anomaly recognition database for each storage node in turn, and obtain the average interval time of all storage nodes; Step S4-4: When, in a feature data sequence of a certain feature cluster, the average interval duration is less than a duration threshold, or the proportion of storage nodes whose corresponding interval duration is less than the average interval duration in all storage nodes contained in the feature data sequence is greater than the proportion threshold, the certain feature cluster is marked.

5. The multi-source data annotation method based on artificial intelligence according to claim 4 is characterized in that: The step S4 comprises: Step S5-1: Obtain the total number F of data collection terminals connected to the automated data annotation platform, set the data collection terminal with at least one marked feature cluster among the F data collection terminals as the warning terminal, and accumulate the total number R of warning terminals. For each warning terminal, accumulate the total number K of feature clusters formed in the corresponding anomaly recognition database, and the total number D of marked feature clusters among the K feature clusters; Step S5-2: Evaluate and calculate the updated warning value of the automated data annotation platform ξ=(R / F)×[D1 / K1+D2 / K2+...+D R / K R ]; where K1, K2, ..., K R They represent the total number of feature clusters formed in the anomaly recognition database of the 1st, 2nd, ..., Rth warning terminals respectively; D1, D2, ..., D R They respectively represent the total number of marked feature clusters in the anomaly recognition database of the 1st, 2nd, ..., Rth warning terminals.

6. A multi-source data annotation system, used to execute the multi-source data annotation method based on artificial intelligence according to any one of claims 1 to 5, characterized in that: The system comprises: a target record extraction module, a feature data segment processing module, a feature record extraction module, an abnormality recognition database management module, and an update warning value evaluation module; The target record extraction module is used to extract, from the automated data annotation platform, historical data annotation records whose final output data annotation results are human intervention factors, and set them as target records; wherein the data annotation results with human intervention factors refer to the data annotation results obtained after manual verification and adjustment of the data annotation results planned to be output by the automated data annotation platform; The characteristic data segment processing module is used to extract characteristic data segments with abnormal annotations in each target record based on the difference in data annotation results before and after manual verification and adjustment, and divide and process the characteristic data segments based on the difference in data collection terminals; The feature record extraction module: calculates and evaluates the feature adjustment value presented by the automated data annotation platform under the human intervention factor based on each target record; sets the target record whose corresponding feature adjustment value is greater than the threshold as the feature record; The anomaly recognition database management module is used to pre-build an anomaly recognition database for each data collection terminal connected to the automated data annotation platform; extract each feature data set corresponding to each target adjustment terminal from each feature record, and store it into the anomaly recognition database of each target adjustment terminal; perform trend analysis on the data clustering phenomenon presented in each anomaly recognition database, and mark the corresponding feature cluster; The update warning value evaluation module is used to sort out the label distribution of the corresponding feature clusters in each abnormal recognition database, evaluate and calculate the update warning value of the automatic data annotation platform, and feedback to the management terminal when the update warning value is greater than the warning threshold.

7. A multi-source data annotation system according to claim 6, characterized in that: The characteristic data segment processing module includes a characteristic data segment identification and extraction unit and a data division management unit; The characteristic data segment identification and extraction unit is used to extract characteristic data segments showing abnormal annotations in each target record based on the difference in data annotation results before and after manual verification and adjustment; The data division management unit is used to divide the characteristic data segments based on the differences in data collection terminals.

8. A multi-source data annotation system according to claim 6, characterized in that: The update warning value evaluation module includes a feature clustering management unit and an update warning value calculation unit; The feature clustering management unit is used to sort out the distribution of labels of corresponding feature clusters in each anomaly recognition database; The update warning value calculation unit is used to evaluate and calculate the update warning value of the automatic data annotation platform, and feedback to the management terminal when the update warning value is greater than the warning threshold.

Images