A large model assisted data labeling quality checking system and method

Abstract

The application discloses a large model auxiliary data labeling quality checking system and method, and relates to the technical field of artificial intelligence, and comprises the following steps: S1: labeling data field frequency statistics; S2: field distribution cumulative ranking generation; S3: labeling subject error ratio calculation; S4: dynamic threshold and sensitivity parameter generation, a threshold compensation coefficient is obtained according to a historical error ratio query compensation mapping table and added to a basic confidence threshold value; combined with an artificial and reference label cumulative probability ranking difference absolute value, a sensitivity coefficient is mapped to generate a dynamic checking threshold; and S5: quality deviation determination and state marking. The application can quantize the rarity degree of a label value by traversing a full amount of labeling data sets to count the occurrence times of independent values and calculate individual probability values, and can reduce model inference confidence fluctuation when processing rare categories.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a system and method for verifying the quality of large-scale model-assisted data annotation. Background Technology

[0002] Assisted data annotation quality verification refers to a technical solution that uses pre-trained large-scale language or visual models to automatically review manually annotated results. This solution addresses labeling errors, boundary deviations, and category confusion that occur during the data annotation process. Specifically, it calls a large model to independently infer reference labels from the original data samples, compares the reference labels with the manually annotated labels field by field, filters out samples that exceed the limit of the confidence threshold and need to be reviewed, and locates the specific locations of the labeling inconsistencies based on the semantic interpretation information output by the large model. This completes the quality screening and anomaly identification of the annotated dataset.

[0003] Chinese patent document CN119577388A discloses a method, apparatus, and storage medium for verifying the quality of autonomous driving annotation data. The method includes: acquiring segment data of a vehicle's driving and vehicle decision annotation data of the segment data, wherein the segment data is used to describe at least the driving situation of the vehicle and the game-like obstacles interacting with the vehicle; verifying the vehicle decision annotation data based on the segment data and outputting the verification result; wherein the verification includes at least one of open-loop verification and closed-loop verification, wherein the open-loop verification is frame-level verification of the segment data, and the closed-loop verification is segment-level verification of the segment data. This can improve the quality of data annotation and enhance the efficiency of the closed-loop control data.

[0004] The existing technology has the following problems:

[0005] Existing technologies rely on preset confidence thresholds to screen samples for review. This static mechanism cannot distinguish between high-frequency common labels and low-frequency rare labels. When dealing with rare categories, the model's inference confidence often fluctuates significantly, causing normal samples to be incorrectly labeled as anomalous, increasing the burden of manual review. Furthermore, the existing process applies a uniform standard to all annotators, ignoring individual historical performance differences. High-error operators and stable operators receive the same verification intensity, resulting in wasted review resources on reliable data, while errors from unstable sources may be missed. For example, if an annotator frequently confuses a specific category, and the model's confidence is high, the static threshold cannot capture such regular errors, leading to accumulated bias in the dataset. This rigid process results in inefficient resource allocation, and inconsistent quality control standards across different data distributions and personnel affect the final model training effect. Summary of the Invention

[0006] The main objective of this invention is to provide a system and method for verifying the quality of auxiliary data annotation for large models, which can effectively solve the problem of large fluctuations in the confidence of model inference when processing rare class data.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A method for quality verification of auxiliary data annotation in large models, comprising the following steps:

[0009] S1: Frequency statistics of labeled data fields: Obtain the structured field values ​​of the labeled records to be verified, traverse the entire dataset to count the occurrence of each independent value, calculate the ratio of the occurrence of a single field value to the total number of samples in the field, and obtain the individual probability value of the field value;

[0010] S2: Cumulative Ranking Generation for Field Distribution: Sort individual field values ​​in descending order by their probability values, sum the sorted individual probability values, and generate a cumulative probability ranking for each independent value;

[0011] S3: Calculation of the error ratio of the annotation subject: Call the annotation subject identifier, retrieve the number of erroneous markers and the total number of markers in the most recent N data windows, calculate the ratio between the two, and obtain the historical error ratio;

[0012] S4: Dynamic Threshold and Sensitivity Parameter Generation: The threshold compensation coefficient is obtained by querying the compensation mapping table based on the historical error ratio, and added to the basic confidence threshold; combined with the absolute value of the difference between the cumulative probability ranking of manual and reference labels, it is mapped to the sensitivity coefficient to generate the dynamic verification threshold;

[0013] S5: Quality Deviation Judgment and Status Marking: Calculate the sensitivity coefficient based on the cumulative probability ranking, multiply the original difference identifier by the coefficient to obtain the weighted difference score, compare it with the dynamic verification threshold, and generate a review status flag bit.

[0014] Preferably, the individual probability value P of the field value includes the frequency Φ of the specific field value and the total number of samples N, and the cumulative probability ranking R includes the independent value sorting sequence I and the cumulative probability value P. cum The historical error ratio Including the number of error markers within the window Φ err Total number of markers N in the window win The dynamic verification threshold T includes the basic confidence benchmark T0, the error compensation coefficient α, and the difference sensitivity coefficient β. The verification status flag S includes the qualified status identifier S. + Non-conforming status indicator S - Pending status indicator S0.

[0015] Preferably, the frequency statistics of the labeled data fields in S1 specifically includes: first, obtaining all structured field values ​​in the labeled records to be verified, wherein the structured field values ​​are numerical or text identifiers corresponding to the standardized fields preset during the labeling process; then, traversing the entire labeled data set and counting the occurrence times of each independent field value one by one; finally, performing a division operation between the occurrence times of a single field value and the total sample count value corresponding to that field, and the calculation result is the individual probability value corresponding to that field value, wherein the individual probability value retains four decimal places.

[0016] Preferably, the S2 field distribution cumulative ranking generation specifically includes: based on the individual probability values ​​of each field value obtained in S1, sorting them in descending order from largest to smallest to obtain a field value probability sorting sequence; then, according to the order of the sorting sequence, sequentially accumulating the individual probability values ​​corresponding to each field value; during the accumulation process, recording the accumulated probability value corresponding to each independent field value in real time; and generating a unique cumulative probability ranking value for each independent value based on the magnitude of the accumulated probability value, with the larger the accumulated probability value, the smaller the cumulative probability ranking value.

[0017] Preferably, the S3 annotation subject error ratio calculation specifically includes: calling the unique identifier of the annotation subject through the large model interface. The identifier is a unique code corresponding to the annotation personnel or annotation equipment. Based on the identifier, a statistical window consisting of the most recent X annotation data is retrieved, where X is a positive integer with a value range of 50≤X≤500. The error mark count value and the total mark count value of the annotation data in the window are counted. The error mark count value and the total mark count value are divided. The calculation result is the historical error ratio value corresponding to the annotation subject. The historical error ratio value is retained to three decimal places.

[0018] Preferably, in the generation of dynamic threshold and sensitivity parameters in S4, the acquisition of the threshold compensation coefficient specifically includes: querying a preset compensation mapping table based on the historical error ratio value obtained in S3. The compensation mapping table is a one-to-one correspondence table between the historical error ratio and the threshold compensation coefficient. The corresponding threshold compensation coefficient is obtained based on the query result. Then, the preset basic confidence threshold and the threshold compensation coefficient are added together to obtain the initial verification threshold.

[0019] Preferably, in the generation of the S4 dynamic threshold and sensitivity parameters, the acquisition of the sensitivity coefficient specifically includes: reading the cumulative probability ranking value corresponding to the manually labeled label and the cumulative probability ranking value corresponding to the reference label, calculating the absolute value of the difference between the two ranking values, and then mapping the absolute value of the difference to the corresponding sensitivity coefficient according to a preset mapping rule. The mapping rule is that the larger the absolute value of the difference, the larger the sensitivity coefficient value, and the sensitivity coefficient value range is 0.5≤β≤2.0.

[0020] Preferably, in the generation of the dynamic threshold and sensitivity parameter in S4, the generation of the dynamic verification threshold value specifically includes multiplying the initial verification threshold obtained in S4 with the sensitivity coefficient, and the calculation result is the dynamic verification threshold value. The dynamic verification threshold value is updated in real time, and the update frequency is consistent with the frequency of new annotation data.

[0021] Preferably, the S5 quality deviation judgment and status marking specifically includes: calculating a sensitivity coefficient based on the cumulative probability ranking value obtained in S2 and combining it with a preset algorithm; multiplying the original difference identifier value corresponding to the labeled data with the sensitivity coefficient to obtain a weighted difference score; comparing the weighted difference score with the dynamic verification threshold value generated in S4; and generating a qualified status mark S if the weighted difference score is less than the dynamic verification threshold value. + If the weighted difference score is greater than the dynamic verification threshold value, then an unqualified status identifier S is generated. - If the weighted difference score equals the dynamic verification threshold value, then a pending state identifier S0 is generated.

[0022] Preferably, it includes a frequency statistics module, a distribution generation module, a main calculation module, a dynamic parameter generation module, and a quality judgment module;

[0023] The frequency statistics module obtains the structured field values ​​of the labeled records to be verified, traverses the entire labeled data set to count the occurrence of each independent value, and performs a division operation between the occurrence of a single field value and the total sample count of the field to obtain the individual probability value of the field value.

[0024] The distribution generation module sorts the individual probability values ​​of the field values ​​in descending order, accumulates the individual probability values ​​according to the sorting order, and generates a cumulative probability ranking value for each independent value.

[0025] The main calculation module calls the main identity identifier, retrieves the error mark count and total mark count in the most recent X data windows, and performs a division operation to obtain the historical error ratio value.

[0026] The dynamic parameter generation module determines the interval segment by comparing the historical error ratio value with the interval boundary value of the compensation mapping table, extracts the threshold compensation coefficient corresponding to the interval segment, performs an addition operation on the basic confidence threshold and the threshold compensation coefficient, reads the cumulative probability ranking value of the manual label and the reference label, performs a subtraction operation to calculate the absolute value of the difference and maps it to the sensitivity coefficient, performs a multiplication operation on the sensitivity coefficient and the original difference identifier value to generate the verification threshold value.

[0027] The quality judgment module calculates the sensitivity coefficient based on the cumulative probability ranking value, multiplies the original difference identifier value by the sensitivity coefficient to obtain the weighted difference score, compares the weighted difference score with the verification threshold value, and generates a review status flag bit.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This invention provides a large model-assisted data annotation quality verification system and method. By traversing the entire annotated data set, counting the occurrence frequency of independent values ​​and calculating individual probability values, a cumulative ranking value of field distribution is constructed, which can quantify the rarity of label values.

[0030] 2. This invention provides a large model-assisted data annotation quality verification system and method, which calls the annotation subject's identity identifier to retrieve the error mark count value in the most recent window, obtains the historical error ratio value, and realizes real-time perception of the current status of the annotation personnel.

[0031] 3. This invention provides a large model-assisted data annotation quality verification system and method, which performs an addition operation on the basic confidence threshold and the threshold compensation coefficient, and combines the cumulative probability ranking numerical difference to map the sensitivity coefficient to generate a dynamic verification threshold value, thereby avoiding the defect that fixed thresholds cannot adapt to changes in data distribution.

[0032] 4. This invention provides a large model-assisted data annotation quality verification system and method. It calculates the sensitivity coefficient based on the cumulative probability ranking value, and multiplies the original difference label value with the sensitivity coefficient to obtain a weighted difference score, so that the differences generated by rare labels or high error personnel are given higher weight.

[0033] 5. This invention provides a large model-assisted data annotation quality verification system and method, which ultimately compares the weighted difference score with the dynamic verification threshold value to generate a verification status flag, thereby improving the targeting and accuracy of quality screening, effectively reducing the false negative rate and false positive rate, and ensuring the overall consistency of the dataset. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0035] Figure 2 This is a schematic diagram of the frequency statistics process for labeled data fields in this invention;

[0036] Figure 3 This is a schematic diagram of the cumulative ranking generation process for field distribution in this invention;

[0037] Figure 4 This is a schematic diagram of the annotation subject error ratio calculation process of the present invention;

[0038] Figure 5 This is a schematic diagram of the dynamic threshold and sensitivity parameter generation process of the present invention;

[0039] Figure 6 This is a schematic diagram of the quantity deviation determination and status marking process of the present invention;

[0040] Figure 7 This is a schematic diagram of the system flow of the present invention. Detailed Implementation

[0041] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0042] Example 1: Obtaining the structured field values ​​of the annotation records to be verified. First, the application scenario of the annotation data to be verified is defined as a text classification annotation task. The structured fields are preset to three standardized fields: "Annotation Category," "Annotation Confidence," and "Annotation Time." "Annotation Category" is a text identifier (e.g., "Positive," "Negative," "Neutral"), while "Annotation Confidence" and "Annotation Time" are numerical fields. 1000 user comment annotation records are selected as the full set of data to be verified. First, the three structured field values ​​of each annotation record are read through the large model interface and stored in a temporary database. Then, a traversal program is started to iterate through the 1000 records one by one. For the "Annotation Category" field, the frequency of occurrence of each independent text identifier is counted. Assuming "Positive" appears 420 times, "Negative" appears 380 times, and "Neutral" appears 200 times. For the "Annotation Confidence" field, independent values ​​are divided into intervals of 0.1, and the frequency of occurrence in each interval is counted. Assuming the 0.8-0.9 interval appears 350 times, the 0.7-0.8 interval appears 280 times, and the remaining intervals total 370 times. For the "Annotation Time" field, independent values ​​were defined by hour, and the number of annotations for each hour was counted. For example, 150 annotations occurred between 10-11 AM, 180 between 11-12 PM, and a total of 670 annotations for the remaining time periods. The total sample size for each field was then calculated, with 1000 samples for each of the three fields. The probability of each independent field value was then divided by the total sample size for that field. The individual probability for the "positive" category was calculated as 420 ÷ 1000 = 0.4200, and for the "negative" category as 380 ÷ 1000 = 0.4200. 00 = 0.3800, "neutral" is 200 ÷ 1000 = 0.2000, "label confidence level" is 350 ÷ 1000 = 0.3500 for the 0.8-0.9 range and 280 ÷ 1000 = 0.2800 for the 0.7-0.8 range, "labeling time" is 150 ÷ ​​1000 = 0.1500 for 10-11 o'clock and 180 ÷ 1000 = 0.1800 for 11-12 o'clock. All calculation results are rounded to four decimal places to obtain the individual probability value corresponding to each field value.

[0043] Example 2 clarifies the individual probability value P, cumulative probability ranking R, and historical error ratio of the field values. The parameters constituting the dynamic verification threshold T and the review status flag S are first analyzed by breaking down the specific value basis of each parameter. In the individual probability value P of the field value, the frequency Φ of the specific field value is selected as the occurrence count of the "positive" category in the "labeled category" field, which is 420. The total number of samples N is selected as the total number of samples in this field, which is 1000. P is calculated as 420 ÷ 1000 = 0.4200. In the cumulative probability ranking R, the independent value sorting sequence I is sorted by P from largest to smallest as "positive" (0.4200), "negative" (0.3800), and "neutral" (0.2000). The cumulative probability value Pcum is calculated as follows: 0.4200, 0.4200 + 0.3800 = 0.8000, and 0.8000 + 0.2000 = 1.0000. Historical error ratio... In the middle, the number of error markers Φ within the window err Select 12 erroneous tags from the most recent 200 annotation records of a certain annotator, and the total number of tags N in the window. win =200, calculate =12÷200=0.060; In the dynamic verification threshold T, the basic confidence benchmark T0 is preset to 0.3, and is set with reference to historical verification data of similar annotation tasks. The error compensation coefficient α is based on... =0.060 Querying the compensation mapping table yields 0.05, and the difference sensitivity coefficient β is mapped to 1.2 based on the absolute value of the ranking difference; in the review status flag S, the qualified status is S + Set to "0", the non-compliance status indicator S - Set to "1", set the pending state flag S0 to "2", clarify the specific values ​​and calculation methods of each parameter, and determine the composition and value specifications of each parameter.

[0044] Example 3, S1 annotation data field frequency statistics specifically includes: first, obtaining all structured field values ​​from the annotation records to be verified; selecting the data to be verified in the image target detection annotation task; the preset standardized fields include "target category", "target coordinates", and "annotation precision", where "target category" is a text identifier (e.g., "pedestrian", "vehicle", "road sign"), "target coordinates" is a numerical identifier (e.g., x1=100, y1=200, x2=300, y2=400), and "annotation precision" is a numerical identifier (range 0-1); selecting 800 image annotation records as the full dataset; batch extracting the three structured field values ​​of each record through the data interface; storing them in an Excel spreadsheet and archiving them by field; then starting the statistical program to traverse the full annotation data set in the Excel spreadsheet; for the "target category" field, reading the category identifier of each record row by row; incrementing the count by 1 when encountering the same identifier; after the traversal, the statistics show that "pedestrian" appears 320 times, "vehicle" appears 380 times, and "road sign" appears 100 times; for the "target category" field, the statistics show that "pedestrian" appears 320 times, "vehicle" appears 380 times, and "road sign" appears 100 times; for the "target category" field, the statistics show that "pedestrian" appears 320 times, "vehicle" appears 380 times, and "road sign" appears 100 times. For the "Annotation Coordinates" field, independent values ​​were divided into intervals of 50 for x1 coordinates. The frequency of occurrence in each interval was counted. The interval x1=50-100 appeared 150 times, and the interval 100-150 appeared 220 times. For the "Annotation Precision" field, independent values ​​were divided into intervals of 0.05. The interval 0.9-0.95 appeared 280 times, and the interval 0.85-0.9 appeared 210 times. The total sample count for each field was then counted. The total sample size for all three fields was 800. Finally, the frequency of occurrence of each field value was counted. The number is divided by the total sample count corresponding to the field. The calculation is 320 ÷ 800 = 0.4000 for "pedestrians", 380 ÷ 800 = 0.4750 for "vehicles", 100 ÷ 800 = 0.1250 for "road signs", 150 ÷ ​​800 = 0.1875 for the x1 = 50-100 range, and 280 ÷ 800 = 0.3500 for the annotation precision range of 0.9-0.95. All calculation results are rounded to four decimal places to obtain the individual probability value corresponding to the field value.

[0045] The S2 field distribution cumulative ranking generation specifically includes the following steps: Based on the individual probability values ​​of each field obtained in S1, select the individual probability values ​​of the "Target Category" field, which are 0.4750 for "Vehicle", 0.4000 for "Pedestrian", and 0.1250 for "Road Sign". Start the sorting program to arrange them in descending order from largest to smallest, resulting in the field value probability ranking sequence: "Vehicle" (0.4750), "Pedestrian" (0.4000), and "Road Sign" (0.1250). Then, according to the order of this ranking sequence, start the accumulation program to sequentially accumulate the individual probability values ​​corresponding to each field value. First, accumulate the probability value of the first field value, "Vehicle", to obtain the accumulated probability value of 0.4750, and record the field value corresponding to this value as "Vehicle". Then, accumulate the probability value of the second field value, "Pedestrian". 0.4750 + 0.4000 = 0.8750, and the corresponding field value is recorded as "pedestrian". Finally, the probability value of the third field value, "road sign", is accumulated: 0.8750 + 0.1250 = 1.0000, and the corresponding field value is recorded as "road sign". During the accumulation process, each independent field value is bound and stored with its corresponding accumulated probability value in real time. Then, a unique cumulative probability ranking value is generated for each independent value based on the magnitude of the accumulated probability value. The larger the accumulated probability value, the smaller the cumulative probability ranking value. Among them, the accumulated probability of "vehicle" is the largest at 0.4750, and its ranking is 1; the accumulated probability of "pedestrian" is the second largest at 0.8750, and its ranking is 2; the accumulated probability of "road sign" is the smallest at 1.0000, and its ranking is 3. This completes the generation of the cumulative probability ranking for each independent value.

[0046] The S3 annotation subject error ratio calculation specifically includes: calling the unique identifier of the annotation subject through the large model interface; selecting three annotators in the image annotation task and assigning them unique codes BZ001, BZ002, and BZ003 respectively; and assigning unique codes SB001 and SB002 to the annotation equipment. Assuming the current search is for annotation data of annotator BZ001, the search program is initiated based on this identifier. The statistical window X is set to 200 (X values ​​conform to the range 50≤X≤500, combined with the annotator's average daily annotation volume of 250). The most recent 200 annotation data of this annotator are retrieved, stored in a temporary database, and the verification result (correct or incorrect) of each data point is marked. The statistical program is then initiated to iterate through these 200 data points one by one, counting the error mark count. Assuming that 10 annotation data points have target category annotation errors and 3 have coordinate annotation errors, the total error mark count is Φ. err =13, Total marker count N within the window win=200. Perform a division operation between the error mark count and the total mark count, and calculate 13 ÷ 200 = 0.065. The result is rounded to three decimal places, and the historical error ratio value corresponding to annotator BZ001 is 0.065. Similarly, retrieve the most recent 300 annotation data of annotator BZ002. The error mark count is 18. Calculate 18 ÷ 300 = 0.060, and the historical error ratio is 0.060. Retrieve the most recent 150 annotation data of annotation device SB001. The error mark count is 9. Calculate 9 ÷ 150 = 0.060, and the historical error ratio is 0.060. The calculation of the historical error ratio of the annotation subject is completed.

[0047] In the generation of dynamic threshold and sensitivity parameters in S4, the acquisition of the threshold compensation coefficient specifically includes: based on the historical error ratio values ​​obtained in S3, a preset compensation mapping table is established. This mapping table establishes a one-to-one correspondence between historical error ratios and threshold compensation coefficients. The mapping rules are set as follows: historical error ratios of 0.000-0.050 correspond to a compensation coefficient of 0.03, 0.051-0.080 correspond to a compensation coefficient of 0.05, 0.081-0.100 correspond to a compensation coefficient of 0.08, and values ​​above 0.101 correspond to a compensation coefficient of 0.10. This mapping table is set with reference to the annotation quality verification data of the past 6 months to ensure that the compensation coefficient is positively correlated with the error ratio. Assuming the current historical error ratio value is 0.065, the query program is launched to retrieve the compensation mapping table and determine that 0.065 falls within the range of 0.051-0.051. For the .080 range, the corresponding threshold compensation coefficient is 0.05. A preset basic confidence threshold is then established, set with reference to the passing standards of similar annotation tasks. Considering the overall quality of the annotated data, a basic confidence threshold T0 = 0.3 is set. The preset basic confidence threshold and its compensation coefficient are added together, resulting in 0.3 + 0.05 = 0.35, yielding an initial verification threshold of 0.35. If the historical error ratio is 0.045, the compensation coefficient 0.03 is obtained from the mapping table, and 0.3 + 0.03 = 0.33 is calculated, resulting in an initial verification threshold of 0.33. If the historical error ratio is 0.090, the compensation coefficient 0.08 is obtained from the mapping table, and 0.3 + 0.08 = 0.38 is calculated, resulting in an initial verification threshold of 0.38. This completes the calculation of the initial verification threshold.

[0048] In the generation of S4 dynamic threshold and sensitivity parameters, the acquisition of the sensitivity coefficient specifically includes: reading the cumulative probability ranking value corresponding to the manually labeled label and the cumulative probability ranking value corresponding to the reference label; selecting a labeled data from the text sentiment labeling task, where the manually labeled label is "positive" with a cumulative probability ranking of 1, and the reference label (pre-labeled label of the large model) is "neutral" with a cumulative probability ranking of 3; starting the calculation program to calculate the absolute value of the difference between the two ranking values, i.e., |1-3|=2; and then preset the mapping rule, which is set according to the distribution range of the cumulative probability ranking. The correspondence between the absolute value of the ranking difference and the sensitivity coefficient is as follows: an absolute value of difference of 1 corresponds to a sensitivity coefficient of 0.8, an absolute value of difference of 2 corresponds to a sensitivity coefficient of 1.2, and an absolute value of difference of 3 corresponds to a sensitivity coefficient of 1.2. A sensitivity coefficient of 1.6 corresponds to a difference of 4 or more, and a sensitivity coefficient of 2.0 corresponds to a difference of 4 or more. This mapping rule ensures that the larger the absolute value of the difference, the larger the sensitivity coefficient, and the range of the sensitivity coefficient is controlled within 0.5 ≤ β ≤ 2.0. Based on the difference of 2, the mapping rule yields a sensitivity coefficient β = 1.2. If the manually labeled label ranks 2 and the reference label ranks 5, the difference of 3 corresponds to a sensitivity coefficient of 1.6. If the manually labeled label ranks 4 and the reference label ranks 5, the difference of 1 corresponds to a sensitivity coefficient of 0.8. If the manually labeled label ranks 1 and the reference label ranks 5, the difference of 4 corresponds to a sensitivity coefficient of 2.0. All of these values ​​meet the range requirements, thus completing the acquisition of the sensitivity coefficient.

[0049] In the generation of dynamic threshold and sensitivity parameters in S4, the generation of the dynamic verification threshold value specifically includes multiplying the initial verification threshold obtained in S4 by the sensitivity coefficient. Selecting the initial verification threshold of annotation personnel BZ001 as 0.35, with a corresponding sensitivity coefficient β=1.2, the multiplication operation program is started, calculating 0.35×1.2=0.42, resulting in a dynamic verification threshold value of 0.42. If the initial verification threshold is 0.33 and the sensitivity coefficient is 0.8, 0.33×0.8=0.264 is calculated, resulting in a dynamic verification threshold of 0.264. If the initial verification threshold is 0.38 and the sensitivity coefficient is 1.6, the calculation... Calculate 0.38 × 1.6 = 0.608, and obtain the dynamic verification threshold as 0.608. Set the dynamic verification threshold value to be updated in real time, with the update frequency consistent with the frequency of new annotation data. Assuming that the annotation task adds 50 annotation data every 10 minutes, set the dynamic verification threshold to be updated once every 10 minutes. Each time it is updated, re-obtain the current historical error ratio, the ranking difference between manual and reference labels, recalculate the initial verification threshold and sensitivity coefficient, and then perform multiplication to obtain the new dynamic verification threshold. This ensures that the dynamic verification threshold can adapt to the quality changes of real-time annotation data, and completes the generation of the dynamic verification threshold value.

[0050] The S5 quality deviation judgment and status labeling specifically includes: based on the cumulative probability ranking value obtained in S2, a sensitivity coefficient is calculated using a preset algorithm. Three labeled data points from the text classification labeling task are selected. The first data point is manually labeled "positive" with a cumulative probability ranking of 1, and the reference label is "neutral" with a ranking of 3. The absolute value of the difference, |1-3|=2, is calculated, resulting in a sensitivity coefficient of 1.2. The second data point is manually labeled "negative" with a ranking of 2, and the reference label is "positive" with a ranking of 1. The absolute value of the difference is 1, resulting in a sensitivity coefficient of 0.8. The third data point is manually labeled "neutral" with a ranking of 3, and the reference label is "neutral" with a ranking of 3. The absolute value of the difference is 0, resulting in a sensitivity coefficient of 0.5. Finally, an original difference indicator value is set, which is determined based on the degree of difference between the manually labeled data and the reference label. The larger the difference, the larger the numerical value, ranging from 0 to 1. The original difference indicator for the first data point is 0.35, for the second it is 0.28, and for the third it is 0.15. Multiplying the original difference indicator value corresponding to the labeled data by the sensitivity coefficient, the weighted difference score is 0.42 for the first data point (0.35 × 1.2 = 0.42), 0.224 for the second data point (0.28 × 0.8 = 0.224), and 0.075 for the third data point (0.15 × 0.5 = 0.075). Comparing these weighted difference scores with the dynamic verification threshold value of 0.42 generated in S4, the weighted difference score for the first data point (0.42 = dynamic verification threshold 0.42) generates a pending state indicator S0; the weighted difference score for the second data point (0.224 < 0.42) generates a qualified state indicator S0. + The third data point has a weighted difference score of 0.075 < 0.42, generating a qualified status indicator S. + If the weighted difference score of a certain data point is 0.45 > 0.42, then an unqualified status label S is generated. - Complete the quality deviation judgment and status marking.

[0051] In Example 4, the frequency statistics module obtains the structured field values ​​of the annotation records to be verified. It selects the data to be verified from the speech annotation task. The structured fields are preset as "speech category," "annotation clarity," and "annotation duration." "Speech category" is a text identifier (e.g., "instruction speech," "idle speech," "invalid speech"), while "annotation clarity" and "annotation duration" are numerical identifiers. The module reads these three structured field values ​​from 500 speech annotation records in batches through the data interface, stores them in the local database, and categorizes them by field. It then starts a traversal program to iterate through the 500 full annotation data sets one by one. For the "speech category" field, it reads the category identifier and counts each record, finding that "instruction speech" appears 220 times and "idle speech" appears 220 times. The "Invalid Speech" appeared 200 times, and the "Annotation Clarity" appeared 80 times. For the "Annotation Clarity" field, independent values ​​were divided into intervals of 0.1, and the frequency of occurrence in each interval was counted: 180 times in the 0.9-1.0 interval and 150 times in the 0.8-0.9 interval. For the "Annotation Duration" field, independent values ​​were divided into intervals of 10 seconds: 160 times in the 10-20 second interval and 140 times in the 20-30 second interval. The total sample count for each field was then counted, both being 500. The frequency of each field value was divided by the total sample count for that field: "Command Speech" was calculated as 220 ÷ 500 = 0.4400, and "Annotation Clarity" (0.9-1.0 interval) was calculated as 180 ÷ 500 = 0.36. 00, for the "annotation duration" of 10-20 seconds, 160 ÷ 500 = 0.3200, yielding the individual probability value of the field value; the distribution generation module sorts the individual probability values ​​of the field value in descending order: "command voice" 0.4400, "casual voice" 0.4000, "invalid voice" 0.1600, and accumulates the individual probability values ​​according to the sorting order: 0.4400, 0.8400, 1.0000, generating cumulative probability ranking values ​​of 1, 2, and 3 for each independent value; the main calculation module calls the annotation subject identity identifier BZ004, sets X=250, retrieves the most recent 250 data windows, counts the error mark count value of 12 and the total mark count value of 250, and executes 12 ÷ 250 = 0.048. The historical error ratio value of 0.048 is obtained. The dynamic parameter generation module queries the compensation mapping table based on the historical error ratio of 0.048 to determine the corresponding threshold compensation coefficient of 0.03. The basic confidence threshold is preset to 0.3. 0.3 + 0.03 = 0.33 is executed. The manual label ranking 1 and the reference label ranking 3 are read. The absolute value of the difference 2 is calculated and mapped to the sensitivity coefficient of 1.2. The original difference identifier is 0.3. 0.3 × 1.2 = 0.36 is executed to generate the verification threshold value of 0.36. The quality judgment module calculates the sensitivity coefficient of 1.2 based on the cumulative probability ranking and multiplies it with the original difference identifier of 0.3 to obtain the weighted difference score of 0.36. It is compared with the verification threshold value of 0.36 to generate the review status flag S0.

[0052] The working principle of this invention is as follows: First, by traversing the entire labeled dataset, the occurrence frequency of independent values ​​is counted and individual probability values ​​are calculated to construct a cumulative ranking value for field distribution, which can quantify the rarity of label values. Then, the error mark count value within the most recent window is retrieved by calling the labeling subject's identity identifier to obtain the historical error ratio value, enabling real-time perception of the labeler's current status. Next, the basic confidence threshold and threshold compensation coefficient are added together, and the difference in cumulative probability ranking values ​​is combined with a sensitivity coefficient to generate a dynamic verification threshold value, avoiding the defect of fixed thresholds being unable to adapt to changes in data distribution. Second, the sensitivity coefficient is calculated based on the cumulative probability ranking value, and the original difference identifier value is multiplied by the sensitivity coefficient to obtain a weighted difference score, so that the differences generated by rare labels or high-error personnel are given higher weight. Finally, the weighted difference score is compared with the dynamic verification threshold value to generate a review status flag, improving the targeting and accuracy of quality screening, effectively reducing the false negative rate and false positive rate, and ensuring the overall consistency of the dataset.

[0053] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for verifying the quality of auxiliary data annotation for large models, comprising the following steps, characterized in that: S1: Frequency statistics of labeled data fields: Obtain the structured field values ​​of the labeled records to be verified, traverse the entire dataset to count the occurrence of each independent value, calculate the ratio of the occurrence of a single field value to the total number of samples in the field, and obtain the individual probability value of the field value; S2: Cumulative Ranking Generation for Field Distribution: Sort individual field values ​​in descending order by their probability values, sum the sorted individual probability values, and generate a cumulative probability ranking for each independent value; S3: Calculation of the error ratio of the annotation subject: Call the annotation subject identifier, retrieve the number of erroneous markers and the total number of markers in the most recent N data windows, calculate the ratio between the two, and obtain the historical error ratio; S4: Dynamic threshold and sensitivity parameter generation: The threshold compensation coefficient is obtained by querying the compensation mapping table based on the historical error ratio and added to the basic confidence threshold; By combining the absolute value of the cumulative probability ranking difference between manual and reference labels, a sensitivity coefficient is mapped to generate a dynamic verification threshold; S5: Quality Deviation Judgment and Status Marking: Calculate the sensitivity coefficient based on the cumulative probability ranking, multiply the original difference identifier by the coefficient to obtain the weighted difference score, compare it with the dynamic verification threshold, and generate a review status flag bit.

2. The method for verifying the quality of auxiliary data annotation for large models according to claim 1, characterized in that: The individual probability value P of the field value includes the frequency Φ of the specific field value and the total number of samples N. The cumulative probability ranking R includes the independent value sorting sequence I and the cumulative probability value P. cum The historical error ratio Including the number of error markers within the window Φ err Total number of markers N in the window win The dynamic verification threshold T includes the basic confidence benchmark T0, the error compensation coefficient α, and the difference sensitivity coefficient β. The verification status flag S includes the qualified status identifier S. + Non-conforming status indicator S - Pending status indicator S0.

3. The method for verifying the quality of auxiliary data annotation for large models according to claim 2, characterized in that: The frequency statistics of the S1 labeled data fields specifically include: first, obtaining all structured field values ​​in the labeled records to be verified. The structured field values ​​are numerical or text identifiers corresponding to the standardized fields preset during the labeling process. Then, traversing the entire labeled data set, counting the occurrence times of each independent field value one by one. Finally, performing a division operation between the occurrence times of a single field value and the total sample count value corresponding to that field, the calculation result is the individual probability value corresponding to that field value. The individual probability value retains four decimal places.

4. The method for verifying the quality of auxiliary data annotation for large models according to claim 2, characterized in that: The S2 field distribution cumulative ranking generation specifically includes: based on the individual probability values ​​of each field value obtained in S1, sorting them in descending order to obtain a field value probability sorting sequence; then, according to the order of this sorting sequence, accumulating the individual probability value corresponding to each field value in turn; during the accumulation process, recording the accumulated probability value corresponding to each independent field value in real time; and generating a unique cumulative probability ranking value for each independent value based on the magnitude of the accumulated probability value, with the larger the accumulated probability value, the smaller the cumulative probability ranking value.

5. The method for verifying the quality of auxiliary data annotation for large models according to claim 4, characterized in that: The S3 annotation subject error ratio calculation specifically includes: calling the unique identifier of the annotation subject through the large model interface. The identifier is a unique code corresponding to the annotation personnel or annotation equipment. Based on the identifier, a statistical window consisting of the most recent X annotation data is retrieved, where X is a positive integer with a value range of 50≤X≤500. The error mark count value and the total mark count value of the annotation data in the window are counted. The error mark count value and the total mark count value are divided. The calculation result is the historical error ratio value corresponding to the annotation subject.

6. The method for verifying the quality of auxiliary data annotation for large models according to claim 5, characterized in that: In the generation of dynamic threshold and sensitivity parameters in S4, the acquisition of the threshold compensation coefficient specifically includes: querying a preset compensation mapping table based on the historical error ratio value obtained in S3. The compensation mapping table is a one-to-one correspondence table between the historical error ratio and the threshold compensation coefficient. The corresponding threshold compensation coefficient is obtained based on the query result. Then, the preset basic confidence threshold and the threshold compensation coefficient are added together to obtain the initial verification threshold.

7. The method for verifying the quality of auxiliary data annotation for large models according to claim 6, characterized in that: In the generation of the S4 dynamic threshold and sensitivity parameters, the acquisition of the sensitivity coefficient specifically includes reading the cumulative probability ranking value corresponding to the manually labeled label and the cumulative probability ranking value corresponding to the reference label, calculating the absolute value of the difference between the two ranking values, and then mapping the absolute value of the difference to the corresponding sensitivity coefficient according to the preset mapping rule. The mapping rule is that the larger the absolute value of the difference, the larger the sensitivity coefficient value, and the value range of the sensitivity coefficient is 0.5≤β≤2.

0.

8. The method for verifying the quality of auxiliary data annotation for large models according to claim 7, characterized in that: In the generation of the dynamic threshold and sensitivity parameters in S4, the generation of the dynamic verification threshold value specifically includes multiplying the initial verification threshold obtained in S4 with the sensitivity coefficient, and the result is the dynamic verification threshold value. The dynamic verification threshold value is updated in real time, and the update frequency is consistent with the frequency of new annotation data.

9. The method for verifying the quality of auxiliary data annotation for large models according to claim 1, characterized in that: The S5 quality deviation judgment and status marking specifically includes: calculating a sensitivity coefficient based on the cumulative probability ranking value obtained in S2 and a preset algorithm; multiplying the original difference identifier value corresponding to the labeled data by the sensitivity coefficient to obtain a weighted difference score; comparing the weighted difference score with the dynamic verification threshold value generated in S4; and generating a qualified status mark S if the weighted difference score is less than the dynamic verification threshold value. + If the weighted difference score is greater than the dynamic verification threshold value, then an unqualified status identifier S is generated. - If the weighted difference score equals the dynamic verification threshold value, then a pending state identifier S0 is generated.

10. The large model auxiliary data annotation quality verification system according to claim 1, which is used to implement the large model auxiliary data annotation quality verification method according to any one of claims 1-9, characterized in that: It includes a frequency statistics module, a distribution generation module, a main calculation module, a dynamic parameter generation module, and a quality judgment module; The frequency statistics module obtains the structured field values ​​of the labeled records to be verified, traverses the entire labeled data set to count the occurrence of each independent value, and performs a division operation between the occurrence of a single field value and the total sample count of the field to obtain the individual probability value of the field value. The distribution generation module sorts the individual probability values ​​of the field values ​​in descending order, accumulates the individual probability values ​​according to the sorting order, and generates a cumulative probability ranking value for each independent value. The main calculation module calls the main identity identifier, retrieves the error mark count and total mark count in the most recent X data windows, and performs a division operation to obtain the historical error ratio value. The dynamic parameter generation module determines the interval segment by comparing the historical error ratio value with the interval boundary value of the compensation mapping table, extracts the threshold compensation coefficient corresponding to the interval segment, performs an addition operation on the basic confidence threshold and the threshold compensation coefficient, reads the cumulative probability ranking value of the manual label and the reference label, performs a subtraction operation to calculate the absolute value of the difference and maps it to the sensitivity coefficient, performs a multiplication operation on the sensitivity coefficient and the original difference identifier value to generate the verification threshold value. The quality judgment module calculates the sensitivity coefficient based on the cumulative probability ranking value, multiplies the original difference identifier value by the sensitivity coefficient to obtain the weighted difference score, compares the weighted difference score with the verification threshold value, and generates a review status flag bit.

Images