Data labeling quality and cost collaborative optimization method for multi-label scene

Abstract

The invention discloses a data labeling quality and cost collaborative optimization method for a multi-label scene. The method comprises the following steps: firstly, evaluating sample scarcity and ambiguity to generate a priority list; secondly, constructing a decision arm, and balancing budget distribution between exploration and utilization by adopting a multi-arm Huger machine algorithm; according to the distribution result, differential hierarchy verification is executed, and the higher the depth is, the more complex the verification round and the model combination is; periodically comparing the quality and cost indexes with a threshold value to generate a feedback signal, and driving the strategy to be dynamically adjusted; meanwhile, a sliding window counts historical data to construct a parameter library of difficulty levels, cost coefficients and quality weights, and the parameter library is synchronized to each link to realize adaptive optimization; and the invalid sample budget is automatically returned and preferentially redistributed to the high-revenue queue. According to the method, budget allocation is driven through quality-cost feedback, the verification depth is guided through cost-income, dynamic balance of labeling quality, resource efficiency and cost control is achieved, and the labeling cost performance in a multi-label scene is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for collaboratively optimizing data labeling quality and cost in multi-label scenarios. Background Technology

[0002] With the explosive growth in demand for AI training data, data annotation in multi-label scenarios has become a key bottleneck for model performance. Traditional annotation processes employ a fixed review depth and a uniform budget allocation model, subjecting all samples, regardless of their value, to the same verification process. This results in insufficient budget for high-value samples and wasted resources for low-value samples, leading to a severe disconnect between quality and cost. Existing static allocation mechanisms lack real-time feedback capabilities; when the annotation quality of a certain type of sample falls below expectations, resources cannot be dynamically added, and when costs exceed budgets, losses cannot be stopped in time, resulting in a budget utilization rate generally below 60%.

[0003] To address the challenges of multi-labeling, current technologies primarily rely on manual experience to set priorities, lacking quantitative models for label difficulty, cost coefficients, and quality weights, making it difficult to adapt to the differentiated needs of different label types. Most systems employ an open-loop architecture, where labeling execution and budget decisions are independent, lacking quality-cost linkage monitoring and dynamic strategy adjustment mechanisms, making them prone to getting stuck in local optima over the long term. Invalid sample processing lacks a budget recovery mechanism, resulting in direct cost losses, and resources cannot be recycled. These problems are particularly pronounced at the million-sample scale, making it difficult to continuously improve the cost-effectiveness of labeling. There is an urgent need for a collaborative optimization method that can automatically balance exploration and utilization, dynamically optimize parameter configuration, and achieve a closed-loop budget flow. Summary of the Invention

[0004] To address this, the present invention provides a method for synergistic optimization of data annotation quality and cost in multi-label scenarios, thereby resolving the aforementioned problems in the prior art.

[0005] To achieve the above objectives, a method for collaboratively optimizing data annotation quality and cost in multi-label scenarios includes:

[0006] Step S1: Obtain the sample set to be detected, extract the key features of the sample set to be detected, calculate the value score and risk score, and then obtain the sample priority list;

[0007] Step S2: Construct decision arms based on the sample priority list and the preset multi-level review depth combination, use the multi-armed slot machine algorithm to dynamically calculate the unit budget quality benefits of each decision arm, perform the exploration-exploitation balance decision under the total budget constraint and generate a budget allocation scheme.

[0008] Step S3: Based on the budget allocation scheme, automatically label and verify the sample set according to the review depth level corresponding to the decision arm to generate a process dataset;

[0009] Step S4: Collect quality indicators and cost indicators of each decision arm according to the process dataset and budget cycle. Compare the quality indicators with the preset quality threshold and the cost indicators with the budget threshold to obtain feedback signals. Output the feedback signals to step S2 to adjust the budget allocation strategy for the next cycle.

[0010] Step S5: Configure a differentiated parameter library based on the tag type. The parameter library includes tagging difficulty levels divided according to historical error rates, unit cost coefficients determined according to processing complexity, and quality weight coefficients set according to business importance. Synchronize the differentiated parameter library to steps S1, S2, and S3 to adjust the value score calculation, decision arm cost estimation, and review depth benchmark.

[0011] Furthermore, the process of step S2 includes:

[0012] By cross-combining different priorities and multi-level review depths, multiple decision arms are constructed and the cost parameters of each arm are initialized;

[0013] Based on the historical labeling accuracy and real-time feedback data of each decision arm, the budget benefit index of each decision ratio is dynamically calculated.

[0014] Under the total budget constraint, the current budget is allocated proportionally to the fully tested arm with the highest current revenue and the potentially high-return arm without sufficient testing, and resource allocation decisions are executed to obtain the budget allocation scheme.

[0015] Furthermore, the process of making resource allocation decisions to obtain the budget allocation scheme includes:

[0016] The first proportion of the current budget is allocated to the fully tested arm with the highest current budget revenue indicator. The fully tested arm is the decision arm whose historical allocation sample size has reached a preset threshold.

[0017] The second proportion of the current budget is allocated to the under-tested potential high-yield arms, and the allocation probability of the under-tested arms is determined based on the reciprocal of the square root of their allocated sample size;

[0018] The cumulative budget consumption of a single decision arm shall not exceed the maximum percentage of the total budget.

[0019] Furthermore, the process of step S3 includes:

[0020] Samples are distributed to the differential processing queue based on the priority label corresponding to the decision arm;

[0021] A multi-level verification pipeline is executed based on the differential processing queue to obtain a confidence score;

[0022] When the confidence score is lower than the preset score threshold, an anomaly handling process is triggered to identify and correct correctable errors in order to obtain a corrected result.

[0023] Receive the correction results and cost data of each level of verification, record the annotation results, confidence scores, processing time and actual cost, and generate the process dataset.

[0024] Furthermore, the multi-level verification pipeline sets different verification rounds and model combinations based on the review depth level corresponding to the decision arm, and outputs a confidence score for each verification round.

[0025] Furthermore, the process of step S4 includes:

[0026] Extract quality and cost metrics for each decision arm from the dataset according to the budget cycle. The quality metrics include labeling accuracy and label consistency, and the cost metrics include actual unit cost and budget overrun rate.

[0027] The labeling accuracy and label consistency of each decision arm are normalized and then summed and averaged to obtain the overall quality score; the budget overrun rate and invalid sample cost loss rate are calculated based on the cost indicators.

[0028] The overall quality score is compared with a preset quality threshold to generate a quality warning signal; the budget overrun rate is compared with a budget threshold to generate a cost warning signal; and the unit budget quality revenue change rate in adjacent periods is compared with a preset ratio threshold to generate a strategy optimization signal.

[0029] The quality warning signal, cost warning signal, and strategy optimization signal are combined to form a feedback signal, which is output to step S2 to drive the adjustment of the budget allocation strategy in the next cycle.

[0030] Furthermore, the process of normalizing the labeling accuracy and label consistency of each decision arm and then summing and averaging them to obtain the overall quality score includes:

[0031] The labeling accuracy of each decision arm is standardized and mapped to a unified dimension range, and the label consistency index is processed into percentages.

[0032] The standardized labeling accuracy rate and the percentage label consistency are weighted and summed according to a preset ratio coefficient to obtain the arm-level quality score of each decision arm.

[0033] The overall quality score is obtained by taking the arithmetic mean of the arm-level quality scores of all decision arms.

[0034] Furthermore, the process of step S5 includes:

[0035] Calculate the error rate distribution, processing time distribution, and model impact distribution for each label type based on historical annotation data;

[0036] The labeling difficulty level is divided according to the error rate distribution, the unit cost coefficient is determined according to the processing time distribution, the quality weight coefficient is set according to the model influence distribution and business objectives, and a differentiated parameter library is integrated and constructed.

[0037] The differentiated parameter library is pushed to step S1 to adjust the feature weights in the value score calculation, pushed to step S2 to correct the cost estimate and budget limit of the decision arm, and pushed to step S3 to dynamically set the review depth benchmark for each tag.

[0038] Furthermore, the process of calculating the error rate distribution, processing time distribution, and model influence distribution for each label type based on historical annotation data includes:

[0039] A sliding time window mechanism was used to collect historical annotation data for the most recent N budget periods, and the mean and standard deviation of the dynamic error rate for each label type were calculated.

[0040] Outlier detection and cleaning are performed on the processing time data, and the corrected average processing time per unit sample is calculated as the benchmark for the processing time distribution.

[0041] Based on the performance decay curve of the pre-trained model on the validation set, the sensitivity of each label type to the model accuracy is quantified, and the model influence distribution is generated.

[0042] Furthermore, the method further includes the following steps:

[0043] When an invalid sample is identified in step S3, the budget consumed by the invalid sample is deducted from the current budget cycle and returned to the total budget pool.

[0044] When performing the next cycle budget allocation in step S2, the returned budget is preferentially allocated to the valid sample queues of the same priority, and the redistribution ratio of the returned budget is marked in the budget allocation scheme;

[0045] The redistribution ratio of the returned budget is dynamically adjusted based on the historical unit budget quality benefits of valid samples of the same priority. Sample queues with higher benefits receive a higher proportion of returned budget.

[0046] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention quantifies objective characteristics such as sample scarcity and ambiguity into value and risk scores, providing a comparable decision-making benchmark for resource allocation; based on these scores, a decision arm is constructed, and a multi-armed slot machine algorithm is used to dynamically balance exploration and utilization, allowing the budget to naturally flow towards high-cost-performance combinations; differentiated verification is performed according to the depth of the allocated review, with higher depth levels involving more verification rounds and more complex model combinations, and the resulting confidence scores directly reflecting the annotation quality; quality indicators such as accuracy and consistency are periodically compared with cost indicators, triggering depth upgrade signals when quality falls below a threshold, reduction signals when costs exceed the limit, and exploration signals when returns decrease, driving strategy adjustments; by using a sliding window to statistically analyze historical error rates, processing time, and model impact, difficulty levels are dynamically classified and cost coefficients are determined, ensuring that the parameter library is updated as annotation capabilities change and synchronized with previous stages to adjust evaluation weights and cost benchmarks. The budget return mechanism for invalid samples ensures that wasted resources are automatically returned to the total budget pool and prioritized for allocation to high-return queues of the same priority in the next cycle, forming a virtuous cycle of resource reuse. The entire system drives budget allocation through quality-cost feedback signals, guides verification depth through cost-benefit relationships, and optimizes parameter configuration through historical data distribution. The parameters of each module are mutually restrictive and mutually reinforcing, ultimately achieving a dynamic balance between annotation quality, resource efficiency, and cost control. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a method for collaboratively optimizing data annotation quality and cost in multi-label scenarios provided by this invention;

[0048] Figure 2 This is a flowchart illustrating step S2 in a multi-label scenario data annotation quality and cost collaborative optimization method provided by the present invention.

[0049] Figure 3 This is a flowchart illustrating step S4 in a multi-label scenario data annotation quality and cost collaborative optimization method provided by the present invention. Detailed Implementation

[0050] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0051] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0052] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0053] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0054] Please see Figure 1 As shown, a method for collaboratively optimizing data annotation quality and cost in multi-label scenarios includes:

[0055] Step S1: Obtain the sample set to be detected, extract the key features of the sample set to be detected, calculate the value score and risk score, and then obtain the sample priority list;

[0056] Specifically, the system reads the original sample set to be labeled from the data storage module. Sample types include, but are not limited to, image, text, audio, or video data. The data format is standardized into a unified structure, with each sample containing: a unique sample identifier, original data content, initial label type, and data quality score. Data integrity verification is performed during sample set loading, eliminating invalid samples with format errors or missing data, and generating a list of valid samples. The sample set size is dynamically set according to system configuration, with a single batch processing capacity ranging from 1,000 to 100,000 samples. For each sample in the valid sample list, three types of value-related features and three types of risk-related features are extracted in parallel: Sample Scarcity (Feature_Scarcity): This calculates the proportion of the label type to which the sample belongs in the overall dataset, using the formula: number of samples with this label / total number of samples. Label types with a proportion below 5% have a scarcity score mapped to 1.0; those with a proportion between 5% and 20% are mapped to 0.7; and those with a proportion above 20% are mapped to 0.3. Model Impact (Feature_Impact): Samples are input into the pre-trained model for prediction, and the entropy value of the confidence distribution of the model output is recorded. Samples with an entropy value higher than 0.8 (i.e., difficult for the model to judge) are assigned an impact score of 0.9, entropy values ​​between 0.5 and 0.8 are assigned 0.6, and entropy values ​​lower than 0.5 are assigned 0.3. Business Importance (Feature_Business): The business weight of the sample label is read from the business configuration table. The core business label weight is set to 1.0, the secondary business label weight is set to 0.6, and the auxiliary label weight is set to 0.3.

[0057] Risk-related feature extraction:

[0058] Sample Ambiguity (Feature_Ambiguity): This function calls an ambiguity detection model to score the ambiguity. For image samples, ambiguity is calculated using edge sharpness and signal-to-noise ratio; for text samples, ambiguity is calculated using semantic ambiguity index and entity recognition conflict rate. The output is a standardized score of 0-1, with higher scores indicating greater ambiguity.

[0059] Labeling complexity (Feature_Complexity): This quantifies the number of labels in a sample and the relationships between them. The complexity is 0.3 for a single-label sample, 0.6 for 2-3 related labels, and 1.0 for more than 3 labels or samples with mutually exclusive relationships.

[0060] Historical Error Rate (Feature_ErrorRate): Queries the historical annotation database to extract the average annotation error rate for this label type over the past 30 days. An error rate exceeding 15% is set to 1.0, 5%-15% to 0.6, and below 5% to 0.3.

[0061] The overall score for each sample is calculated using a weighted summation formula, and the weighting coefficients support hot-update configuration.

[0062] Value score calculation: Value_Score = α × Feature_Scarcity + β × Feature_Impact + γ × Feature_Business

[0063] Where α+β+γ=1.0, the default configuration is α=0.4, β=0.3, γ=0.3. The calculation results are normalized to the 0-10 range.

[0064] Risk score calculation: Risk_Score = δ × Feature_Ambiguity + ε × Feature_Complexity + ζ × Feature_ErrorRate

[0065] Where δ+ε+ζ=1.0, the default configuration is δ=0.5, ε=0.3, ζ=0.2. The calculation results are normalized to the 0-10 range.

[0066] Based on the value score and risk score, a two-dimensional matrix mapping is used to divide the priorities into four categories:

[0067] Level P1 (High Value, High Risk): Value_Score ≥ 7.0 and Risk_Score ≥ 7.0;

[0068] Level P2 (High Value, Low Risk): Value_Score ≥ 7.0 and Risk_Score < 7.0;

[0069] Level P3 (Low Value, High Risk): Value_Score < 7.0 and Risk_Score ≥ 7.0;

[0070] Level P4 (Low Value, Low Risk): Value_Score < 7.0 and Risk_Score < 7.0.

[0071] Step S2: Construct decision arms based on the sample priority list and the preset multi-level review depth combination, use the multi-armed slot machine algorithm to dynamically calculate the unit budget quality benefits of each decision arm, perform the exploration-exploitation balance decision under the total budget constraint and generate a budget allocation scheme.

[0072] Specifically, such as Figure 2 As shown, the process of step S2 includes:

[0073] Step S21: Combine different priorities with multi-level review depths to construct multiple decision arms and initialize the cost parameters of each arm.

[0074] Specifically, the Cartesian product cross combination is used to construct 12 decision arms:

[0075] Decision arm identifier format: Arm_{Priority}_{Depth}, for example, Arm_P1_D1, Arm_P2_D3. The data structure for each decision arm includes: arm identifier, priority type, review depth, unit sample cost, historical labeling accuracy, cumulative allocated sample size, cumulative budget consumption, and unit budget quality benefit. Basic cost parameters are read from the system configuration table (default D1=1.0 base unit): D1 depth cost coefficient = 1.0 (single labeling + AI validation); D2 depth cost coefficient = 2.2 (adding cross-labeling and review); D3 depth cost coefficient = 3.8 (adding multi-model collaborative validation).

[0076] Step S22: Based on the historical labeling accuracy and real-time feedback data of each decision arm, dynamically calculate the budget benefit index of each decision ratio;

[0077] Specifically, calculate the unit budget efficiency:

[0078] Budget_Efficiency=(Current_Accuracy-Base_Accuracy) / Arm_Unit_Cost

[0079] Wherein, Current_Accuracy represents the actual labeling accuracy of this decision arm in the previous cycle (using the system default value when there is no initial data: P1=0.95, P2=0.90, P3=0.85, P4=0.80); Base_Accuracy represents the system baseline accuracy, which is fixed at 0.85. Arm_Unit_Cost represents the unit sample cost of the decision arm, which is the comprehensive cost of a single sample under the corresponding priority and review depth of this arm.

[0080] Step S23: Under the total budget constraint, allocate the current budget proportionally to the fully tested arm with the highest current revenue and the potentially high-return arm without sufficient testing, and execute resource allocation decisions to obtain the budget allocation scheme.

[0081] Specifically, the process of making resource allocation decisions to obtain the budget allocation scheme includes:

[0082] The first proportion of the current budget is allocated to the fully tested arm with the highest current budget revenue indicator. The fully tested arm is the decision arm whose historical allocation sample size has reached a preset threshold.

[0083] Specifically, each decision arm maintains a sample counter, and is marked as fully tested when it has accumulated ≥50 samples. For fully tested arms, the actual labeling accuracy is obtained from step S4, and the budgeted revenue metric is calculated as (actual accuracy - 0.85 baseline) / unit cost. 70% of the available budget for the current period is allocated to the top 3 arms with the highest revenue according to their revenue weight.

[0084] The second proportion of the current budget is allocated to the under-tested potential high-yield arms, and the allocation probability of the under-tested arms is determined based on the reciprocal of the square root of their allocated sample size;

[0085] Specifically, the remaining 30% of the budget is allocated to arms that have not been adequately tested, with an allocation probability of 1 / √(sample count). The smaller the sample size, the higher the exploration probability. New arms receive 10% of the exploration budget by default.

[0086] The cumulative budget consumption of a single decision arm shall not exceed the maximum percentage of the total budget.

[0087] Specifically, the cumulative budget of a single decision arm shall not exceed 30% of the total budget. If it exceeds this limit, allocation shall be stopped and the overflow budget shall be redistributed to the suboptimal arm.

[0088] Step S3: Based on the budget allocation scheme, automatically label and verify the sample set according to the review depth level corresponding to the decision arm to generate a process dataset;

[0089] Specifically, step S3 includes the following process:

[0090] Samples are distributed to the differential processing queue based on the priority label corresponding to the decision arm;

[0091] Specifically, the system pre-configures four priority queues (P1-P4), each further divided into sub-queues based on review depth (D1-D3), resulting in a total of 12 processing sub-queues. Queue resource quotas are bound to the budget allocated in step S2. Each decision arm in the budget allocation scheme is traversed, the `allocated_samples` count is read, and a corresponding number of samples are randomly selected from the sample pool to be labeled (ensuring sample priority matches arm identifiers), and pushed to the corresponding sub-queue. The distribution process employs batch asynchronous writing, with 100 samples per batch and a write latency of < 10ms.

[0092] Priority scheduling: High-priority queues (P1 / P2) are allocated more computing resources, and their thread pool size is 3 times that of low-priority queues (P3 / P4) to ensure that high-value samples are processed first.

[0093] A multi-level verification pipeline is executed based on the differential processing queue to obtain a confidence score;

[0094] Specifically, the multi-level verification pipeline sets different verification rounds and model combinations based on the review depth level corresponding to the decision arm, and outputs a confidence score for each verification round.

[0095] Specifically, each sub-queue starts its verification pipeline independently, and the number of verification rounds and model combinations in the pipeline are automatically determined by the review depth of the decision arm:

[0096] D1 Production Line (Basic Review):

[0097] Round 1: Call the basic annotation model (such as ResNet-50 or BERT-base) to generate initial labels and confidence scores conf_score_1.

[0098] Round 2: Call the AI-assisted verification model (lightweight adversarial network) to perform secondary prediction on the initial label, and output the verification label and confidence score conf_score_2.

[0099] Confidence score aggregation: If the difference between conf_score_1 and conf_score_2 is less than 10%, the final confidence score is the average of the two; otherwise, it is marked as low_confidence and enters the exception handling process.

[0100] D2 Production Line (Enhanced Review):

[0101] Rounds 1-2: Same as D1 production line.

[0102] Round 3: For 30% of the samples (randomly selected), call the cross-validation model (heterogeneous models with different architectures, such as ResNet-101 versus ViT), and output the third label cross_label and confidence score conf_score_3.

[0103] Round 4: The advanced review model (a weighted voting system integrating three expert models) is invoked to adjudicate inconsistencies in the samples from the first three rounds, outputting the final confidence score `final_conf_score`. The inconsistency criterion is: any two rounds have different labels or a confidence score difference > 15%.

[0104] D3 Production Line (In-Depth Review):

[0105] Rounds 1-4: Same as the D2 pipeline, but cross-validation coverage is increased to 100% of the samples.

[0106] Round 5: Call the multi-model collaborative verification engine (at least 5 heterogeneous models in parallel inference), use Bayesian voting fusion, and output the fusion confidence score fusion_conf_score.

[0107] Round 6: Invoke the anomaly detection model (a One-Class SVM trained on a historical error case library), scan high-risk annotations (such as samples with a fusion confidence level between 85% and 95%), and output anomaly score (anomaly_score). Samples with anomaly_score > 0.5 trigger forced correction.

[0108] The confidence score output for each round of validation is a standardized value of 0-1, and the processing time is automatically recorded by the model performance monitoring module.

[0109] When the confidence score is lower than the preset score threshold, an anomaly handling process is triggered to identify and correct correctable errors in order to obtain a corrected result.

[0110] Specifically, the system presets a scoring threshold of confidence_threshold = 0.85 (configurable). When the final confidence score of a sample is less than 0.85, the exception handling process is automatically triggered.

[0111] Correctable error identification: The anomaly handling engine reads the verification log of the sample. If the error type is a correctable type, such as label confusion (e.g., cat / dog confusion in object recognition) or bounding box offset (IoU < 0.7), the automatic correction model (a self-correcting network based on reinforcement learning) is invoked to generate a correction result. The correction result outputs a new label and a new confidence score `revised_conf_score`, which must be ≥ 0.9 to pass. If the error type is an uncorrectable type, such as data corruption (image cannot be decoded) or label definition conflict (multiple labels mutually exclusive), the sample is marked as invalid, removed from the sample set, and step S2 is notified to return the budget.

[0112] Receive the correction results and cost data of each level of verification, record the annotation results, confidence scores, processing time and actual cost, and generate the process dataset.

[0113] Step S4: Collect quality indicators and cost indicators of each decision arm according to the process dataset and budget cycle. Compare the quality indicators with the preset quality threshold and the cost indicators with the budget threshold to obtain feedback signals. Output the feedback signals to step S2 to adjust the budget allocation strategy for the next cycle.

[0114] Specifically, such as Figure 3 As shown, the process of step S4 includes:

[0115] Step S41: Extract the quality indicators and cost indicators of each decision arm from the dataset according to the budget cycle. The quality indicators include labeling accuracy and label consistency, and the cost indicators include unit actual cost and budget overrun rate.

[0116] Specifically, the system automatically triggers the metric extraction process at the start of each budget cycle. The budget cycle is set to 1 hour by default and is started on time by a timer. The metric extraction engine reads all sample records from the process dataset generated in step S3 and performs grouping and aggregation calculations according to the decision arm dimension. For each decision arm, the system independently calculates the labeling accuracy and label consistency in the quality metrics. The labeling accuracy is obtained by dividing the number of samples in the arm that meet the confidence standard and are not marked as anomalies by the total number of labeled samples in the arm. Label consistency is obtained by comparing the consistency of the output labels in each round of multi-round validation and calculating the proportion of samples with the same label. Regarding cost metrics, the unit actual cost is calculated by dividing the sum of the actual costs of all samples in the arm by the number of valid samples, where invalid samples are not included in the calculation. The budget overrun rate is obtained by dividing the difference between the cumulative actual budget consumed and the cumulative allocated budget of the arm by the cumulative allocated budget, with the result retained to two decimal places.

[0117] Step S42: Normalize the labeling accuracy and label consistency of each decision arm, then sum and average them to obtain the overall quality score; calculate the budget overrun rate and invalid sample cost loss rate based on the cost indicators.

[0118] Specifically, the system reads the actual cost field values ​​of all samples within the current period from the database and sums them to obtain the cumulative actual consumption budget. Then, it reads the budget amount planned for the arm in the current period from the budget allocation record table, i.e., the cumulative allocated budget. The difference between the two is divided by the cumulative allocated budget, with the result rounded to two decimal places. A negative result indicates a budget surplus. To avoid invalid samples affecting calculation accuracy, the system automatically deducts the budget refund amount corresponding to identified invalid samples, ensuring that the overspending rate reflects true effective consumption. Invalid samples refer to samples marked as having uncorrectable errors and needing to be removed from the sample set during the annotation and verification process. During calculation, the system traverses the process dataset, filters out sample records marked as invalid, extracts the actual cost values ​​of these samples, and sums them to obtain the total cost of invalid samples. The total cost of invalid samples is divided by the cumulative allocated budget of the decision arm to obtain the invalid sample cost loss rate. This indicator directly reflects the proportion of budget waste caused by data quality issues. If the loss rate consistently exceeds 5%, the system will trigger a data source quality warning.

[0119] Specifically, the process of normalizing the labeling accuracy and label consistency of each decision arm and then summing and averaging them to obtain the overall quality score includes:

[0120] The labeling accuracy of each decision arm is standardized and mapped to a unified dimension range, and the label consistency index is processed into percentages.

[0121] Specifically, the system calculates the historical labeling accuracy mean and standard deviation for the arm over the past 10 budget periods. Then, it compares the current period's accuracy with the historical mean to determine the degree of deviation, and normalizes this deviation using the standard deviation. The normalized value is mapped to a uniform dimension range of 0 to 1 using a non-linear function, ensuring that decision arms of different difficulty levels are comparable on the same scale. The label consistency index itself is already a quantified value within the range of 0 to 1; the system directly multiplies it by 100 to convert it to a percentage format.

[0122] The standardized labeling accuracy rate and the percentage label consistency are weighted and summed according to a preset ratio coefficient to obtain the arm-level quality score of each decision arm.

[0123] Specifically, the arm-level quality score for each decision arm is synthesized using a preset fixed weighting ratio. The system is configured with a 60% weighting for labeling accuracy and a 40% weighting for label consistency; the two are weighted and summed according to this ratio to obtain the arm-level quality score. For newly created decision arms lacking historical data, the global mean of all arms is used as the default value for historical mean and standard deviation to ensure that the calculation can proceed normally.

[0124] The overall quality score is obtained by taking the arithmetic mean of the arm-level quality scores of all decision arms.

[0125] Specifically, the arm-level quality scores of all 12 decision arms are summed, divided by 12, and the arithmetic mean is taken to obtain the comprehensive quality score that reflects the overall annotation level of the system.

[0126] Step S43: Compare the overall quality score with the preset quality threshold to generate a quality warning signal, compare the budget overrun rate with the budget threshold to generate a cost warning signal, and compare the unit budget quality revenue change rate of adjacent periods with the preset proportion threshold to generate a strategy optimization signal.

[0127] Specifically, the system is configured with three types of threshold parameters to trigger different feedback signals. These parameters support online hot updates without requiring a service restart. The quality warning signal is triggered when the overall quality score falls below a preset threshold of 0.85, or when the quality score of any decision arm falls below 0.75. The generated signal includes a type identifier, the trigger reason, and a recommended adjustment action, typically increasing the review depth. The cost warning signal is triggered when the budget overrun exceeds 10%. When a decision arm's budget overrun exceeds this limit, a cost warning signal is generated, with the recommended action being to reduce sample allocation or decrease review depth. The strategy optimization signal is generated based on the rate of change in unit budget quality returns between adjacent periods. The system records the return indicators for each decision arm in the previous period and calculates the change in the current period compared to the previous period. If the decrease exceeds a preset 20% threshold, a strategy optimization signal is generated, with the recommended action being to increase exploration allocations to other decision arms.

[0128] Step S44: Combine the quality warning signal, cost warning signal and strategy optimization signal to form a feedback signal, and output it to step S2 to drive the adjustment of the budget allocation strategy in the next cycle.

[0129] Specifically, the signal aggregation engine packages the generated signals into standardized feedback signal packages, including period identifiers, signal lists, detailed indicators for each decision arm, and overall evaluation data. These feedback signal packages are synchronously sent to the budget allocation service in step S2 via a high-performance remote call interface, ensuring that step S2 receives the latest feedback information before initiating the next period's allocation. Simultaneously, the signal packages are written to the search engine for real-time display on the monitoring dashboard. The latency of the entire output process is controlled within 100 milliseconds, guaranteeing the real-time nature of the system's closed-loop response.

[0130] Step S5: Configure a differentiated parameter library based on the tag type. The parameter library includes tagging difficulty levels divided according to historical error rates, unit cost coefficients determined according to processing complexity, and quality weight coefficients set according to business importance. Synchronize the differentiated parameter library to steps S1, S2, and S3 to adjust the value score calculation, decision arm cost estimation, and review depth benchmark.

[0131] Specifically, step S5 includes the following process:

[0132] Calculate the error rate distribution, processing time distribution, and model impact distribution for each label type based on historical annotation data;

[0133] Specifically, the process of calculating the error rate distribution, processing time distribution, and model influence distribution for each label type based on historical annotation data includes:

[0134] A sliding time window mechanism was used to collect historical annotation data for the most recent N budget periods, and the mean and standard deviation of the dynamic error rate for each label type were calculated.

[0135] Specifically, upon system startup, records from the historical annotation database are automatically loaded, and a sliding time window mechanism is used for dynamic statistical analysis. The window size N is set to 10 budget periods (approximately 10 hours) by default. As the window progresses, data from the oldest period is automatically discarded and data from the latest period is included, ensuring that the statistical results reflect recent changes in annotation capabilities. For samples of each label type, the number of erroneous labeled samples in all annotation records within the window period is extracted and divided by the total number of labeled samples for that label to obtain the dynamic error rate for each period. The arithmetic mean of these period error rates is calculated as the mean dynamic error rate, and its standard deviation is calculated to measure the fluctuation range. For label types with fewer than 100 annotations, the system uses aggregated data of labels of the same difficulty level to supplement the calculation, avoiding data sparsity issues.

[0136] Outlier detection and cleaning are performed on the processing time data, and the corrected average processing time per unit sample is calculated as the benchmark for the processing time distribution.

[0137] Specifically, the processing time field for each sample is extracted from the process dataset, and outlier cleaning is performed first. The system uses the interquartile range (IQR) detection method to calculate the upper and lower quartiles of the processing time for all samples. Samples exceeding 1.5 times the upper quartile are identified as outliers and removed. The arithmetic mean of the cleaned data is calculated to obtain the corrected average processing time per unit sample, which serves as the baseline for the processing time distribution of this label type.

[0138] Based on the performance decay curve of the pre-trained model on the validation set, the sensitivity of each label type to the model accuracy is quantified, and the model influence distribution is generated.

[0139] Specifically, the impact of each label type on the accuracy of the pre-trained model is tested on an independent validation set. The process involves progressively reducing the label quality (e.g., injecting 5%, 10%, and 15% errors) for all samples of a specific label in the validation set, and observing the overall accuracy decay curve. A steeper curve indicates greater sensitivity of the label to the model. The system calculates the accuracy decay slope for each label type and normalizes it to the 0-1 range, generating a model influence distribution.

[0140] The labeling difficulty level is divided according to the error rate distribution, the unit cost coefficient is determined according to the processing time distribution, the quality weight coefficient is set according to the model influence distribution and business objectives, and a differentiated parameter library is integrated and constructed.

[0141] Specifically, the system automatically categorizes users into three levels based on the average dynamic error rate. An average error rate below 5% is classified as low difficulty, 5% to 15% as medium difficulty, and above 15% as high difficulty. Level thresholds can be adjusted online, taking effect the following day. Based on the processing time distribution benchmark, the tag type with the lowest processing time is set as the baseline of 1.0, with other tags mapped proportionally. For example, a tag with an average processing time of 60 seconds has a unit cost coefficient of 6.0 compared to a 10-second baseline. The coefficient range is limited to 0.5 to 10.0; exceeding this range triggers a manual confirmation mechanism. The system also combines model impact distribution with business objective weights. Core business tags (such as fraud tags in financial risk control) have a weight coefficient of 1.0, secondary tags 0.6, and auxiliary tags 0.3. The system automatically recalculates model impact monthly and suggests weight adjustments, which are updated after review. All parameters are integrated into a structured configuration file containing fields such as tag code, difficulty level, cost coefficient, quality weight, and last update time. The configuration file is stored in a distributed configuration center in YAML format, supporting version rollback and canary releases.

[0142] The differentiated parameter library is pushed to step S1 to adjust the feature weights in the value score calculation, pushed to step S2 to correct the cost estimate and budget limit of the decision arm, and pushed to step S3 to dynamically set the review depth benchmark for each tag.

[0143] Specifically, the configuration center listens for parameter change events. When it detects an update in the difficulty level or weight coefficient, it actively pushes the update to the feature weight calculation module in step S1. After receiving the update, step S1 recalculates the value score using the new weights in the next round of sample evaluation, thus achieving hot updating of feature weights.

[0144] After the unit cost coefficient is updated, the configuration center broadcasts it to the arm cost initialization module in step S2 via a message queue. When constructing a new decision arm or resetting a cycle, step S2 reads the latest cost coefficient to correct the unit sample cost, ensuring the accuracy of budget allocation.

[0145] After reviewing the updated baseline parameters, the configuration center synchronizes them to the pipeline configuration module in step S3. Step S3 dynamically adjusts the default validation rounds and model combinations for each label based on the new parameters. For example, high-difficulty labels are automatically upgraded to the D2 baseline without manual intervention.

[0146] A push-pull hybrid approach is adopted. Configuration changes are actively pushed out, while each step is passively pulled every 10 minutes to ensure consistency. If a push fails, three retries are triggered. If a retry fails, an alarm log is recorded and cached parameters are used as a fallback to ensure system availability.

[0147] Specifically, the method further includes the following steps:

[0148] When an invalid sample is identified in step S3, the budget consumed by the invalid sample is deducted from the current budget cycle and returned to the total budget pool.

[0149] Specifically, the criteria for determining invalid samples include: data corruption preventing decoding, logical conflicts in label definitions, mutually exclusive and irreconcilable validation results from multiple models, and sample ambiguity exceeding a correctable threshold. When a sample is marked as invalid, the system immediately extracts the actual cost field value of that sample and adds it to the budget pool to be returned. The amount corresponding to the invalid sample is deducted from the current decision arm's consumed budget, updating the arm's cumulative consumed budget value, while maintaining an independent return budget pool in memory. The return budget pool is stored using a key-value pair structure, with the key being the decision arm identifier and the value being a list of return amounts, ensuring traceability of the return funds' source. After each batch of return operations is completed, the return engine sends a return notification to step S2 via a message queue, containing the return amount, source arm identifier, and priority information. The notification uses at least-once delivery semantics to ensure that step S2 does not lose return information.

[0150] When performing the next cycle budget allocation in step S2, the returned budget is preferentially allocated to the valid sample queues of the same priority, and the redistribution ratio of the returned budget is marked in the budget allocation scheme;

[0151] Specifically, in step S2, when initiating a new cycle budget allocation, the status of the return budget pool is first queried. If there are unallocated return budgets, their reallocation is prioritized. The budget allocation engine merges the return budgets with the newly added budgets for the current period to form a total available budget, with the return budget portion marked as "return funds". Return budgets are preferentially allocated to valid sample queues with the same priority as the source arm. The system checks the latest sample priority list generated in step S1, filters out the subset of valid samples that are not yet labeled under that priority, and forms the target allocation queue. If there are not enough samples in the queue of the same priority, the return budget automatically overflows to the adjacent priority queue, in the order of P1→P2→P3→P4, ensuring that the return funds are not idle.

[0152] The redistribution ratio of the returned budget is dynamically adjusted based on the historical unit budget quality benefits of valid samples of the same priority. Sample queues with higher benefits receive a higher proportion of returned budget.

[0153] Specifically, the redistribution ratio is determined by the historical unit budget quality gains of each valid sample queue within the priority. The historical gain data for each queue is periodically calculated and cached in step S4. The gain is calculated as: the average quality score of the queue over the past 5 periods divided by the average unit sample cost.

[0154] The dynamic adjustment algorithm process is as follows: First, all valid sample queues within the target priority are sorted from highest to lowest based on their historical returns. The return percentage of each queue is calculated, which is the queue's return value divided by the total return value within the priority. Then, the return budget is allocated according to the return percentage, with the queue with the highest return receiving the maximum return percentage.

[0155] Specifically, to prevent the Matthew effect, the algorithm sets a minimum guarantee ratio: even if a queue has the lowest return, it can still receive at least 10% of the return budget to ensure the basic resource supply for low-yield queues. At the same time, the return budget cap for a single queue is set at 50% of the total return amount to avoid excessive resource concentration. The redistribution ratio is recalculated every cycle to ensure dynamic adaptation to changes in labeling capabilities.

[0156] Specifically, this invention quantifies objective characteristics such as sample scarcity and ambiguity into value and risk scores, providing a comparable decision-making benchmark for resource allocation. Based on these scores, a decision arm is constructed, and a multi-armed slot machine algorithm is used to dynamically balance exploration and utilization, ensuring the budget naturally flows towards high-cost-performance combinations. Differentiated verification is performed based on the depth of review allocated; higher depth levels involve more verification rounds and more complex model combinations, with the resulting confidence score directly reflecting annotation quality. Quality indicators such as accuracy and consistency are periodically compared with cost indicators. When quality falls below a threshold, a depth upgrade signal is triggered; when costs exceed the limit, a reduction signal is triggered; and when returns decrease, an exploration signal is triggered. These signals drive strategy adjustments. Historical error rates, processing time, and model impact are statistically analyzed using a sliding window to dynamically classify difficulty levels and determine cost coefficients, ensuring the parameter library is updated as annotation capabilities change and synchronized with previous stages to adjust evaluation weights and cost benchmarks. An invalid sample budget return mechanism ensures that wasted resources are automatically returned to the total budget pool and prioritized for allocation to high-return queues of the same priority in the next cycle, forming a virtuous cycle of resource reuse. The entire system drives budget allocation through quality-cost feedback signals, guides verification depth through cost-benefit relationships, and optimizes parameter configuration through historical data distribution. The parameters of each module are mutually restrictive and mutually reinforcing, ultimately achieving a dynamic balance between annotation quality, resource efficiency, and cost control.

[0157] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0158] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for collaboratively optimizing data annotation quality and cost in multi-label scenarios, characterized in that, include: Step S1: Obtain the sample set to be detected, extract the key features of the sample set to be detected, calculate the value score and risk score, and then obtain the sample priority list; Step S2: Construct decision arms based on the sample priority list and the preset multi-level review depth combination, use the multi-armed slot machine algorithm to dynamically calculate the unit budget quality benefits of each decision arm, perform the exploration-exploitation balance decision under the total budget constraint and generate a budget allocation scheme. Step S3: Based on the budget allocation scheme, automatically label and verify the sample set according to the review depth level corresponding to the decision arm to generate a process dataset; Step S4: Collect quality indicators and cost indicators of each decision arm according to the process dataset and budget cycle. Compare the quality indicators with the preset quality threshold and the cost indicators with the budget threshold to obtain feedback signals. Output the feedback signals to step S2 to adjust the budget allocation strategy for the next cycle. Step S5: Configure a differentiated parameter library based on the tag type. The parameter library includes tagging difficulty levels divided according to historical error rates, unit cost coefficients determined according to processing complexity, and quality weight coefficients set according to business importance. Synchronize the differentiated parameter library to steps S1, S2, and S3 to adjust the value score calculation, decision arm cost estimation, and review depth benchmark.

2. The method for collaborative optimization of data annotation quality and cost in multi-label scenarios according to claim 1, characterized in that, The process of step S2 includes: By cross-combining different priorities and multi-level review depths, multiple decision arms are constructed and the cost parameters of each arm are initialized; Based on the historical labeling accuracy and real-time feedback data of each decision arm, the budget benefit index of each decision ratio is dynamically calculated. Under the total budget constraint, the current budget is allocated proportionally to the fully tested arm with the highest current revenue and the potentially high-return arm without sufficient testing, and resource allocation decisions are executed to obtain the budget allocation scheme.

3. The method for collaborative optimization of data annotation quality and cost in multi-label scenarios according to claim 2, characterized in that, The process of making resource allocation decisions to obtain the budget allocation plan includes: The first proportion of the current budget is allocated to the fully tested arm with the highest current budget revenue indicator. The fully tested arm is the decision arm whose historical allocation sample size has reached a preset threshold. The second proportion of the current budget is allocated to the under-tested potential high-yield arms, and the allocation probability of the under-tested arms is determined based on the reciprocal of the square root of their allocated sample size; The cumulative budget consumption of a single decision arm shall not exceed the maximum percentage of the total budget.

4. The method for collaborative optimization of data annotation quality and cost in multi-label scenarios according to claim 3, characterized in that, The process of step S3 includes: Samples are distributed to the differential processing queue based on the priority label corresponding to the decision arm; A multi-level verification pipeline is executed based on the differential processing queue to obtain a confidence score; When the confidence score is lower than the preset score threshold, an anomaly handling process is triggered to identify and correct correctable errors in order to obtain a corrected result. Receive the correction results and cost data of each level of verification, record the annotation results, confidence scores, processing time and actual cost, and generate the process dataset.

5. The method for collaborative optimization of data annotation quality and cost in multi-label scenarios according to claim 4, characterized in that, The multi-level verification pipeline sets different verification rounds and model combinations based on the review depth level corresponding to the decision arm, and outputs a confidence score for each verification round.

6. The method for collaborative optimization of data annotation quality and cost in a multi-label scenario according to claim 5, characterized in that, The process of step S4 includes: Extract quality and cost metrics for each decision arm from the dataset according to the budget cycle. The quality metrics include labeling accuracy and label consistency, and the cost metrics include actual unit cost and budget overrun rate. The labeling accuracy and label consistency of each decision arm are normalized and then summed and averaged to obtain the overall quality score; the budget overrun rate and invalid sample cost loss rate are calculated based on the cost indicators. The overall quality score is compared with a preset quality threshold to generate a quality warning signal; the budget overrun rate is compared with a budget threshold to generate a cost warning signal; and the unit budget quality revenue change rate in adjacent periods is compared with a preset ratio threshold to generate a strategy optimization signal. The quality warning signal, cost warning signal, and strategy optimization signal are combined to form a feedback signal, which is output to step S2 to drive the adjustment of the budget allocation strategy in the next cycle.

7. The method for collaborative optimization of data annotation quality and cost in a multi-label scenario according to claim 6, characterized in that, The process of normalizing the labeling accuracy and label consistency of each decision arm and then summing and averaging them to obtain the overall quality score includes: The labeling accuracy of each decision arm is standardized and mapped to a unified dimension range, and the label consistency index is processed into percentages. The standardized labeling accuracy rate and the percentage label consistency are weighted and summed according to a preset ratio coefficient to obtain the arm-level quality score of each decision arm. The overall quality score is obtained by taking the arithmetic mean of the arm-level quality scores of all decision arms.

8. The method for collaborative optimization of data annotation quality and cost in multi-label scenarios according to claim 7, characterized in that, The process of step S5 includes: Calculate the error rate distribution, processing time distribution, and model impact distribution for each label type based on historical annotation data; The labeling difficulty level is divided according to the error rate distribution, the unit cost coefficient is determined according to the processing time distribution, the quality weight coefficient is set according to the model influence distribution and business objectives, and a differentiated parameter library is integrated and constructed. The differentiated parameter library is pushed to step S1 to adjust the feature weights in the value score calculation, pushed to step S2 to correct the cost estimate and budget limit of the decision arm, and pushed to step S3 to dynamically set the review depth benchmark for each tag.

9. The method for collaborative optimization of data annotation quality and cost in a multi-label scenario according to claim 8, characterized in that, The process of calculating the error rate distribution, processing time distribution, and model influence distribution for each label type based on historical annotation data includes: A sliding time window mechanism was used to collect historical annotation data for the most recent N budget periods, and the mean and standard deviation of the dynamic error rate for each label type were calculated. Outlier detection and cleaning are performed on the processing time data, and the corrected average processing time per unit sample is calculated as the benchmark for the processing time distribution. Based on the performance decay curve of the pre-trained model on the validation set, the sensitivity of each label type to the model accuracy is quantified, and the model influence distribution is generated.

10. The method for collaborative optimization of data annotation quality and cost in a multi-label scenario according to claim 9, characterized in that, The method further includes the following steps: When an invalid sample is identified in step S3, the budget consumed by the invalid sample is deducted from the current budget cycle and returned to the total budget pool. When performing the next cycle budget allocation in step S2, the returned budget is preferentially allocated to the valid sample queues of the same priority, and the redistribution ratio of the returned budget is marked in the budget allocation scheme; The redistribution ratio of the returned budget is dynamically adjusted based on the historical unit budget quality benefits of valid samples of the same priority. Sample queues with higher benefits receive a higher proportion of returned budget.

Images