Data annotation quality inspection optimization system based on reinforcement learning
By introducing a risk-driven stochastic deep residual network and a hierarchical trust region policy optimization algorithm, this paper addresses the problem of insufficient efficiency and accuracy in existing data annotation quality inspection methods for complex tasks. It achieves an adaptive balance in feature extraction and policy stability, thereby improving the accuracy and resource utilization of quality inspection.
Patent Information
- Application Number
- CN202511721169.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-17
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing data annotation quality control methods lack risk-driven feature extraction and dynamic depth control mechanisms, making it impossible to balance efficiency and accuracy in complex annotation tasks. Furthermore, they lack a hierarchical trust domain optimization structure, making it difficult to maintain a balance between exploration and stability during policy updates.
We employ a risk-driven stochastic deep residual network and a hierarchical trust region policy optimization algorithm. By dynamically adjusting the network depth through a stochastic depth control signal and combining the collaborative updates of high-level and low-level policy networks, we achieve an adaptive balance in feature extraction and policy stability.
It improved the accuracy of quality inspection and the efficiency of resource utilization, reduced resource consumption, and enhanced the robustness and practical application effect of the system.
Smart Images
Figure CN121542670A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and data processing technology, and in particular to a data annotation quality inspection and optimization system based on reinforcement learning. Background Technology
[0002] In artificial intelligence and big data applications, the quality of data annotation directly affects the accuracy and stability of model training. Existing data annotation quality control methods largely rely on manual sampling or fixed rule-based judgment. These detection strategies are static and have delayed feedback, making them ill-suited to the demands of large-scale and diverse labeled data. Some methods attempt to introduce deep learning for automated quality control, but these generally employ fixed network structures and single optimization strategies, lacking the ability to respond to dynamic factors such as data risk and task complexity. This results in unstable model performance under different risk levels, leading to low quality control accuracy and resource utilization.
[0003] While existing reinforcement learning methods can achieve dynamic decision-making through policy optimization, they are mostly based on single-layer trust region constraints, which cannot distinguish between high-level policy planning and low-level action control. This can easily lead to problems such as over-updating, unstable convergence, or policy oscillation. In addition, traditional residual networks often use fixed depth and constant paths in the feature extraction stage, with a fixed ratio of shallow to deep feature fusion. They lack a mechanism to adaptively adjust the network depth according to the risk level, resulting in insufficient feature representation for high-risk samples and computational redundancy for low-risk samples.
[0004] Existing technologies generally have two shortcomings: first, they lack risk-driven feature extraction and dynamic depth control mechanisms, making it difficult to balance efficiency and accuracy in complex annotation tasks; second, they lack a hierarchical trust region optimization structure, making it difficult to maintain a balance between exploration and stability during policy updates.
[0005] Therefore, how to provide a data annotation quality inspection and optimization system based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] One objective of this invention is to propose a data labeling quality inspection optimization system based on reinforcement learning. This invention employs a risk-driven stochastic deep residual network and a hierarchical trust region policy optimization algorithm to perform dynamic quality inspection decisions and policy updates on labeled data, thereby achieving adaptive adjustment of feature extraction depth and trust region radius. It has the advantages of high quality inspection accuracy, strong convergence stability, and high resource utilization efficiency.
[0007] A data annotation quality inspection and optimization system based on reinforcement learning according to an embodiment of the present invention includes:
[0008] The data acquisition and preprocessing module is used to acquire the status data and risk data of the labeled batch, perform preprocessing, and form status vectors and risk vectors.
[0009] The random deep residual network module is used to set shallow residual paths and deep residual paths, set path selection gates, and initialize the candidate residual layer set, policy network parameters, and value network parameters.
[0010] The random depth control module is used to generate random depth control signals based on risk vectors, dynamically determine the number of activation layers in the candidate residual layer set based on the random depth control signals, perform forward propagation through shallow residual paths and deep residual paths, and output policy features and value features.
[0011] The hierarchical policy generation module constructs a high-level policy network based on policy features and generates low-level constraint parameters, constructs a low-level policy network based on policy features and low-level constraint parameters and generates actions, and constructs a value network based on value features and generates state value and value credibility.
[0012] The adaptive trust region calculation and update module is used to calculate the trust region radius update amount based on the value credibility and risk vector. Based on the trust region update amount, trust region update thresholds are set for the high-level policy network and the low-level policy network respectively. The high-level and low-level policy updates are performed and the value network parameters are updated synchronously. The process is repeated iteratively until the performance converges, and the quality inspection action sequence and audit allocation results are output.
[0013] Optionally, modules can be integrated using the following methods:
[0014] S1. Obtain the status data and risk data of the labeled batch, perform preprocessing, and form status vectors and risk vectors;
[0015] S2. Construct a random deep residual network, set shallow residual paths and deep residual paths, set path selection gates, and initialize the candidate residual layer set, policy network parameters and value network parameters.
[0016] S3. Generate random depth control signals based on risk vectors, dynamically determine the number of activation layers in the candidate residual layer set based on random depth control signals, perform forward propagation through shallow residual paths and deep residual paths, and output policy features and value features.
[0017] S4. Construct a high-level policy network and generate low-level constraint parameters. Based on policy features and low-level constraint parameters, construct a low-level policy network and generate actions. Based on value features, generate state value and value credibility.
[0018] S5. Perform quality inspection operations on labeled batches based on actions and generate reports, leveraging the advantages of reporting and state value calculation.
[0019] S6. Calculate the trust region radius update based on value credibility and risk vector, and set trust region update thresholds for the high-level policy network and the low-level policy network respectively according to the trust region radius update, forming an adaptive trust region constraint.
[0020] S7. Based on the adaptive trust region constraint, perform high-level policy update and low-level policy update, and update the value network parameters based on the advantage. Write the updated high-level policy network parameters, low-level policy network parameters and value network parameters into the dual-channel residual network and value network. Repeat S3 to S7 until the convergence condition is reached, and output the quality inspection action sequence and audit allocation results.
[0021] Optionally, S2 specifically includes:
[0022] S21. Receive the state vector, establish a random deep residual network input layer, perform linear normalization mapping, obtain the basic feature representation and cache it as the basic feature tensor;
[0023] S22. Construct shallow residual paths and register 3 to 8 residual units in sequence. Each residual unit consists of a linear transformation layer, a nonlinear activation layer, and an identity bypass of the main branch. Set the number of channels and stride to parameters consistent with the basic feature tensor. When the dimension of the input state vector is less than 256, take 3 to 5; when the dimension is greater than 256, take 6 to 8. For the identity bypass, if the dimension of the main branch is inconsistent with the input dimension, add a 1×1 mapping layer to the identity bypass.
[0024] S23. Construct deep residual paths, register residual units in sequence, the number of residual units is more than twice the number of residual units in shallow residual paths, with a maximum of 20, and set the number of channels to be 2 to 4 times the number of channels in shallow residual paths.
[0025] S24. Set up a path selection gate, establish a connection between the input end and the basic feature tensor, define the path weight vector, initialize the gate parameter vector, and establish a connection between the output end and the fusion node of the shallow residual path and the deep residual path.
[0026] S25. Initialize the policy network parameters and value network parameters, and the candidate residual layer set.
[0027] Optionally, S24 specifically includes:
[0028] S241. Calculate the mean, variance, and maximum value of the risk vector. The monotonic risk score is the weighted sum of the mean, variance, and maximum value, with weights of 0.6, 0.3, and 0.1, respectively.
[0029] S242. Normalize the monotonic risk score and use it as the deep path weight. Subtract the deep path weight from the shallow path weight to get the shallow path weight.
[0030] S243. Read the output of the shallow residual path and the output of the deep residual path, and perform gated weighted fusion, with the weights being the shallow path weight and the deep path weight, respectively.
[0031] Optionally, S3 specifically includes:
[0032] S31. Set the total number of layers in the candidate residual layer set. Multiply the risk mean by 0.5 and add 0.5 to get the value as the ratio. The product of the total number of layers and the ratio is the number of activated layers.
[0033] S32. Based on the number of activation layers, select a corresponding number of residual layers from the shallow residual paths and deep residual paths in proportion to form an activation layer set.
[0034] S33. Perform forward propagation based on the set of activation layers to obtain the fused feature tensor, policy features, and value features.
[0035] Optionally, S4 specifically includes:
[0036] S41. Receive policy features in the high-level policy network. The high-level policy network consists of an input layer, a two-level fully connected feature extraction layer, and a policy distribution output layer. The feature extraction layer uses a non-linear activation function, and the output layer uses a Gaussian distribution parameterization method to calculate the pattern-level policy distribution. Initialize the high-level policy parameter matrix and bias vector, use the trust region policy optimization algorithm to calculate the update step size and policy gradient, and output the pattern decision vector.
[0037] S42. Generate a set of constraint parameters based on the pattern decision vector of the high-level policy network. The set of constraint parameters includes action amplitude constraints, step size constraints, trust region update threshold, and action confidence boundary.
[0038] S43. Input the pattern decision vector and constraint parameter set into the low-level policy network. The low-level policy network consists of an input layer, a conditional feature fusion layer, a policy mapping layer, and an action generation layer. The conditional feature fusion layer concatenates the pattern decision vector and the fused feature tensor and performs a linear transformation to form a conditional feature tensor. The policy mapping layer adjusts the network weight distribution according to the constraint parameters given by the higher layer, performs forward propagation, and generates action vectors within the trust region limit. The action vectors include instructions for adjusting the sampling ratio, assigning tasks, determining sample priority, and allocating review resources.
[0039] S44. Generate state value from value features through linear regression and value credibility through variance estimation.
[0040] Optionally, S5 specifically includes:
[0041] S51. Perform quality inspection operations and calculate the immediate return value based on the quality inspection results. The return value is calculated with a weighted score based on the sample accuracy, number of reworks and operation time to form batch-level return data.
[0042] S52. Calculate the advantage function, which is the value of the next state multiplied by the discount factor plus the immediate reward value minus the value of the current state.
[0043] S53. Mark action samples with positive advantage as priority update samples, and mark action samples with negative advantage as suppression samples.
[0044] Optionally, S6 specifically includes:
[0045] S61. Set the initial trust region radius and calculate the trust region radius update. The update is obtained by weighted summation of stability index and volatility index. The initial value of the weighting coefficient is between 0.3 and 0.7.
[0046] S62. Update the trust region radius and limit its value range to 0.01 to 0.2; truncate the value to 0.01 if the calculated result is less than 0.01, and truncate the value to 0.2 if the result is greater than 0.2.
[0047] S63. The trust region radius is passed to the high-level policy network and the low-level policy network respectively, and the trust region threshold is calculated according to the task complexity. The thresholds are 0.9 times the trust region radius and 1.1 times the trust region radius respectively.
[0048] S64. In the high-level policy network, the upper limit of the policy update step size is calculated based on the trust domain threshold, which is 0.5 multiplied by the corresponding threshold. In the low-level policy network, the upper limit of the action update step size is calculated as 0.8 multiplied by the corresponding threshold.
[0049] S65. Complete the synchronization of high-level and low-level policy parameter updates, and store the current trust region radius, weight coefficient, and step size upper limit as the initial state for the next iteration.
[0050] Optionally, S7 specifically includes:
[0051] S71. After updating the parameters of the high-level policy network, low-level policy network, and value network, write the updated policy parameters into the random deep residual network and the value network.
[0052] S72. Perform forward calculations on the new batch of labeled data based on the updated strategy network to generate the corresponding quality inspection action sequence and audit allocation results;
[0053] S73. Determine the performance convergence condition. When the average advantage improvement rate of five consecutive iterations is less than 1%, terminate the training and output the final quality inspection optimization result.
[0054] The beneficial effects of this invention are:
[0055] (1) By introducing a risk-driven stochastic deep residual network structure, the network depth and path weights are dynamically adjusted according to the risk of the labeling task, so that the shallow path focuses on local features and the deep path captures global patterns, thereby achieving an adaptive balance in feature extraction and improving the feature expression ability and quality inspection accuracy under different risk samples.
[0056] (2) A hierarchical trust domain policy optimization mechanism is adopted, in which the high-level policy network is used for global pattern decision-making and the low-level policy network is used for specific action control. The upper and lower layers are updated collaboratively through constraint parameters, which overcomes the problems of unstable policy convergence, excessive step size or over-update in traditional single-layer reinforcement learning, and improves the stability of the training process and the reliability of policy execution.
[0057] (3) Construct an adaptive trust region radius adjustment method, which uses value credibility and risk volatility to jointly adjust the trust region range and update threshold, so that the strategy update can be dynamically balanced between exploration and stability, thereby achieving efficient convergence of the quality inspection optimization process, reducing resource consumption, and improving the overall robustness and practical application effect of the system. Attached Figure Description
[0058] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0059] Figure 1 This is a structural diagram of a data annotation quality inspection optimization system based on reinforcement learning proposed in this invention;
[0060] Figure 2 Here is a flowchart of the reinforcement learning closed-loop process for a data annotation quality inspection and optimization system based on reinforcement learning proposed in this invention.
[0061] Figure 3 This is a hierarchical trust region strategy structure diagram for a data annotation quality inspection optimization system based on reinforcement learning proposed in this invention. Detailed Implementation
[0062] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0063] refer to Figure 1-3 A data annotation quality inspection and optimization system based on reinforcement learning includes:
[0064] The data acquisition and preprocessing module is used to acquire the status data and risk data of the labeled batch, perform preprocessing, and form status vectors and risk vectors.
[0065] The random deep residual network module is used to set shallow residual paths and deep residual paths, set path selection gates, and initialize the candidate residual layer set, policy network parameters, and value network parameters.
[0066] The random depth control module is used to generate random depth control signals based on risk vectors, dynamically determine the number of activation layers in the candidate residual layer set based on the random depth control signals, perform forward propagation through shallow residual paths and deep residual paths, and output policy features and value features.
[0067] The hierarchical policy generation module constructs a high-level policy network based on policy features and generates low-level constraint parameters, constructs a low-level policy network based on policy features and low-level constraint parameters and generates actions, and constructs a value network based on value features and generates state value and value credibility.
[0068] The adaptive trust region calculation and update module is used to calculate the trust region radius update amount based on the value credibility and risk vector. Based on the trust region update amount, trust region update thresholds are set for the high-level policy network and the low-level policy network respectively. The high-level and low-level policy updates are performed and the value network parameters are updated synchronously. The process is repeated iteratively until the performance converges, and the quality inspection action sequence and audit allocation results are output.
[0069] In this embodiment, the modules are interconnected using the following method:
[0070] S1. Obtain the status data and risk data of the labeled batch, perform preprocessing, and form status vectors and risk vectors;
[0071] S2. Construct a random deep residual network, set shallow residual paths and deep residual paths, set path selection gates, and initialize the candidate residual layer set, policy network parameters and value network parameters.
[0072] S3. Generate random depth control signals based on risk vectors, dynamically determine the number of activation layers in the candidate residual layer set based on random depth control signals, perform forward propagation through shallow residual paths and deep residual paths, and output policy features and value features.
[0073] S4. Construct a high-level policy network and generate low-level constraint parameters. Based on policy features and low-level constraint parameters, construct a low-level policy network and generate actions. Based on value features, generate state value and value credibility.
[0074] S5. Perform quality inspection operations on labeled batches based on actions and generate reports, leveraging the advantages of reporting and state value calculation.
[0075] S6. Calculate the trust region radius update based on value credibility and risk vector, and set trust region update thresholds for the high-level policy network and the low-level policy network respectively according to the trust region radius update, forming an adaptive trust region constraint.
[0076] S7. Based on the adaptive trust region constraint, perform high-level policy update and low-level policy update, and update the value network parameters based on the advantage. Write the updated high-level policy network parameters, low-level policy network parameters and value network parameters into the dual-channel residual network and value network. Repeat S3 to S7 until the convergence condition is reached, and output the quality inspection action sequence and audit allocation results.
[0077] In this embodiment, S1 specifically includes:
[0078] Status data is used to characterize the current execution status of a task. Its components include: task-level features, namely total sample size, annotation completion rate, and average duration; personnel-level features, namely average workload per person, error rate, and review pass rate; system-level features, namely automatic detection accuracy and latency; and historical reward features, namely previous round reward value, cumulative advantage, and batch variance.
[0079] Risk data is used to measure uncertainty and anomaly risk within a batch. Its components include: sample risk, which is the proportion of low-confidence samples and the rate of duplicate labeling; labeling risk, which is the proportion of recent erroneous samples, personnel fluctuation coefficient, and rework frequency; and model risk, which is the prediction distribution entropy, bias, variance, and feature drift.
[0080] In this embodiment, S2 specifically includes:
[0081] S21. Receive the state vector, establish a random deep residual network input layer, perform linear normalization mapping, obtain the basic feature representation and cache it as the basic feature tensor;
[0082] S22. Construct shallow residual paths and register 3 to 8 residual units in sequence. Each residual unit consists of a linear transformation layer, a nonlinear activation layer, and an identity bypass of the main branch. Set the number of channels and stride to parameters consistent with the basic feature tensor. When the dimension of the input state vector is less than 256, take 3 to 5; when the dimension is greater than 256, take 6 to 8. For the identity bypass, if the dimension of the main branch is inconsistent with the input dimension, add a 1×1 mapping layer to the identity bypass.
[0083] S23. Construct deep residual paths, register residual units in sequence, the number of residual units is more than twice the number of residual units in shallow residual paths, with a maximum of 20, and set the number of channels to be 2 to 4 times the number of channels in shallow residual paths.
[0084] S24. Set up a path selection gate, establish a connection between the input end and the basic feature tensor, define the path weight vector, initialize the gate parameter vector, and establish a connection between the output end and the fusion node of the shallow residual path and the deep residual path.
[0085] S25. Initialize the policy network parameters and value network parameters, and the candidate residual layer set.
[0086] In this embodiment, S24 specifically includes:
[0087] S241. Calculate the mean, variance, and maximum value of the risk vector. The monotonic risk score is the weighted sum of the mean, variance, and maximum value, with weights of 0.6, 0.3, and 0.1, respectively.
[0088] S242. Normalize the monotonic risk score and use it as the deep path weight. Subtract the deep path weight from the shallow path weight to get the shallow path weight.
[0089] S243. Read the output of the shallow residual path and the output of the deep residual path, and perform gated weighted fusion, with the weights being the shallow path weight and the deep path weight, respectively.
[0090] In this embodiment, S3 specifically includes:
[0091] S31. Set the total number of layers in the candidate residual layer set. Multiply the risk mean by 0.5 and add 0.5 to get the value as the ratio. The product of the total number of layers and the ratio is the number of activated layers.
[0092] S32. Based on the number of activation layers, select a corresponding number of residual layers from the shallow residual paths and deep residual paths in proportion to form an activation layer set.
[0093] S33. Perform forward propagation based on the set of activation layers to obtain the fused feature tensor, policy features, and value features.
[0094] In this embodiment, S4 specifically includes:
[0095] S41. Receive policy features in the high-level policy network. The high-level policy network consists of an input layer, a two-level fully connected feature extraction layer, and a policy distribution output layer. The feature extraction layer uses a non-linear activation function, and the output layer uses a Gaussian distribution parameterization method to calculate the pattern-level policy distribution. Initialize the high-level policy parameter matrix and bias vector, use the trust region policy optimization algorithm to calculate the update step size and policy gradient, and output the pattern decision vector.
[0096] S42. Generate a set of constraint parameters based on the pattern decision vector of the high-level policy network. The set of constraint parameters includes action amplitude constraints, step size constraints, trust region update threshold, and action confidence boundary.
[0097] S43. Input the pattern decision vector and constraint parameter set into the low-level policy network. The low-level policy network consists of an input layer, a conditional feature fusion layer, a policy mapping layer, and an action generation layer. The conditional feature fusion layer concatenates the pattern decision vector and the fused feature tensor and performs a linear transformation to form a conditional feature tensor. The policy mapping layer adjusts the network weight distribution according to the constraint parameters given by the higher layer, performs forward propagation, and generates action vectors within the trust region limit. The action vectors include instructions for adjusting the sampling ratio, assigning tasks, determining sample priority, and allocating review resources.
[0098] S44. Generate state value from value features through linear regression and value credibility through variance estimation.
[0099] In this embodiment, S41 specifically includes:
[0100] The input layer receives the policy feature tensor. The first feature extraction layer is a linear fully connected layer with ReLU activation function. The second feature extraction layer has a similar structure to the first layer, but the output dimension is reduced to 0.75 times the input dimension. The policy distribution output layer calculates the mean vector and variance vector of the policy distribution and constructs a Gaussian distribution, which is used to sample high-level actions, i.e., the pattern decision output.
[0101] The high-level policy network uses a trust-region policy optimization algorithm for updates. The system first calculates the KL divergence between the current policy and the old policy distribution; when the KL divergence exceeds the set trust-region threshold (0.05 to 0.15), the update step size is automatically reduced to prevent excessive policy distribution shift.
[0102] The pattern decision vector output by the high-level policy network is composed of a combination of sampled actions and distribution parameters, and contains four types of policy information: task mode selection parameters, action amplitude baseline parameters, step size control parameters, and confidence boundary parameters.
[0103] In this embodiment, S42 specifically includes:
[0104] The system calculates the motion amplitude constraint parameters by multiplying the motion amplitude benchmark and the mean of the risk vector to determine the upper limit of the motion amplitude. The product value is multiplied by a coefficient of 0.2 and a constant term of 0.8 is added to obtain the maximum value. If the current maximum value is less than 1, then 1 is selected as the maximum value. The lower limit is set by subtracting the mean of the risk vector from 1 and multiplying it by a coefficient of 0.05 to obtain the minimum value.
[0105] The step size constraint parameter is dynamically determined based on the step size control parameter and the trust region radius. The product of the step size control parameter and the trust region radius is taken, and the coefficient is 0.6. A constant term of 0.3 is added as the calculation formula.
[0106] The trust region update threshold is determined based on the confidence level of the high-level pattern. It is calculated by subtracting the average standard deviation of the high-level policy distribution from 1, with a coefficient of 0.1, and adding a constant term of 0.05 as the calculation formula.
[0107] The action confidence boundary parameter is set based on the covariance relationship between the confidence boundary control parameter and the variance of the high-level output. The action confidence boundary is determined by subtracting the average standard deviation of the high-level strategy distribution from 1 and multiplying it by the confidence boundary control parameter. The coefficient is 0.4 and the constant term is 0.6 as the calculation formula.
[0108] After the above calculations, the four types of constraint factors are combined to form a set of constraint parameters, which are then passed to the input of the low-level policy network to provide parameterized boundary conditions during the action generation stage.
[0109] In this embodiment, S43 specifically includes:
[0110] Receive the fused feature tensor and the pattern decision vector, and perform a concatenation operation at the input end to form a joint input vector; perform linear transformation and normalization operations on the joint input vector to form a conditional feature tensor;
[0111] The conditional feature tensor is input into the policy mapping layer, which consists of two fully connected units. The first mapping layer uses the ReLU activation function, and the second mapping layer uses the Sigmoid activation function.
[0112] In this process, the step size constraint parameter and the action amplitude constraint parameter are used to control the gradient update range and the output action boundary, respectively.
[0113] In the action generation layer, the network samples action vectors based on the output probability distribution of the policy mapping layer, and adjusts the range of output probability density according to the confidence boundary parameters during sampling.
[0114] The value network adopts a three-layer structure, consisting of an input layer, a hidden layer, and an output layer.
[0115] The input layer receives the value feature tensor, and the input dimension is consistent with the output dimension of the random deep residual network. The input vector is linearly transformed before entering the hidden layer.
[0116] The hidden layer consists of two fully connected layers. The first layer uses the ReLU activation function, and the second layer uses the Tanh activation function.
[0117] The output layer is a linear regression unit, used to calculate state value through linear regression;
[0118] The system calculates the value credibility by simultaneously calculating the prediction variance at the output layer and using a sample sliding window to statistically analyze the prediction error variance of the most recent N rounds. The value credibility is defined based on the variance result, and the derivative of 1 plus the variance result is taken as the value credibility.
[0119] The trust region radius is updated using a weighted approach, subtracting the variance of the value confidence level, the basic trust region radius, and the risk vector from 1, with weights of 0.05, 1, and -0.03, respectively.
[0120] During the training phase, the value network parameters are updated by minimizing the temporal difference error.
[0121] In this embodiment, S5 specifically includes:
[0122] S51. Perform quality inspection operations and calculate the immediate return value based on the quality inspection results. The return value is calculated with a weighted score based on the sample accuracy, number of reworks and operation time to form batch-level return data.
[0123] S52. Calculate the advantage. The advantage function is the next state value multiplied by the discount factor plus the immediate reward value minus the current state value. In this implementation, the discount factor is 0.9.
[0124] S53. Mark action samples with positive advantage as priority update samples, and mark action samples with negative advantage as suppression samples.
[0125] In this embodiment, S6 specifically includes:
[0126] S61. Set the initial trust region radius and calculate the trust region radius update. The update is obtained by weighted summation of stability index and volatility index. The initial value of the weighting coefficient is between 0.3 and 0.7.
[0127] S62. Update the trust region radius and limit its value range to 0.01 to 0.2; truncate the value to 0.01 if the calculated result is less than 0.01, and truncate the value to 0.2 if the result is greater than 0.2.
[0128] S63. The trust region radius is passed to the high-level policy network and the low-level policy network respectively, and the trust region threshold is calculated according to the task complexity. The thresholds are 0.9 times the trust region radius and 1.1 times the trust region radius respectively.
[0129] S64. In the high-level policy network, the upper limit of the policy update step size is calculated based on the trust domain threshold, which is 0.5 multiplied by the corresponding threshold. In the low-level policy network, the upper limit of the action update step size is calculated as 0.8 multiplied by the corresponding threshold.
[0130] S65. Complete the synchronization of high-level and low-level policy parameter updates, and store the current trust region radius, weight coefficient, and step size upper limit as the initial state for the next iteration.
[0131] In this embodiment, S7 specifically includes:
[0132] S71. After updating the parameters of the high-level policy network, low-level policy network, and value network, write the updated policy parameters into the random deep residual network and the value network.
[0133] S72. Perform forward calculations on the new batch of labeled data based on the updated strategy network to generate the corresponding quality inspection action sequence and audit allocation results;
[0134] S73. Determine the performance convergence condition. When the average advantage improvement rate of five consecutive iterations is less than 1%, terminate the training and output the final quality inspection optimization result.
[0135] Example 1:
[0136] To verify the feasibility of this invention in practice, it was applied to a large-scale data annotation quality inspection optimization scenario. In this scenario, the quality inspection process includes sample sampling, error identification, rework allocation, and manual review. Traditional quality inspection systems typically rely on fixed-ratio sampling and static task allocation rules, resulting in uneven sample distribution, high rework rates, slow system learning speed, and low resource allocation efficiency. Especially when data distribution changes dynamically, traditional solutions struggle to achieve adaptive optimization, leading to bottlenecks in quality inspection accuracy and return efficiency.
[0137] In this embodiment, the data annotation quality inspection and optimization system based on reinforcement learning described in this invention is used for experimental verification. The system extracts annotation batch status data and risk data through a data acquisition and preprocessing module. The status data includes task progress, number of annotated samples, operation time, number of error records, and human involvement; the risk data includes annotation accuracy fluctuations, inconsistency of review results, rework rate, sample feature complexity, and abnormal annotation rate. All data undergoes standardized preprocessing to form state vectors and risk vectors, which are then input into the reinforcement learning framework.
[0138] In the stochastic deep residual network module, feature abstraction layers of different depths are constructed through shallow and deep paths. A stochastic depth control signal is generated using risk vectors to dynamically determine the number of activation layers. In experiments, batches with higher mean risk (>0.6) had an average of 12 activation layers, while batches with lower mean risk (<0.3) had an average of 5 activation layers, thus achieving a dynamic balance between model complexity and task stability. The policy features and value features output by the system are used as inputs to the high-level policy network and value network, respectively.
[0139] In the hierarchical policy generation module, the high-level policy network generates pattern-level decision vectors based on policy characteristics to determine task allocation strategies, such as priority sampling strategies, risk-stratified sampling ratios, and review resource allocation. The low-level policy network generates specific actions under trust region constraints, including the sampling ratio (dynamically adjusted within the range of 0.05–0.3), the task reallocation factor (0.4–1.2 times the dynamic coefficient), and the sample priority score (normalized within the range of 0–1). The value network simultaneously outputs state value and value confidence, with a confidence level ranging from 0.82 to 0.95, indicating that the system's predictions are stable in most batches.
[0140] To verify performance, the system of this invention was compared with a traditional static sampling inspection system. The tests included: quality inspection accuracy, number of rework attempts, average processing time, and audit resource utilization. The experiment underwent multiple rounds of training iterations, and the system's performance converged after the 2000th iteration. Test results show that the sampling strategy of the system of this invention dynamically adjusts with risk, resulting in improved average quality inspection accuracy, reduced number of error rework attempts, and a decrease in overall audit time.
[0141] In high-risk batches (mean risk > 0.6), the system increases the sampling rate to an average of 22.3%, effectively reducing the false negative rate. In low-risk batches, the system automatically reduces the sampling rate to 8.1%, improving resource utilization efficiency. Compared to a fixed-ratio strategy, the dynamic strategy of this invention improves the reward score by approximately 18.6%. Furthermore, the reliability variance of the value network decreases faster than the baseline system (by 45.2%), indicating that the system outperforms traditional methods in both stability and learning speed. The adaptive update mechanism of the trust region maintains policy convergence in highly volatile data, avoiding the gradient explosion problem.
[0142] From the perspective of overall system performance indicators, the proposed solution demonstrates better adaptability and convergence stability in complex annotation environments. Final quality control results show that the average sample accuracy increased to 97.2%, the rework rate decreased to 2.8%, and the average review time per batch decreased from 6.5 minutes to 4.1 minutes. Overall review resource utilization improved by approximately 23%, the variance convergence curve of the system training process became smoother, and the stability of reinforcement learning was enhanced.
[0143] Table 1: Performance Comparison of Quality Inspection Optimization Systems Based on Reinforcement Learning
[0144] Indicator Categories Traditional fixed sampling inspection system This invention is a reinforcement learning system. Performance improvement Average quality inspection accuracy 91.8% 97.2% ↑5.4% Average rework rate 7.6% 2.8% ↓63.2% Average review time (minutes / batch) 6.5 4.1 ↓36.9% Sampling rate (average) 15.0% Variable 8.1–22.3% Dynamic adjustment Audit resource utilization rate 68.4% 84.1% ↑22.9% Policy convergence iterations 2800 2000 ↓28.6% Value network variance reduction rate 0.014 0.0077 ↓45.2% Average score of the reward function 0.64 0.76 ↑18.6% Model parameter update step size average 0.11 0.085 ↓22.7%
[0145] The data in the table shows that the reinforcement learning quality inspection system of this invention demonstrates advantages in all key indicators. The average quality inspection accuracy rate increased from 91.8% to 97.2%, reflecting the model's enhanced ability to identify high-risk samples; the rework rate decreased from 7.6% to 2.8%, indicating that the system's task allocation is more reasonable; and the average review time was shortened to 4.1 minutes, proving that dynamic action generation is significantly effective in improving efficiency.
[0146] The trust region adaptive mechanism effectively suppresses over-update phenomena during multi-round training, enabling the model to converge in only about 2000 iterations, approximately 800 rounds faster than traditional systems. The improved resource utilization indicates that this invention can achieve higher output under the same computing power conditions.
[0147] In summary, this embodiment verifies the feasibility and superiority of the proposed "random deep residual feature extraction + hierarchical trust region reinforcement learning" framework in the field of data annotation and quality inspection. The system can achieve risk-driven dynamic decision-making, adaptive action generation, and efficient resource allocation, improving the accuracy, stability, and automation level of data quality inspection, and achieving the expected optimization results.
[0148] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A data labeling quality inspection optimization system based on reinforcement learning, characterized in that, The method comprises the following steps: a data acquisition and preprocessing module for acquiring state data and risk data of a labeling batch, performing preprocessing, and forming a state vector and a risk vector; a random deep residual network module for setting a shallow residual path and a deep residual path, setting a path selection gate, initializing a candidate residual layer set, policy network parameters, and value network parameters; a random depth control module for generating a random depth control signal based on the risk vector, dynamically determining the number of activated layers in the candidate residual layer set according to the random depth control signal, performing forward propagation through the shallow residual path and the deep residual path, and outputting policy features and value features; a hierarchical policy generation module for constructing a high-level policy network based on the policy features, generating low-level constraint parameters, constructing a low-level policy network based on the policy features and the low-level constraint parameters, generating actions, constructing a value network based on the value features, and generating state values and value confidence levels; an adaptive trust region calculation and update module for calculating a trust region radius update amount according to the value confidence level and the risk vector, setting a trust region update threshold for the high-level policy network and the low-level policy network according to the trust region update amount, performing high-level and low-level policy updates and synchronously updating the value network parameters, repeatedly iterating until performance convergence, and outputting a quality inspection action sequence and an audit allocation result.
2. The system of claim 1, wherein, The modules are connected through the following methods: S1, acquiring state data and risk data of a labeling batch, performing preprocessing, and forming a state vector and a risk vector; S2, constructing a random deep residual network, setting a shallow residual path and a deep residual path, setting a path selection gate, initializing a candidate residual layer set, policy network parameters, and value network parameters; S3, generating a random depth control signal based on the risk vector, dynamically determining the number of activated layers in the candidate residual layer set according to the random depth control signal, performing forward propagation through the shallow residual path and the deep residual path, and outputting policy features and value features; S4, constructing a high-level policy network based on the policy features, generating low-level constraint parameters, constructing a low-level policy network based on the policy features and the low-level constraint parameters, generating actions, constructing a value network based on the value features, and generating state values and value confidence levels; S5, performing quality inspection operations on the labeling batch based on the actions and generating returns, calculating advantages based on the returns and the state values; S6, calculating a trust region radius update amount based on the value confidence level and the risk vector, setting a trust region update threshold for the high-level policy network and the low-level policy network according to the trust region radius update amount, and forming an adaptive trust region constraint; S7, performing high-level policy updates and low-level policy updates based on the adaptive trust region constraint, updating the value network parameters based on the advantages, writing the updated high-level policy network parameters, low-level policy network parameters, and value network parameters into a double-channel residual network and a value network, and repeatedly performing S3 to S7 until a convergence condition is reached, and outputting a quality inspection action sequence and an audit allocation result.
3. The system of claim 2, wherein the system is configured to: The S2 specifically comprises: S21, receiving a state vector, establishing a random deep residual network input layer, performing linear normalization mapping, obtaining a basic feature representation, and caching it as a basic feature tensor; S22, construct a shallow residual path, register 3 to 8 residual units in order, each residual unit is composed of linear transformation layer of main branch, nonlinear activation layer and identity bypass, set the number of channels and the stride as the parameters consistent with the basic feature tensor; when the input state vector dimension is less than 256, take 3 to 5; when the dimension is greater than 256, take 6 to 8; identity bypass, if the main branch dimension is not consistent with the input dimension, add a 1x1 mapping layer in the identity bypass; S23, construct a deep residual path, register residual units in order, the number of residual units is twice the number of residual units of the shallow residual path, the highest is 20, set the number of channels to 2 to 4 times the number of channels of the shallow residual path; S24, set the path selection gate, establish the connection between the input end and the basic feature tensor, define the path weight vector, initialize the gating parameter vector, and establish the fusion node connection between the output end and the shallow residual path and the deep residual path; S25, initialize the strategy network parameter and the value network parameter, and the candidate residual layer set.
4. The system of claim 3, wherein the system is configured to: The S24 specifically comprises: S241, calculate the average value, variance and maximum value of the risk vector, the monotonic risk score is the weighted sum of the average value, variance and maximum value, and the weights are 0.6, 0.3 and 0.1 respectively; S242, perform normalization on the monotonic risk score and take it as the deep path weight, and the shallow path weight is obtained by subtracting the deep path weight; S243, read the shallow residual path output and the deep residual path output, and perform gated weighted fusion, the weights are the shallow path weight and the deep path weight respectively.
5. The system of claim 4, wherein the system is configured to: The S3 specifically comprises: S31, set the total number of layers in the candidate residual layer set, the value obtained by multiplying the average value of the risk vector by 0.5 and then adding 0.5 is taken as the proportion, and the product of the total number of layers and the proportion is taken as the activation layer number; S32, according to the activation layer number, select corresponding number of residual layers from the shallow residual path and the deep residual path in proportion to form an activation layer set; S33, perform forward propagation according to the activation layer set to obtain the fused feature tensor, the strategy feature and the value feature.
6. The system of claim 5, wherein the system is configured to: The S4 specifically comprises: S41, receive the strategy feature in the high-level strategy network, the high-level strategy network is composed of an input layer, two levels of fully connected feature extraction layers and a strategy distribution output layer, and a trust domain strategy optimization algorithm is used to calculate the update step and the strategy gradient, and an output mode decision vector is output; S42, generate a constraint parameter set according to the mode decision vector of the high-level strategy network, the constraint parameter set includes action amplitude constraint, step constraint, trust domain update threshold and action confidence boundary; S43, input the mode decision vector and the constraint parameter set to the low-level strategy network, the low-level strategy network is composed of an input layer, a conditional feature fusion layer, a strategy mapping layer and an action generation layer, the conditional feature fusion layer splices the mode decision vector and the fused feature tensor and performs linear transformation to form a conditional feature tensor, the strategy mapping layer adjusts the network weight distribution according to the constraint parameters given by the high level, performs forward propagation, and generates an action vector within the trust domain limit range; the action vector includes sampling proportion adjustment, task allocation, sample priority judgment and review resource allocation instruction; S44, generating state value through linear regression of value feature, and generating value reliability through variance estimation.
7. The system of claim 6, wherein the system is configured to: The S5 specifically includes: S51, performing quality inspection operation, calculating instant return value according to quality inspection result, the return value is calculated by weighting score according to sample accuracy, rework times and operation time consumption, forming batch-level return data; S52, calculating advantage, the advantage function is next state value multiplied by discount factor plus instant return value minus current state value; S53, marking action sample with positive advantage as priority update sample, and marking action sample with negative advantage as inhibition sample.
8. The system of claim 7, wherein the system is configured to: The S6 specifically includes: S61, setting initial trust domain radius, calculating trust domain radius update, the update is obtained by weighted sum of stability index and fluctuation index, and the initial value range of the weighting coefficient is 0.3 to 0.7; S62, updating the trust domain radius, and limiting the value range to 0.01 to 0.2; when the calculation result is less than 0.01, it is truncated to 0.01, and when it is greater than 0.2, it is truncated to 0.2; S63, respectively transmitting the trust domain radius to the high-level policy network and the low-level policy network, and respectively calculating the trust domain threshold according to the task complexity, the threshold is respectively 0.9 times the trust domain radius and 1.1 times the trust domain radius; S64, calculating the upper limit of policy update step in the high-level policy network according to the trust domain threshold, which is 0.5 times the corresponding threshold, and calculating the upper limit of action update step in the low-level policy network, which is 0.8 times the corresponding threshold; S65, completing high-level and low-level policy parameter update synchronization, storing the current trust domain radius, weight coefficient and step upper limit as the initial state of next iteration.
9. The system of claim 8, wherein the system is configured to: The S7 specifically includes: S71, after completing the parameter update of the high-level policy network, the low-level policy network and the value network, writing the updated policy parameters into the random deep residual network and the value network; S72, performing forward calculation on the new batch of labeled data according to the updated policy network, generating corresponding quality inspection action sequence and audit allocation result; S73, judging performance convergence condition, terminating training when the average advantage improvement rate of five consecutive iterations is less than 1%, and outputting the final quality inspection optimization result.