Automatic training data screening method and device, electronic equipment and storage medium

Abstract

The invention provides an automatic training data screening method and device, electronic equipment and a storage medium. The method comprises the following steps: sequentially inputting each training sample in an original training data set into a target reference model to obtain a prediction label and a confidence value corresponding to the training sample, and generating a prediction result set; performing consistency comparison on prediction labels in the prediction result set and labeling labels of the original training data set to obtain a sample set with prediction results inconsistent with original labels; matching a preset confidence interval according to the confidence value of the training sample, calling a corresponding weight, and calculating a comprehensive score by using the confidence value and the weight; and comparing the comprehensive score according to a preset first threshold value and a preset second threshold value, grading a sample set of which the prediction result is inconsistent with the original mark, and generating a screening result containing different levels of problem samples. According to the method, the screening efficiency and accuracy can be improved, labeled error samples are quickly eliminated, and the training data quality is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an automatic training data screening method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the process of training a machine learning and deep learning model, the accuracy and integrity of the training data directly determine the convergence speed and generalization ability of the model. In particular, in high data-intensive tasks such as image classification, target detection and natural language processing, quality control of a large number of labeled samples has become a basic link for improving the performance of the model.

[0003] The current common data screening methods mainly include two types: 1. Artificial review method. This method checks each data set that has been constructed by an annotator to eliminate obvious annotation errors. 2. Automatic screening method based on rules or statistical features. This method usually combines static rules such as anomaly detection, noise filtering or confidence threshold to batch process suspected problem samples.

[0004] However, the artificial review method is limited by labor costs and subjective standards, and is difficult to adapt to large-scale data sets; the rule-based automatic method lacks dynamic use of model reasoning output and its confidence, resulting in insufficient screening accuracy; both methods cannot flexibly adjust the screening strategy for different tasks and training stages, which easily causes false deletion, omission and is difficult to effectively eliminate harmful data samples for model training. SUMMARY

[0005] Therefore, the embodiments of the present application provide an automatic training data screening method and device, electronic equipment and a storage medium to solve the problems of low screening efficiency, insufficient accuracy and inability to combine model prediction and confidence to accurately eliminate problem samples in the prior art.

[0006] In a first aspect, the embodiment of the present application provides an automatic training data screening method, comprising: determining a target reference model from a candidate model set based on a preset model evaluation reference; inputting each training sample in an original training data set into the target reference model in sequence, obtaining a prediction label and a confidence value corresponding to the training sample, and generating a prediction result set; performing consistency comparison between the prediction label in the prediction result set and a labeled label of the original training data set, to obtain a sample set with inconsistent prediction result and original label; for each training sample in the sample set with inconsistent prediction result and original label, matching a preset confidence interval according to the confidence value of the training sample and calling a corresponding weight, and calculating a comprehensive score by using the confidence value and the weight; comparing the comprehensive score according to preset first and second thresholds, grading the sample set with inconsistent prediction result and original label, and generating a screening result containing problem samples of different levels; and outputting the training data set corresponding to the problem samples in the screening result to a subsequent data processing module for calling by a subsequent model training process.

[0007] In a second aspect, the embodiment of the present application provides an automatic training data screening device, comprising: a determination module configured to determine a target reference model from a candidate model set based on a preset model evaluation reference; a generation module configured to input each training sample in an original training data set into the target reference model in sequence, obtain a prediction label and a confidence value corresponding to the training sample, and generate a prediction result set; a comparison module configured to perform consistency comparison between the prediction label in the prediction result set and a labeled label of the original training data set, to obtain a sample set with inconsistent prediction result and original label; a matching module configured to, for each training sample in the sample set with inconsistent prediction result and original label, match a preset confidence interval according to the confidence value of the training sample and call a corresponding weight, and calculate a comprehensive score by using the confidence value and the weight; a grading module configured to compare the comprehensive score according to preset first and second thresholds, grade the sample set with inconsistent prediction result and original label, and generate a screening result containing problem samples of different levels; and an output module configured to output the training data set corresponding to the problem samples in the screening result to a subsequent data processing module for calling by a subsequent model training process.

[0008] In a third aspect, the embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the above method.

[0010] The at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0011] The target reference model is determined from the candidate model set based on a preset model evaluation reference; each training sample in the original training data set is sequentially input into the target reference model to obtain a prediction label and a confidence value corresponding to the training sample, and a prediction result set is generated; the prediction label in the prediction result set is compared with the labeled label of the original training data set for consistency to obtain a sample set in which the prediction result is inconsistent with the original label; for each training sample in the sample set in which the prediction result is inconsistent with the original label, the confidence value of the training sample is matched with a preset confidence interval and a corresponding weight is called to calculate a comprehensive score by using the confidence value and the weight; the comprehensive score is compared according to preset first and second threshold values, the sample set in which the prediction result is inconsistent with the original label is graded, and a screening result containing problem samples of different levels is generated; the training data set corresponding to the problem samples in the screening result is output to a subsequent data processing module for calling by a subsequent model training process. The present application can improve screening efficiency and accuracy, quickly eliminate labeled error samples, and optimize training data quality. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 is a flowchart of an automatic training data screening method provided by the embodiments of the present application;

[0014] Figure 2 is a structural schematic diagram of an automatic training data screening device provided by the embodiments of the present application;

[0015] Figure 3 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0017] In the machine learning training process, the quality of training data plays a crucial role in the final performance of the model. Low-quality or incorrectly labeled data can interfere with the model's learning process, leading to overfitting, poor generalization, and other problems, affecting the accuracy and reliability of the model in practical applications. Currently, manual screening of training data has low efficiency, strong subjectivity, and is prone to omissions, while existing automated screening methods often fail to fully consider the model's prediction and the credibility of the prediction, resulting in inaccurate data selection and ineffective removal of harmful data for model training. Therefore, there is an urgent need for a more scientific, efficient, and accurate automated training problem data screening method.

[0018] In view of the problems existing in the prior art, the present application provides an automated training data screening method based on prediction-labeling difference. The present application relates to the field of machine learning and deep learning, and specifically relates to an automated training problem data screening method for improving the quality of training data, which realizes efficient screening of data with inconsistent labeling and prediction problems in training data by combining model inference, confidence evaluation, and a two-level screening mechanism. The main contents of the technical solution of the present application are described in detail as follows:

[0019] Model inference: use the currently optimal model to perform a round of inference on all training data. During the inference process, the model predicts each training data sample and generates the corresponding prediction results.

[0020] Primary screening layer: model prediction-labeling inconsistency identification.

[0021] The core principle of the primary screening layer is to compare the model prediction results with the original labeling data directly. The best trained model is used to infer the training data, and the model will generate prediction results for each training sample based on the knowledge and patterns it has learned. Since this model is currently the best performer, its prediction results can reflect the true characteristics of the data to some extent, so comparing it with the original labeling can preliminarily locate the data samples that may have problems.

[0022] The specific implementation process of the automated training problem data screening method of the present application is as follows:

[0023] Data input and inference: input all training data into the selected optimal model one by one, and the model analyzes and calculates each data sample to output the corresponding prediction results. For example, in an image classification task, the model will output the predicted class label for each image; in a target detection task, the model will output the class, position, and other information of the detected target.

[0024] Result comparison: Compare the model's prediction results with the original labeled data one by one. For example, if the model predicts that a certain image is a "cat", but the original label is "dog", then the model's prediction and the label are inconsistent.

[0025] Primary labeling: Once inconsistencies between model predictions and labels are found, the corresponding training data samples are immediately labeled as potentially problematic data. These labeled data will enter the next level of screening, the second-level decision optimization layer, for more in-depth analysis and processing.

[0026] Effects and limitations: The primary screening layer can quickly and intuitively identify obvious inconsistencies between labels and predictions, with high screening efficiency, providing a basis for further in-depth screening. However, due to the simple inconsistency comparison, there may be some misjudgments, such as the model's inaccurate prediction on some complex samples, leading to the misjudgment of correctly labeled data as problematic data, so the second-level screening is needed for further optimization.

[0027] Second-level decision optimization layer: confidence weighted analysis and sample classification

[0028] 1. Confidence weighted analysis

[0029] Confidence concept and acquisition: During model prediction, most modern machine learning and deep learning models can output the confidence of the prediction result. Confidence is a numerical value that measures the model's certainty about its own prediction. For example, in image classification, the model predicts that a certain image is a "cat" with a confidence of 0.95, indicating that the model is 95% sure that the image belongs to the "cat" category. In the second-level screening, the confidence values of the inconsistent samples marked by the primary screening layer are first obtained.

[0030] Weighted analysis method: To more reasonably evaluate these inconsistent samples, a weighted analysis method is used. Different confidence intervals correspond to different weights. Generally, high confidence (such as greater than 0.8) is given a higher weight, and low confidence (such as less than 0.2) is given a lower weight. By multiplying the sample's confidence with the set weight, and combining other related factors (such as the type and severity of inconsistency), the comprehensive score of each sample is calculated. For example, for a sample with inconsistent prediction and label and a confidence of 0.9, due to its high confidence, it is given a higher weight and the comprehensive score calculated is relatively high, indicating that the sample has a higher probability of being problematic; while for a sample with a confidence of 0.3, the comprehensive score is relatively low, indicating that the severity of the problem may be low.

[0031] 2. Sample classification

[0032] Classification basis and standard: According to the comprehensive score calculated, the inconsistent samples marked in the primary screening layer are classified. A clear score threshold is set, and samples with scores higher than a certain threshold (such as 0.7) are determined as data with more serious problems, which may have annotation errors or data abnormalities and need to be processed first, such as re-labeling or rejection. Samples with scores lower than another threshold (such as 0.3) are determined as data that may have minor problems or misjudgments, and for such data, manual review can be arranged or they can be processed as sub-priority data in subsequent training. Samples with scores in the middle interval can be further analyzed and processed according to actual conditions.

[0033] Significance of hierarchical management: Through this sample classification method, hierarchical management of problem data is achieved. Resources can be concentrated to prioritize the processing of serious problem data, improving the targeting and effectiveness of data screening, while avoiding excessive processing of a large number of possible misjudgment data. On the premise of ensuring screening accuracy, the overall data processing efficiency is improved.

[0034] 3. Synergistic effect with the primary screening layer:

[0035] The second-level decision optimization layer is a further refinement and optimization of the results of the primary screening layer. The primary screening layer provides the range of data that may have problems, and the second-level screening layer accurately identifies the problem data that needs to be focused on and processed from these data through confidence weighting analysis and sample classification, which compensates for the limitations of the primary screening layer. The two work together to achieve efficient and accurate training problem data screening.

[0036] The above technical solutions can achieve the following technical effects:

[0037] Improved screening accuracy: By combining the model's confidence evaluation and two-level screening mechanism, problems in the training data can be more comprehensively and accurately identified. Compared with traditional simple comparison or single rule screening methods, the screening accuracy is greatly improved, reducing misjudgment and omission.

[0038] Improved screening efficiency: The training problem data screening is automated, eliminating the need for extensive manual intervention, allowing for rapid processing of large-scale training data, significantly improving data screening efficiency and saving manpower and time costs.

[0039] Optimized training data quality: After accurately screening problem data, these data can be processed for correction, supplementation, or rejection, effectively improving the overall quality of training data, providing a better data foundation for model training, and helping to improve the training effect and performance of the model.

[0040] Enhance model stability: high-quality training data helps the model learn more accurate knowledge, reduces model bias caused by incorrect data, and enhances the stability and generalization ability of the model in different scenarios.

[0041] The technical solutions of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0042] Figure 1 is a flowchart of an automatic training data screening method provided by an embodiment of the present application. As shown in Figure 1 , the automatic training data screening method can specifically include:

[0043] S101, determining a target reference model from a candidate model set based on a preset model evaluation reference;

[0044] S102, inputting each training sample in the original training data set into the target reference model in turn, obtaining the prediction label and confidence value corresponding to the training sample, and generating a prediction result set;

[0045] S103, comparing the prediction label in the prediction result set with the labeled label of the original training data set for consistency, obtaining a sample set in which the prediction result and the original label are inconsistent;

[0046] S104, for each training sample in the sample set in which the prediction result and the original label are inconsistent, matching the confidence value of the training sample with a preset confidence interval and calling the corresponding weight, and calculating a comprehensive score using the confidence value and the weight;

[0047] S105, comparing the comprehensive score according to a preset first threshold and a second threshold, classifying the sample set in which the prediction result and the original label are inconsistent, and generating a screening result containing problem samples of different levels;

[0048] S106, outputting the training data set corresponding to the problem sample in the screening result to a subsequent data processing module for subsequent model training process calling.

[0049] In some embodiments, determining a target reference model from a candidate model set based on a preset model evaluation reference includes:

[0050] According to the model precision requirement and inference overhead limit of the to-be-processed task, a verification data set and an evaluation index matrix containing at least two types of evaluation indexes are preset, and the evaluation index matrix is configured with a weight coefficient for each evaluation index;

[0051] Using each candidate model in the candidate model set to perform inference on the verification data set to obtain a corresponding evaluation index value set;

[0052] The evaluation index value set of each candidate model is weighted according to the weight coefficient of the evaluation index matrix, and the corresponding aggregated evaluation score is obtained.

[0053] The aggregated evaluation scores of the candidate models are compared, and the candidate model that meets the set preferred condition and has the highest aggregated evaluation score is selected as the target reference model.

[0054] Specifically, the embodiment takes an image classification task of a certain intelligent security scene as an example to illustrate the whole process of determining the target reference model from the candidate model set based on the preset model evaluation reference. The task requires that the average single image inference time delay on the NVIDIA T4 GPU is not higher than 10 ms, while the Top-1 accuracy rate is not lower than 95%.

[0055] Firstly, the system constructs a verification dataset according to the task requirements. The development team extracts 15000 images that have been manually verified from historical production line snapshot data, and randomly divides them into an evaluation subset and a fault tolerance subset in a ratio of seven to three to ensure balanced sample distribution and cover a variety of scene lighting and poses.

[0056] Subsequently, the system configures an evaluation index matrix based on the above requirements. The matrix contains four indicators: Top-1 accuracy, weighted F1 score, single inference time delay, and peak memory occupancy. In order to balance accuracy and deployment cost, the system assigns weight coefficients of 0.45, 0.25, 0.20, and 0.10 to the four indicators, respectively, and takes the inverse of the time delay and memory occupancy to make all indicators show that the larger the value, the better.

[0057] Next, nine candidate models such as ResNet50, EfficientNet-B0, and MobileNetV3 are loaded into the inference environment. The system executes inference on the verification dataset through a unified inference service interface, and collects the original measurement values of each model on the four indicators in real time. The inference time delay takes the average of 100 consecutive single inferences, and the memory occupancy takes the peak memory reading during inference.

[0058] The system normalizes the index values of each candidate model, and then performs weighted operation according to the weight coefficients of the evaluation index matrix to calculate the aggregated evaluation score. The example result shows that ResNet50 scores 0.86, EfficientNet-B0 scores 0.81, and MobileNetV3 scores 0.78, and the scores of the remaining models are all lower than 0.75.

[0059] To ensure that the final model meets the lower limit of accuracy and the upper limit of time delay, the system re-verifies the hard constraints after calculating the score: only when the Top-1 accuracy is greater than or equal to 95% and the time delay is less than or equal to 10 ms, the candidate model is considered to meet the set preferred condition. After the comprehensive score ranking and hard constraint filtering, ResNet50 becomes the only candidate model that meets both conditions and has the highest aggregate score, and the system registers it as the target reference model and records the model identifier, evaluation date, index snapshot, and version number in the model registration center.

[0060] After the target reference model is determined, the system issues the model identifier and the validation dataset reference address to the data screening pipeline so that the model can be directly called for prediction in the subsequent training sample inference stage. At this point, the embodiment has completed the whole process of selecting the target reference model from the candidate model set.

[0061] Through the above implementation scheme, the reference model with the best comprehensive performance can be quickly locked in a quantitative manner under the premise of meeting the dual constraints of accuracy and inference overhead, providing a reliable and consistent inference reference for subsequent automated training data screening.

[0062] In some embodiments, each training sample in the original training dataset is sequentially input into the target reference model, the prediction label and the confidence value corresponding to the training sample are obtained, and a prediction result set is generated, including:

[0063] The original training dataset is divided into a plurality of data batches according to a preset batch size, and a unique sample identifier is assigned to each training sample in the data batch;

[0064] Each training sample is sequentially input into the target reference model, the prediction label and the confidence value corresponding to the training sample are obtained, and a prediction result set is generated, including:

[0065] For each training sample, the class corresponding to the maximum probability in the class probability vector is determined as the prediction label, and the probability value is extracted as the confidence value;

[0066] The prediction label and the confidence value are written into a structured storage table with the sample identifier as the index, and the prediction result set is continuously aggregated to form a one-to-one correspondence between the prediction result set and the original training dataset in the sample identifier dimension.

[0067] Specifically, in the present embodiment, the target is to complete batch inference on the original training dataset and generate a structured prediction result set. To this end, the system first constructs batch scheduling logic, injects a unique sample identifier for each training sample, then calls the target benchmark model to perform inference, and writes the result to the storage table based on the sample identifier after the inference ends. The key technical features of this implementation are: (1) a batch inference mechanism is used to reduce the memory pressure and fully utilize the GPU parallelism; (2) the maximum probability item is directly extracted as the prediction label by the category probability vector, and the confidence value is recorded synchronously, realizing label confidence binding; (3) the sample identifier is used as the primary key to write the inference result in the structured storage, ensuring the one-to-one correspondence between the prediction result and the original training data, and laying the index foundation for subsequent difference detection. The specific implementation process is as follows:

[0068] First, the original training dataset D_total is divided into several data batches according to the batch size batch_size configured by the system. In the production environment, batch_size is usually set to 256 pictures to balance throughput and memory usage. The data loader automatically assigns continuous 64-bit integer sample identifiers sample_id for internal samples when each batch is generated. The generation rule of the sample identifier is 〈timestamp + batch sequence number + batch sequence number within the batch〉, which ensures global uniqueness and incremental.

[0069] Secondly, the system initiates an inference request to the target benchmark model by batch. The inference engine inputs the batch tensor to the GPU, and returns a category probability vector matrix of batch_size×C dimensions after a single forward propagation, where C is the total number of categories. The inference engine also outputs batch-level time consumption, memory usage and other running state indicators for operation and maintenance monitoring.

[0070] Thirdly, for each returned sample probability vector, the system performs Top-1 operation to locate the category index corresponding to the maximum probability item, and maps the category index to the category label label_pred; at the same time, the maximum probability value is taken as the confidence score score_pred. The Top-1 operation is completed on the CPU side using SIMD instructions, with an average time consumption of less than 0.2ms / sample, which does not affect the overall throughput.

[0071] Subsequently, the system starts the result writing task. The result writing task uses the sample identifier sample_id as the primary key to insert (or update) the structured storage table prediction_table. Each record contains the fields: sample_id, label_pred, score_pred, model_version, batch_timestamp. In order to ensure consistency, the write operation adopts batch insertion mode and enables ACID transaction; if the insertion fails, it will automatically retry for a maximum of three times.

[0072] Finally, with the inference and writing of all data batches completed, the prediction_table is formed as the prediction result set R_pred. Since the sample_id is always used as the index when writing, the records in the prediction_table naturally maintain a one-to-one correspondence with the original training data set D_total in the sample identification dimension. Downstream modules can consult the original annotation label and model prediction result based on the same sample_id, thereby performing subsequent consistency comparison and confidence weighting analysis.

[0073] Through the above implementation, the system ensures efficient use of GPU resources while achieving accurate recording of prediction labels and confidence at the training sample level, laying a reliable data foundation for difference detection and problem sample classification.

[0074] In some embodiments, the prediction labels in the prediction result set are compared with the annotation labels of the original training data set for consistency, resulting in a sample set in which the prediction results and the original annotations are inconsistent, including:

[0075] Based on the sample identification, the prediction labels in the prediction result set are matched one by one with the annotation labels of the corresponding samples in the original training data set;

[0076] According to a preset consistency determination rule, consistency determination is performed on each training sample; when the prediction label and the annotation label are not the same, or in the target detection scenario, the intersection over union of the predicted target and the annotated target is lower than the preset threshold, the training sample is marked as an inconsistent sample;

[0077] Collecting all sample identifications, prediction labels, annotation labels, and corresponding confidence values marked as inconsistent samples, a sample set in which the prediction results and the original annotations are inconsistent is generated.

[0078] Specifically, the present embodiment describes the sample-by-sample consistency comparison process between the prediction result set and the original annotation. The core technical means of this process is: (1) using only the sample identification sample_id as an index, accurately pairing the prediction label with the annotation label; (2) configuring difference determination rules according to the task type, directly comparing the labels in classification tasks, and determining based on the intersection over union IoU threshold in detection tasks; (3) collecting necessary fields for all samples determined to be inconsistent, generating an independent inconsistent sample set for subsequent confidence weighting analysis and classification processing. The specific implementation process is as follows:

[0079] Firstly, the system loads the original annotation table annotation_table and the prediction result table prediction_table in the difference detection service, both tables are primary keys with sample_id. The difference detection service extracts the label_true and label_pred fields from the two tables through internal high-speed join operations, and synchronously reads the score_pred field.

[0080] Further, for the image classification task, the difference detection service executes a simple label comparison statement. When label_true ≠ label_pred, the system immediately marks the corresponding sample as "inconsistent". For the object detection task, the system also needs to parse the boundary box coordinates stored in the prediction_table and the annotation coordinates in the annotation_table, and calculate the IoU for the same class. When the maximum IoU value < 0.5 (preset threshold), it is determined that the prediction and annotation are inconsistent. The IoU calculation link is batch executed on the CPU end, using vectorized operations to improve throughput.

[0081] Further, after the difference detection is completed, the system writes all "inconsistent" sample sample_id, label_true, label_pred and score_pred into the structured result table mismatch_table, and adds the fields task_type and timestamp for tracing. In order to speed up the downstream processing, the mismatch_table is automatically partitioned when writing, where the image classification samples and the object detection samples are stored in different partitions to ensure query efficiency.

[0082] In some examples, taking an actual operation as an example, the validation set contains 15000 images, of which 10000 are classification task samples and 5000 are detection task samples. The difference detection service takes 23s to complete all comparisons in a 12-core CPU environment, and finally identifies 421 inconsistent classification samples and 137 inconsistent detection samples, a total of 558 records written to the mismatch_table. The table is then pulled by the confidence weighted analysis service to implement subsequent comprehensive scoring and grading processing. Through the above implementation, the system significantly improves the processing performance while ensuring the accuracy of the comparison, laying a reliable data foundation for the accurate screening of problem samples.

[0083] In some embodiments, according to the confidence value of the training sample, the preset confidence interval is matched and the corresponding weight is called, and the comprehensive score is calculated by using the confidence value and the weight, including:

[0084] A confidence interval mapping table is constructed in advance, and the confidence interval mapping table stores the corresponding relationship between a plurality of confidence intervals and target weight coefficients.

[0085] For each training sample in the sample set whose prediction result is inconsistent with the original label, obtain a confidence value corresponding to each training sample;

[0086] Determine the confidence interval to which the confidence value belongs in the confidence interval mapping table, and read the weight coefficient corresponding to the confidence interval;

[0087] According to a preset scoring formula, the confidence value and the weight coefficient are combined and operated to generate a comprehensive score of each training sample, and the comprehensive score and the sample identifier of the training sample are associated and stored.

[0088] Specifically, the embodiment describes how to calculate the comprehensive score of each sample based on the confidence interval and the weight coefficient after obtaining the sample set whose prediction result is inconsistent with the original label. The key technical features of this process are: (1) the confidence interval mapping table is predefined in the system deployment stage and is persisted in the form of key-value; (2) the confidence value score_pred of the inconsistent sample is read in batches during the running period, and the corresponding weight w is retrieved through the interval matching mechanism; (3) the comprehensive score S is generated according to the unified scoring formula, and the sample identifier sample_id is written into the scoring result table as the primary key, realizing the quantitative input required for subsequent hierarchical judgment.

[0089] First, the operation and maintenance personnel construct the confidence interval mapping table according to the business scenario. Taking the image classification task as an example, the confidence interval is divided into four segments: [0, 0.2), [0.2, 0.5), [0.5, 0.8), [0.8, 1], and the weight coefficients are configured as 0.1, 0.3, 0.7, and 1.0, respectively. The mapping table is written into the configuration service in the form of "upper bound of interval→weight", and is cached to the memory of the difference evaluation node at the same time to ensure high-speed query.

[0090] Subsequently, the system pulls the records in the mismatch_table in batches in the confidence weighting service. For each record, first read sample_id and score_pred, then perform interval matching in local memory. When score_pred=0.92, hit the interval [0.8, 1] and return the weight 1.0; when score_pred=0.45, hit the interval [0.2, 0.5) and return the weight 0.3.

[0091] Next, the system calculates the comprehensive score according to the unified formula S=score_pred×w. For example, when score_pred is 0.92, S=0.92; when score_pred is 0.45, S=0.45×0.3=0.135. If the business needs to consider the weight of abnormal types, an additional multiplier can be introduced into the formula, but in this embodiment only the single-factor linear scaling method is used.

[0092] After the comprehensive score is generated, the system immediately writes the score result table score_table with sample_id as the primary key. Each record contains the fields: sample_id, score_pred, w, S, timestamp. The write operation is submitted in batch mode at one time to reduce transaction overhead. The overall time for computing and writing for 558 inconsistent samples is about 4s.

[0093] Through the above implementation, the system uses the pre-defined interval weight to amplify or reduce the confidence value, quickly obtains the comprehensive score reflecting the severity of the sample problem, and provides a quantitative basis for the subsequent grading and classification steps.

[0094] In some embodiments, the set of samples whose prediction results are inconsistent with the original labels is graded according to the comparison of the comprehensive score with the first threshold and the second threshold, and a screening result containing samples of different levels of problems is generated, including:

[0095] Read the corresponding relationship between the comprehensive score and the sample identifier, and obtain the comprehensive score for each training sample;

[0096] Interval comparison of the comprehensive score with the first threshold and the second threshold:

[0097] When the comprehensive score is greater than or equal to the first threshold, the corresponding training sample is marked as a first-level problem sample;

[0098] When the comprehensive score is less than or equal to the second threshold, the corresponding training sample is marked as a third-level problem sample;

[0099] When the comprehensive score is between the first threshold and the second threshold, the corresponding training sample is marked as a second-level problem sample;

[0100] According to the level label of the training sample, the sample identifier, the comprehensive score and the level label are written into the preset structured storage table, and are summarized to form a screening result.

[0101] Specifically, the embodiment illustrates how to divide the samples with inconsistent prediction results and original labels into different levels of problem samples by double threshold interval comparison after obtaining the comprehensive score, and write the grading results into structured storage. The key technical features of this process are: (1) The first threshold T_high and the second threshold T_low are pre-configured in the system-level parameter center and dynamically loaded when running; (2) The comprehensive score S is batch read in the grading service memory, and interval judgment is performed in a pipeline manner to realize high throughput and low delay; (3) The result is written with sample-level label level as the partition key to build an appendable columnar storage table problem_table, ensuring that downstream processing can quickly retrieve by level. The specific implementation process is as follows:

[0102] First, the system pulls the threshold parameters from the configuration center during the service startup phase, sets the first threshold T_high = 0.7 and the second threshold T_low = 0.3. After the threshold is loaded, it is written to the read-only memory of the grading service to avoid accidental modification during runtime.

[0103] Subsequently, the grading service starts the batch processor, which pulls all 558 inconsistent sample records from score_table at once. The processor traverses in ascending order of sample_id, extracts the comprehensive score S for each record, and compares it with T_high and T_low. If S ≥ 0.7, level is set to 1; if S ≤ 0.3, level is set to 3; otherwise, level is set to 2. To improve efficiency, the comparison logic is implemented using bitwise operations, with a single record determination taking less than 5μs.

[0104] After the determination is completed, the grading service writes sample_id, S, level, and timestamp into problem_table. problem_table is divided into three partitions based on the level column when created: level = 1, level = 2, and level = 3. Each write operation is appended through a batch submission interface, without random updates, ensuring that the write delay is less than 100ms. During the write process, if network disconnection or disk fullness is detected, the system will cache the to-be-written batch in memory and perform three backoff retries to ensure result consistency.

[0105] Taking this run as an example, among the 558 samples, 197 have a comprehensive score of not less than 0.7, marked as first-level problem samples; 114 samples have a score between 0.3 and 0.7, marked as second-level problem samples; and the remaining 247 samples have a score of not higher than 0.3, marked as third-level problem samples. After all the results are written into problem_table, the grading service sends event logs containing level distribution and write time to the monitoring center, supporting real-time auditing by operation and maintenance personnel.

[0106] Through the above-mentioned embodiments, the system completes the hierarchical archiving of inconsistent samples in milliseconds, and the results are persisted in a structured form, providing an intuitive and efficient data entry for subsequent "reject / review / retain" differentiated processing.

[0107] In some embodiments, the training data set corresponding to the problem sample in the screening result is output to the subsequent data processing module for subsequent model training process calling, including:

[0108] According to the problem level in the screening result, the labeled training samples are grouped, and a first-level problem sample data set, a second-level problem sample data set, and a third-level problem sample data set are respectively generated;

[0109] For each problem sample data set, a structured record containing sample identification, original annotation label, predicted label, confidence value, and comprehensive score is written, and timestamp and version number information are appended;

[0110] Based on the preset data output interface, the problem sample data set is written into the specified storage location to generate the corresponding data location identifier;

[0111] A metadata description file containing data location identifier, problem level identifier, and version number is constructed, and the metadata description file is sent to the subsequent data processing module, so that the subsequent model training process can call the corresponding problem sample data set.

[0112] Specifically, the present embodiment describes the data output process of the screening result. The key technical features of this process are: (1) relying on the hierarchical result to split the problem sample into three independent data sets, which facilitates the downstream module to process according to different priorities; (2) when generating the data set, write complete structured fields, including sample identification, original annotation label, predicted label, confidence value, and comprehensive score, and append immutable version number and timestamp to ensure data traceability; (3) through a unified data output interface, write the data set into object storage, and simultaneously generate a metadata description file for subsequent training pipeline to retrieve and load as needed.

[0113] First, the data export service pulls all records from the problem_table, and groups them according to the level field. The system generates "first-level problem sample data set", "second-level problem sample data set", and "third-level problem sample data set" according to level = 1, level = 2, and level = 3, respectively. If there are no samples at a certain level, the system still creates an empty set to maintain interface compatibility.

[0114] Subsequently, the export service writes structured records for each question sample. The structured fields include sample_id, label_true, label_pred, score_pred, S, timestamp, and version in sequence. The version is composed of date and incremental serial number, such as v20250627_01, for distinguishing multiple iterations in the same day. The structured records are saved in columnar format, so that subsequent access can be read by field clipping.

[0115] Next, the system writes the three question sample data sets to object storage through a preset data output interface. To ensure hierarchical isolation, the system uses a hierarchical path of "bucket / task name / level / version". For example, the first-level question sample data set is written to the path "ml_bad_data / security_project / level1 / v20250627_01 / ". After writing is completed, the system returns a data location identifier for each data set, such as "uri: / / ml_bad_data / security_project / level1 / v20250627_01".

[0116] After successful writing, the export service constructs a metadata description file based on the above data location identifier, question level, and version information. The metadata description file uses a lightweight key-value pair format, and the fields include dataset_uri, problem_level, record_count, schema_hash, version, and write_time. The schema_hash is calculated by the hash value of the structured field definition, ensuring that the subsequent module can accurately verify the field consistency. The metadata file is written to the "metadata" directory of object storage and sent to the model training scheduling center through the message queue.

[0117] When the model training pipeline receives the metadata description file, it can pull the corresponding data set according to the problem_level: the first-level sample can be directly excluded from the training set or trigger the re-labeling process, the second-level sample enters the semi-automatic review link, and the third-level sample can be selectively restored before the next round of training. In this way, a closed loop of "question sample discovery-data set generation-training process processing" can be realized, ensuring seamless connection between data quality management and model training process.

[0118] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0119] Figure 2FIG. 1 is a structural schematic diagram of an automatic training data screening device provided by an embodiment of the present application. As shown in FIG. 1, the multi-agent driven multi-modal cognitive device includes: Figure 2

[0120] The determining module 201 is configured to determine a target benchmark model from a candidate model set based on a preset model evaluation benchmark.

[0121] The generating module 202 is configured to input each training sample in an original training data set into the target benchmark model in sequence, obtain a prediction label and a confidence value corresponding to the training sample, and generate a prediction result set.

[0122] The comparing module 203 is configured to perform consistency comparison between the prediction label in the prediction result set and a labeled label of the original training data set, to obtain a sample set in which the prediction result is inconsistent with the original label.

[0123] The matching module 204 is configured to, for each training sample in the sample set in which the prediction result is inconsistent with the original label, match a preset confidence interval according to the confidence value of the training sample and call a corresponding weight, and calculate a comprehensive score by using the confidence value and the weight.

[0124] The grading module 205 is configured to compare the comprehensive score according to a preset first threshold and a second threshold, grade the sample set in which the prediction result is inconsistent with the original label, and generate a screening result containing problem samples of different levels.

[0125] The output module 206 is configured to output the training data set corresponding to the problem sample in the screening result to a subsequent data processing module, for calling by a subsequent model training process.

[0126] In some embodiments, Figure 2 The determining module 201 of the automatic training data screening device is configured to, according to a model precision requirement and an inference overhead limit of a to-be-processed task, preset a verification data set and an evaluation index matrix containing at least two types of evaluation indexes, the evaluation index matrix being configured with a weight coefficient for each evaluation index; perform inference on the verification data set by using each candidate model in the candidate model set, to obtain a corresponding evaluation index value set; perform weighted operation on the evaluation index value set of each candidate model according to the weight coefficient of the evaluation index matrix, to obtain a corresponding aggregated evaluation score; compare the aggregated evaluation scores of the candidate models, and select a candidate model that meets a set optimization condition and has the highest aggregated evaluation score, as the target benchmark model.

[0127] In some embodiments, Figure 2 ​The generating module 202 divides the original training data set into a plurality of data batches according to a preset batch size, and assigns a unique sample identifier to each training sample in the data batch; calls the target reference model to perform inference batch by batch, and obtains a category probability vector corresponding to each training sample; for each training sample, determines the category corresponding to the maximum probability in the category probability vector as a predicted label, and extracts the probability value as a confidence value; takes the sample identifier as an index, writes the predicted label and the confidence value into a structured storage table, and continuously aggregates to form a prediction result set, so that the prediction result set and the original training data set maintain a one-to-one correspondence in the sample identifier dimension.

[0128] In some embodiments, Figure 2 The matching module 204 of the inconsistency determination system pre-constructs a confidence interval mapping table, which stores the correspondence between a plurality of confidence intervals and target weight coefficients; for each training sample in the sample set in which the prediction result is inconsistent with the original annotation, the confidence value corresponding to each training sample is obtained; the confidence interval to which the confidence value belongs is determined in the confidence interval mapping table, and the weight coefficient corresponding to the confidence interval is read; according to a preset scoring formula, the confidence value and the weight coefficient are combined and operated to generate a comprehensive score of each training sample, and the comprehensive score and the sample identifier of the training sample are associated and stored.

[0129] In some embodiments, Figure 2 The matching module 204 of the inconsistency determination system pre-constructs a confidence interval mapping table, which stores the correspondence between a plurality of confidence intervals and target weight coefficients; for each training sample in the sample set in which the prediction result is inconsistent with the original annotation, the confidence value corresponding to each training sample is obtained; the confidence interval to which the confidence value belongs is determined in the confidence interval mapping table, and the weight coefficient corresponding to the confidence interval is read; according to a preset scoring formula, the confidence value and the weight coefficient are combined and operated to generate a comprehensive score of each training sample, and the comprehensive score and the sample identifier of the training sample are associated and stored.

[0130] In some embodiments, Figure 2 The grading module 205 reads the correspondence between the comprehensive score and the sample identifier, and obtains the comprehensive score for each training sample; the comprehensive score is compared with the first threshold value and the second threshold value respectively: when the comprehensive score is greater than or equal to the first threshold value, the corresponding training sample is marked as a first-level problem sample; when the comprehensive score is less than or equal to the second threshold value, the corresponding training sample is marked as a third-level problem sample; when the comprehensive score is between the first threshold value and the second threshold value, the corresponding training sample is marked as a second-level problem sample; the sample identifier, the comprehensive score and the level label are written into a preset structured storage table according to the level label of the training sample, and a screening result is formed by aggregation.

[0131] In some embodiments,Figure 2 The output module 206 groups the labeled training samples according to the problem level in the screening result, respectively generates a first-level problem sample dataset, a second-level problem sample dataset and a third-level problem sample dataset, writes a structured record containing a sample identifier, an original annotation label, a prediction label, a confidence value and a comprehensive score for each problem sample dataset, and appends a timestamp and a version number information, writes the problem sample dataset into a specified storage location based on a preset data output interface, generates a corresponding data location identifier, constructs a metadata description file containing the data location identifier, the problem level identifier and the version number, and sends the metadata description file to a subsequent data processing module, so that the subsequent model training process can call the corresponding problem sample dataset.

[0132] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0133] Figure 3 is a structural schematic diagram of an electronic device 3 provided by the embodiments of the present application. As shown in Figure 3 The electronic device 3 of this embodiment includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The processor 301 implements the steps in each of the above method embodiments when executing the computer program 303. Alternatively, the processor 301 implements the functions of each module / unit in each of the above device embodiments when executing the computer program 303.

[0134] By way of example, the computer program 303 can be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 303 in the electronic device 3.

[0135] The electronic device 3 can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The electronic device 3 can include but is not limited to the processor 301 and the memory 302. Those skilled in the art can understand that Figure 3 The electronic device 3 is only an example and does not constitute a limitation on the electronic device 3, and can include more or fewer components than those shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc.

[0136] The processor 301 can be a central processing unit (CPU), or other general purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general purpose processor can be a microprocessor, or the processor can be any conventional processor.

[0137] The memory 302 can be an internal storage unit of the electronic device 3, for example, a hard disk or a memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like equipped on the electronic device 3. Further, the memory 302 can include both the internal storage unit and the external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0139] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0140] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0141] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0142] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0143] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0144] The integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can be executed by a processor to implement the steps of the above-mentioned various method embodiments. The computer program can include computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc.

[0145] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features. Such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An automated training data culling method, comprising: The method comprises the following steps: determining a target benchmark model from a candidate model set based on a preset model evaluation benchmark; inputting each training sample in an original training data set into the target benchmark model in sequence, obtaining a predicted label and a confidence value corresponding to the training sample, and generating a prediction result set; comparing the predicted label in the prediction result set with the labeled label of the original training data set for consistency, and obtaining a sample set with inconsistent prediction results and original labels; for each training sample in the sample set with inconsistent prediction results and original labels, matching a preset confidence interval according to the confidence value of the training sample and calling a corresponding weight, and calculating a comprehensive score by using the confidence value and the weight; comparing the comprehensive score according to preset first and second threshold values, classifying the sample set with inconsistent prediction results and original labels, and generating a screening result containing problem samples of different levels; outputting the training data set corresponding to the problem sample in the screening result to a subsequent data processing module for subsequent model training process calling.

2. The method of claim 1, wherein, The method comprises the following steps: presetting a verification data set and an evaluation index matrix containing at least two types of evaluation indexes according to the model accuracy requirement and inference overhead limitation of the task to be processed, the evaluation index matrix being configured with weight coefficients for each evaluation index; using each candidate model in the candidate model set to perform inference on the verification data set to obtain a corresponding evaluation index value set; performing weighted operation on the evaluation index value set of each candidate model according to the weight coefficients of the evaluation index matrix to obtain a corresponding aggregated evaluation score; comparing the aggregated evaluation scores of each candidate model, selecting the candidate model that meets the set preferred condition and has the highest aggregated evaluation score as the target benchmark model.

3. The method of claim 1, wherein, The method comprises the following steps: dividing the original training data set into several data batches according to a preset batch size, and assigning a unique sample identifier to each training sample in the data batch; calling the target benchmark model to perform inference batch by batch to obtain a category probability vector corresponding to each training sample; for each training sample, determining the category corresponding to the maximum probability in the category probability vector as the predicted label, and extracting the probability value as the confidence value; using the sample identifier as an index, writing the predicted label and the confidence value into a structured storage table, continuously gathering to form a prediction result set, so that the prediction result set and the original training data set maintain a one-to-one correspondence in the sample identifier dimension.

4. The method of claim 1, wherein, The method comprises the following steps: based on the sample identifier, matching the predicted label in the prediction result set with the labeled label of the corresponding sample in the original training data set one by one; According to a preset consistency determination rule, consistency determination is performed on each training sample; when the predicted label is different from the labeled label, or in a target detection scene, the intersection over union of the predicted target and the labeled target is lower than a preset threshold, the training sample is marked as an inconsistent sample; Collecting sample identifiers, predicted labels, labeled labels, and corresponding confidence values of all samples marked as inconsistent samples to generate a sample set in which the prediction result is inconsistent with the original label.

5. The method of claim 1, wherein, According to the confidence value of the training sample, the confidence interval is matched and the corresponding weight is called, and the confidence value and the weight are used to calculate the comprehensive score, which includes: A confidence interval mapping table is constructed in advance, and the confidence interval mapping table stores the corresponding relationship between a plurality of confidence intervals and target weight coefficients; For each training sample in the sample set in which the prediction result is inconsistent with the original label, the confidence value corresponding to each training sample is obtained; In the confidence interval mapping table, the confidence interval to which the confidence value belongs is determined, and the weight coefficient corresponding to the confidence interval is read; According to a preset scoring formula, the confidence value and the weight coefficient are combined to generate a comprehensive score of each training sample, and the comprehensive score and the sample identifier of the training sample are associated and stored.

6. The method of claim 1, wherein, According to the preset first threshold and second threshold, the comprehensive score is compared, the sample set in which the prediction result is inconsistent with the original label is graded, and a screening result containing different levels of problem samples is generated, which includes: Read the corresponding relationship between the comprehensive score and the sample identifier, and obtain the comprehensive score for each training sample; The comprehensive score is compared with the first threshold and the second threshold respectively: When the comprehensive score is greater than or equal to the first threshold, the corresponding training sample is marked as a first-level problem sample; When the comprehensive score is less than or equal to the second threshold, the corresponding training sample is marked as a third-level problem sample; When the comprehensive score is between the first threshold and the second threshold, the corresponding training sample is marked as a second-level problem sample; According to the level label of the training sample, the sample identifier, the comprehensive score, and the level label are written into a preset structured storage table, and are summarized to form a screening result.

7. The method of claim 6, wherein, The training data set corresponding to the problem sample in the screening result is output to a subsequent data processing module for subsequent model training process calling, which includes: According to the problem level in the screening result, the marked training samples are grouped to generate a first-level problem sample data set, a second-level problem sample data set, and a third-level problem sample data set respectively; For each problem sample data set, a structured record containing a sample identifier, an original label, a predicted label, a confidence value, and a comprehensive score is written, and a timestamp and a version number information are attached; Based on a preset data output interface, the problem sample data set is written into a specified storage location to generate a corresponding data location identifier; A metadata description file containing the data location identifier, the problem level identifier and the version number is constructed and sent to the subsequent data processing module, so that the subsequent model training process can call the corresponding problem sample dataset.

8. An automated training data culling device, comprising: Comprise: A determination module configured to determine a target benchmark model from a candidate model set based on a preset model evaluation benchmark; A generation module configured to input each training sample in an original training dataset into the target benchmark model in sequence, obtain a predicted label and a confidence value corresponding to the training sample, and generate a prediction result set; A comparison module configured to compare the predicted label in the prediction result set with a labeled label of the original training dataset for consistency, and obtain a sample set in which the prediction result is inconsistent with the original label; A matching module configured to, for each training sample in the sample set in which the prediction result is inconsistent with the original label, match a preset confidence interval and call a corresponding weight according to a confidence value of the training sample, and calculate a comprehensive score using the confidence value and the weight; A grading module configured to compare the comprehensive score with preset first and second threshold values, grade the sample set in which the prediction result is inconsistent with the original label, and generate a screening result containing problem samples of different levels; An output module configured to output a training dataset corresponding to a problem sample in the screening result to a subsequent data processing module for calling by a subsequent model training process.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images