Data filtering method and device, electronic equipment, storage medium and program product
By employing a multi-index scoring and weighted fusion method, combined with inference and truth models, harmful samples are identified and filtered, addressing the low recall rate issue in existing technologies and improving the accuracy and efficiency of model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MOTOVIS TECH SHANGHAI CO LTD
- Filing Date
- 2025-12-17
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies have low recall rates for harmful samples, leading to decreased model performance. Furthermore, their reliance on single metrics and fixed threshold strategies results in poor data filtering, making it difficult to guarantee the reliability and accuracy of model training.
By calculating the confidence distribution anomaly, inconsistency, distribution deviation, and label conflict scores of data samples, and performing weighted fusion, combined with the prediction results of the inference model and the ground truth model, harmful samples are identified and filtered using adaptive filtering thresholds and manual review mechanisms.
It improves the recall rate of harmful data, enhances the accuracy and efficiency of model training, reduces the burden of manual review, expands the detection coverage, and adapts to changes in different data characteristics.
Smart Images
Figure CN122045802A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and more specifically to data filtering methods, devices, electronic devices, storage media, and program products. Background Technology
[0002] In the field of machine learning, the quality of training data directly affects the stability and reliability of model performance. In real-world applications, newly labeled data often contains various types of harmful samples, such as mislabeled samples, adversarial samples, and out-of-distribution samples, which can lead to a decline in model performance. Therefore, it is necessary to filter harmful samples.
[0003] In related technologies, it is common to rely on a single indicator to compare with a fixed threshold, or to analyze only the model prediction results to identify harmful samples. The evaluation criteria are singular, resulting in low recall, poor data filtering effect, and difficulty in ensuring the reliability and accuracy of subsequent model training. Summary of the Invention
[0004] This invention provides a data filtering method, apparatus, electronic device, storage medium, and program product to solve the problem of low recall rate for harmful samples in related technologies.
[0005] In a first aspect, the present invention provides a data filtering method, the method comprising: Obtain the data sample set and the manually labeled results for each data sample; The data sample set is subjected to feature extraction and prediction by inference model to obtain the prediction confidence, first prediction result and intermediate layer features for each data sample. The data sample set is used to extract features and make predictions through a truth model to obtain a second prediction result for each data sample. Based on the prediction confidence, first prediction result, intermediate layer features, manual annotation result, and second prediction result of each data sample, a multi-index score is calculated for each data sample according to preset rules. The multi-index score includes at least the score for the degree of anomaly in the confidence distribution of the prediction confidence, the score for the inconsistency between the first prediction result and the second prediction result, the score for the degree of deviation of the distribution of the intermediate layer features relative to the historical data sample set, and the score for the annotation conflict between the first prediction result and the manual annotation result. The confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score of each data sample are weighted and fused to obtain the comprehensive score of each data sample. The overall score of each data sample is compared with the preset filtering conditions to obtain the filtering judgment result for each data sample.
[0006] In an optional implementation, the method further includes: The inference model is used to extract features and predict for each data sample in the data sample set to obtain the prediction confidence of each data sample; the prediction confidence includes sub-confidence corresponding to multiple prediction categories respectively; The sum of the confidence scores of each sub-confidence level, excluding the maximum predicted probability, is calculated to obtain the total confidence score of each data sample, which is used as the score of the abnormality of the confidence score distribution for each data sample.
[0007] In an optional implementation, the method further includes: Obtain the feature distribution parameters of the historical data sample set; the feature distribution parameters include the mean and covariance of the intermediate layer features corresponding to each historical data sample in the historical data sample set; Extract the intermediate layer features of each data sample; Calculate the Mahalanobis distance between the intermediate layer features of each data sample and the feature distribution parameters of the historical training data, and use it as a score for the degree of distribution deviation of each data sample.
[0008] In one optional implementation, the weighted fusion of the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score for each data sample to obtain a comprehensive score for each data sample includes: For each data sample, a first weight is assigned to the confidence distribution anomaly score, a second weight is assigned to the inconsistency score, a third weight is assigned to the distribution deviation score, and a fourth weight is assigned to the annotation conflict score; wherein, the sum of the first, second, third, and fourth weights is 1. The sum of the products of the confidence distribution anomaly score and the first weight, the inconsistency score and the second weight, the distribution deviation score and the third weight, and the labeling conflict score and the fourth weight is calculated to obtain the comprehensive score for each data sample.
[0009] In one optional implementation, comparing the overall score of each data sample with preset filtering conditions to obtain the filtering determination result for each data sample includes: The overall score of each data sample is compared with the adaptive filtering threshold; when the overall score of a data sample exceeds the adaptive filtering threshold, the filtering determination result of the data sample is that it needs to be filtered. The method further includes: Calculate the mean and standard deviation of the overall score for each data sample; The adaptive filtering threshold is updated based on the mean, the first adjustment factor corresponding to the mean, the standard deviation, and the second adjustment factor corresponding to the standard deviation, and the steps of obtaining the data sample set and the manual annotation results of each data sample are returned.
[0010] In an optional implementation, the method further includes: Send the data samples that are determined to require filtering to the manual review platform; In response to the secondary review results of the manual review platform for each data sample that has been determined to require filtering, perform at least one of the following operations, and return the steps for obtaining the data sample set and the manual annotation results for each data sample to execute the next round of data filtering: Update the first weight, second weight, third weight, and fourth weight; Update the feature distribution parameters of the current data sample set to serve as the feature distribution parameters of the historical data sample set for the next round of data filtering; Update the first adjustment factor and the second adjustment factor; Add the data samples that require filtering based on the second review results to the negative sample library.
[0011] In a second aspect, the present invention provides a data filtering device, the device comprising: The data acquisition module is used to acquire the data sample set and the manually labeled results for each data sample; The first prediction module is used to extract features and predict the data sample set through an inference model to obtain the prediction confidence, first prediction result and intermediate layer features for each data sample. The second prediction module is used to extract features and predict the data sample set through a truth model to obtain a second prediction result for each data sample. The scoring calculation module is used to calculate a multi-index score for each data sample based on the prediction confidence, the first prediction result, the intermediate layer features, the manual annotation result, and the second prediction result, according to preset rules. The multi-index score includes at least a score for the degree of anomaly in the confidence distribution of the prediction confidence, a score for the inconsistency between the first and second prediction results, a score for the degree of deviation of the distribution of the intermediate layer features relative to the historical data sample set, and a score for the annotation conflict between the first prediction result and the manual annotation result. The weighted fusion module is used to weight and fuse the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score of each data sample to obtain the comprehensive score of each data sample. The filtering and determination module is used to compare the comprehensive score of each data sample with the preset filtering conditions to obtain the filtering and determination result of each data sample.
[0012] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data filtering method described in the first aspect or any corresponding embodiment thereof.
[0013] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the data filtering method described in the first aspect or any corresponding embodiment thereof.
[0014] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the data filtering method described in the first aspect or any corresponding embodiment thereof.
[0015] The technical solution provided by this invention may include the following beneficial effects: The data filtering method provided by this invention calculates and weights the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score for each data sample, thereby identifying various harmful samples from different perspectives. It fully leverages the advantages of each dimension, overcomes the limitations of single detection methods, and improves the recall rate of harmful data compared to single-index methods in related technologies. This effectively prevents harmful data from interfering with model training and improves the accuracy of subsequent models. Through high-recall automatic data filtering, it avoids the drawback of requiring manual review in related technologies, improving data filtering efficiency. By using the difference between the prediction results of the inference model and the ground truth model as the basis for data filtering, it can discover potential problematic samples through dual-model cross-validation, increasing the detection coverage. Attached Figure Description
[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of the first type of data filtering method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a second process for a data filtering method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the third process of the data filtering method according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a data filtering device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.
[0020] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0021] In the field of machine learning, the quality of training data directly affects the stability and reliability of model performance. In practical application scenarios, newly labeled data often contains various types of harmful samples, such as mislabeled samples, adversarial samples, and out-of-distribution samples. Once these harmful samples enter the training process, they may cause a decline in model performance, bias, or even security vulnerabilities. Therefore, it is urgent to filter harmful samples.
[0022] The main shortcomings of related technologies for detecting harmful samples are as follows: 1. Relying on a single indicator and lacking multi-dimensional comprehensive analysis capabilities results in limited identification capabilities, leading to a large number of harmful samples failing to be effectively detected. 2. The analysis is based solely on model predictions, failing to fully utilize the comparative information between labeled data and the inference results of deployed algorithms, and lacking a multi-system cross-validation mechanism, resulting in missed detection of hidden and harmful samples; 3. Using a fixed threshold strategy to determine whether a sample is harmful is difficult to adapt to changes in data characteristics under different scenarios, resulting in insufficient recall or an excessive burden on manual review.
[0023] According to an embodiment of the present invention, a data filtering method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0024] This embodiment provides a data filtering method that can be used on laptops, desktops, and industrial computers. Figure 1 This is a flowchart of a data filtering method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the data sample set and the manual annotation results for each data sample.
[0025] This data sample set consists of multiple new data samples, each of which has been pre-annotated manually and has corresponding manual annotation results.
[0026] Step S102: The data sample set is subjected to feature extraction and prediction through the inference model to obtain the prediction confidence, first prediction result and intermediate layer features for each data sample.
[0027] The inference model refers to the main model currently deployed or planned for use in actual tasks. It is responsible for predicting (classification, detection, etc.) new input data, and its output is directly used for business decisions. In this embodiment, the inference model performs feature extraction and forward inference prediction on each data sample in the data sample set, obtaining intermediate layer features, prediction confidence, and a first prediction result for each data sample. The intermediate layer features indicate the feature representation vector output by the hidden layer (non-output layer) during the inference model's processing of input data, capturing the abstract semantic information of the data sample. The prediction confidence indicates the confidence level of each predicted category for the data sample obtained by the inference model; for example, the confidence level for the predicted category of pedestrian is 0.8, and the confidence level for the predicted category of vehicle is 0.2. The first prediction result indicates the final category prediction result made by the inference model for the data sample; for example, the prediction result is pedestrian.
[0028] Step S103: The data sample set is feature extracted and predicted using a truth model to obtain a second prediction result for each data sample.
[0029] The ground truth model provides predictions that approximate the true labels, serving as a reference model to verify the accuracy of the inference model's predictions. The second prediction indicates the final category prediction made by the ground truth model for the data sample, such as a prediction of "vehicle."
[0030] Step S104: Calculate the multi-index score for each data sample based on the prediction confidence, first prediction result, intermediate layer features, manual annotation result, and second prediction result using preset rules.
[0031] The multi-index scoring includes at least the score for the degree of anomaly in the confidence distribution of the prediction confidence, the score for the inconsistency between the first and second prediction results, the score for the degree of deviation of the distribution of intermediate layer features relative to the historical data sample set, and the score for the labeling conflict between the first prediction result and the manually labeled result.
[0032] The preset rules can be set according to actual needs. For example, the greater the difference between the first and second prediction results, the higher the inconsistency score; the greater the deviation of the distribution of intermediate layer features from the historical data sample set, the higher the distribution deviation score.
[0033] The confidence distribution anomaly score indicates the degree of anomaly in the distribution of confidence scores corresponding to each predicted category of the data sample. For example, if the difference in confidence scores among the predicted categories is small, it indicates that the inference model has difficulty confirming the final predicted category, resulting in a high confidence distribution anomaly score. The inconsistency score indicates the similarity between the first prediction result of the inference model and the second prediction result of the ground truth model. A low similarity indicates a high inconsistency score. The distribution deviation score indicates the distance between the intermediate layer features and the data distribution center of the historical data sample set. A high distribution deviation score indicates a high distribution deviation score. The annotation conflict score indicates the similarity between the first prediction result of the inference model and the manually annotated result. A low similarity indicates many annotation conflicts, resulting in a high annotation conflict score.
[0034] Step S105: The confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score of each data sample are weighted and fused to obtain the comprehensive score of each data sample.
[0035] We assign corresponding weights to the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score, and then perform weighted fusion to obtain the comprehensive score of the data sample.
[0036] Step S106: Compare the comprehensive score of each data sample with the preset filtering conditions to obtain the filtering judgment result of each data sample.
[0037] The preset filtering conditions are used to indicate the filtering criteria for data samples. Data samples whose comprehensive scores meet the preset filtering conditions are harmful samples, and the filtering judgment result is that they need to be filtered; data samples whose comprehensive scores do not meet the preset filtering conditions are normal samples, and the filtering judgment result is that they do not need to be filtered.
[0038] The data filtering method provided in this embodiment calculates and weights the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score for each data sample, thereby identifying various harmful samples from different perspectives. It fully leverages the advantages of each dimension, overcomes the limitations of single detection methods, and improves the recall rate of harmful data compared to single-index methods in related technologies. This effectively prevents harmful data from interfering with model training and improves the accuracy of subsequent models. Through high-recall automatic data filtering, it avoids the drawback of requiring manual review in related technologies, improving data filtering efficiency. By using the difference between the prediction results of the inference model and the ground truth model as the basis for data filtering, it can discover potential problematic samples through dual-model cross-validation, increasing the detection coverage.
[0039] This embodiment provides a data filtering method that can be used in laptops, desktops, industrial computers, etc. Figure 2 This is a flowchart of a data filtering method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the data sample set and the manual annotation results for each data sample.
[0040] For example, receiving a newly labeled data sample set D new ={(x i ,y i ), i∈[1,n]}, where x i For the data sample, y i These are the manually labeled results corresponding to the data samples.
[0041] For example, steps S201 to S208 are executed cyclically, and each execution of steps S201 to S208 represents the completion of one round of data filtering cycle task. On the first execution, the currently deployed inference model M needs to be loaded. deploy And the truth model M truthInitialize the feature distribution parameters of the historical data sample set. These feature distribution parameters include the average value μ of the intermediate layer features corresponding to each historical data sample in the historical data sample set. train Covariance Σ train Assign a first weight w1 to the confidence distribution anomaly score, a second weight w2 to the inconsistency score, a third weight w3 to the distribution deviation score, and a fourth weight w4 to the annotation conflict score; the sum of the first, second, third, and fourth weights is 1, i.e., w1 + w2 + w3 + w4 = 1; set an adaptive filtering threshold T. adaptive .
[0042] For example, in parking, driving, and fusion perception scenarios, the manually labeled results include, but are not limited to, the 2D / 3D bounding boxes, category labels, motion states, drivable area divisions, lane topology, traffic signal status, and spatial alignment relationships between multiple sensors. These manually labeled results serve as a ground truth reference, used for comparative analysis with the predicted output of the deployed model to identify potentially harmful data samples with labeling errors, model misjudgments, or abnormal distributions. Specifically, in the automatic parking scenario, the acquired data samples may include surround-view camera video streams (4-6 fisheye channels), radar data collected by ultrasonic radar for near-range obstacle distance detection, and vehicle status signals (e.g., steering angle, speed, gear, whether a parking command has been initiated). The manually labeled results may include drivable areas, parking space boundary points and types, static obstacle locations and categories, dynamic obstacle trajectories, safe zones, and collision risk zones; in the driving scenario… In the Perception scenario, the acquired data samples can include RGB video streams captured by forward / side-facing cameras, 3D spatial point cloud data acquired by LiDAR, speed and orientation information of long-range moving targets acquired by millimeter-wave radar, vehicle position provided by GPS / IMU / map priors, lane topology information, etc. Manual annotation results can include 2D / 3D bounding boxes (including size, orientation, and truncation degree) of target objects (e.g., cars, pedestrians, cyclists, traffic signs, etc.), target categories and attributes (e.g., whether stationary, whether holding an umbrella, whether cycling, etc.), target motion state (e.g., speed, acceleration, predicted trajectory direction), traffic light and sign status (e.g., traffic light colors, speed limit sign values, arrow directions), lane line types and connectivity (e.g., solid / dashed lines, left turn / straight ahead, lane change feasibility); in fusion perception (Sensor) In the FusionPerception scenario, the acquired data samples can include video streams captured by multi-view cameras (front, rear, left, right, front side, and rear side), point cloud data collected by LiDAR, speed and orientation information of multiple moving targets distributed on the front and rear bumpers, covering different angles and distances provided by millimeter-wave radar within their corresponding acquisition range, timestamp synchronization signals, and vehicle attitude information. The manually labeled results can include a 3D target list (each target has a precise 3D bounding box, ID, category, and motion vector), multimodal consistency labels (used to indicate whether the same target is observed by different sensors and whether there is occlusion), occlusion and truncation levels (e.g., the proportion of the target being occluded), abnormal event markers (e.g., false lane lines), and cross-modal correlations (e.g., target objects in images, point clusters of the target object in point cloud data, and radar echo signals corresponding to the target object in radar data).
[0043] Step S202: The data sample set is subjected to feature extraction and prediction through the inference model to obtain the prediction confidence, first prediction result and intermediate layer features for each data sample.
[0044] Please see details Figure 1 Step S102 of the illustrated embodiment will not be described again here.
[0045] Step S203: The data sample set is feature extracted and predicted using a truth model to obtain a second prediction result for each data sample.
[0046] Please see details Figure 1 Step S103 of the illustrated embodiment will not be described again here.
[0047] Step S204: Calculate the multi-index score for each data sample based on the prediction confidence, first prediction result, intermediate layer features, manual annotation result, and second prediction result using preset rules.
[0048] The multi-index scoring includes at least the score for the degree of anomaly in the confidence distribution of the prediction confidence, the score for the inconsistency between the first and second prediction results, the score for the degree of deviation of the distribution of intermediate layer features relative to the historical data sample set, and the score for the labeling conflict between the first prediction result and the manually labeled result.
[0049] Optionally, when calculating the confidence distribution anomaly score, firstly, feature extraction and prediction are performed on each data sample in the data sample set using the inference model to obtain the prediction confidence of each data sample; this prediction confidence includes sub-confidences corresponding to multiple prediction categories. Next, the sum of all sub-confidences except the maximum prediction probability is calculated to obtain the total confidence score for each data sample, which serves as the confidence distribution anomaly score for each data sample. This score indicates the total "competition intensity" of other prediction categories besides the prediction category corresponding to the maximum prediction probability. The lower the confidence distribution anomaly score, the more confident the inference model is in its prediction results. This total confidence score can also be obtained by calculating the difference between 1 and the maximum prediction probability. Finally, the confidence distribution anomaly score for each data sample is compared with a preset confidence threshold. When the confidence distribution anomaly score of a data sample is greater than the preset confidence threshold, it indicates that the data sample is a harmful sample, and it is labeled "suspicious" or "harmful"; otherwise, it indicates that the data sample is not a harmful sample and can participate in subsequent processes normally.
[0050] For example, using inference model M deploy For each data sample x i Make a prediction and obtain the prediction confidence vector P. i =[p i1 ,pi2 ,...,p ik ], where k is the number of categories. Next, calculate the non-highest confidence predicted category confidence and S for the sample. conf (x i )=1-max k p ik Then, set the confidence threshold T. conf When S conf (x i )>T conf If so, the data sample is determined to be a harmful sample.
[0051] For example, when calculating the inconsistency score, the inference model M is first obtained respectively. deploy And the truth model M truth For the same data sample x i The first prediction result argmaxP i deploy The second prediction result argmaxP i truth Next, the consistency score S between the prediction results of the two algorithms is calculated. consist (x i )=I(argmaxP i deploy ≠argmaxP i truth ), where I( The function is an indicator function. It has a value of 1 when the predictions are inconsistent, indicating that the two models are inconsistent; otherwise, it has a value of 0, indicating that the two models are consistent.
[0052] Optionally, when calculating the distribution deviation score, the feature distribution parameters of the historical data sample set are first obtained. These parameters include the mean and covariance of the intermediate layer features corresponding to each historical data sample. Next, the intermediate layer features of each data sample are extracted. Then, the Mahalanobis distance between the intermediate layer features of each data sample and the feature distribution parameters of the historical training data is calculated as the distribution deviation score for each data sample. Finally, the distribution deviation score of each data sample is compared with a preset distribution deviation threshold. If the distribution deviation score is greater than the preset threshold, the data sample is considered a harmful sample and is labeled "suspicious" or "harmful." Otherwise, the data sample is not considered a harmful sample and can proceed normally through subsequent processes.
[0053] For example, firstly, extract the features f of the samples in the intermediate layer of the deployed model. i Next, the intermediate layer features f are calculated. i Mahalanobis distance to the feature distribution parameters of historical training data:
[0054] Wherein, the feature distribution parameter of the historical training data is the average value μ of the intermediate layer features corresponding to each historical data sample in the historical data sample set. train Covariance Σ train Finally, the Mahalanobis distance is compared with the preset distribution deviation threshold T. dist Comparison, when S dist (x i )>T dist If so, the data sample is determined to be a harmful sample.
[0055] For example, when calculating the annotation conflict score, the first prediction result argmaxP is calculated. i deploy Compared with manually labeled results y i The label conflict score between them S conflict (x i )=I(argmaxP i deploy ≠y i ), where I( The function is an indicator function. It is 1 when there is a labeling conflict, indicating that the first prediction result is inconsistent with the manual labeling result; otherwise, it is 0, indicating that the first prediction result is consistent with the manual labeling result.
[0056] Step S205: The confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score of each data sample are weighted and fused to obtain the comprehensive score of each data sample.
[0057] Optionally, a first weight is assigned to the confidence distribution anomaly score of each data sample, a second weight is assigned to the inconsistency score of each data sample, a third weight is assigned to the distribution deviation score of each data sample, and a fourth weight is assigned to the annotation conflict score of each data sample; wherein the sum of the first weight, the second weight, the third weight, and the fourth weight is 1.
[0058] Next, the sum of the product of the confidence distribution anomaly score and the first weight, the product of the inconsistency score and the second weight, the product of the distribution deviation score and the third weight, and the product of the labeling conflict score and the fourth weight is calculated to obtain the comprehensive score for each data sample.
[0059] For example, the overall score S total (x i )=w1×S conf (x i )+w2×S consist (x i )+w3×S dist (xi )+w4×S conflict (x i ).
[0060] Step S206: Compare the comprehensive score of each data sample with the preset filtering conditions to obtain the filtering judgment result of each data sample.
[0061] Optionally, the overall score of each data sample is compared with the adaptive filtering threshold. When the overall score of a data sample exceeds the adaptive filtering threshold, the filtering result of that data sample is that it needs to be filtered; otherwise, the filtering result of that data sample is that it does not need to be filtered.
[0062] For example, the overall score S of the data sample total (x i ) and adaptive filtering threshold T adaptive Comparison, when S total (x i )>T adaptive When determining data sample x i These are harmful samples and need to be filtered. The final output is a set of harmful data samples, D. harmful ={x i |S total (x i )>T adaptive}
[0063] Step S207: Calculate the mean and standard deviation of the comprehensive score for each data sample.
[0064] After obtaining the filtering decision result for each data sample, the adaptive filtering threshold is adjusted to apply to the next round of data filtering loop task. Specifically, the average value of the comprehensive score for each data sample and the covariance of the comprehensive score for each data sample are first calculated.
[0065] Step S208: Update the adaptive filtering threshold based on the average value, the first adjustment coefficient corresponding to the average value, the standard deviation, and the second adjustment coefficient corresponding to the standard deviation, and return the steps of obtaining the data sample set and the manual annotation results of each data sample.
[0066] For example, the adaptive filtering threshold T adaptive =α×μ S +β×σ S , where μ S α is the average value, α is the first adjustment factor corresponding to the average value, and σ is the average value. S denoted as σ, where β is the standard deviation and β is the second adjustment factor corresponding to the standard deviation.
[0067] Furthermore, the distribution changes of historical data samples from different rounds can be analyzed to adjust the first and second adjustment coefficients, thereby further improving the accuracy of data filtering.
[0068] Optionally, a second manual review is also conducted to further determine whether the filtered harmful data is indeed harmful. Specifically, each data sample that is determined to require filtering is sent to the manual review platform for a second review. In response to the second review results of the manual review platform, the data samples that are determined to require filtering are selected, and at least one of the following operations is performed to update the relevant parameters in the next round of data filtering loop task. The step of obtaining the data sample set and the manual annotation results of each data sample (step S201) is then returned to execute the next round of data filtering loop task, achieving closed-loop optimization: Update the first, second, third, and fourth weights for use in the next round of weighted fusion; Update the feature distribution parameters of the current data sample set to serve as the feature distribution parameters of the historical data sample set for the next round of data filtering; Update the first adjustment coefficient and the second adjustment coefficient to adjust the adaptive filtering threshold for the next round; Data samples that require filtering after secondary review are added to the negative sample library as historical records for subsequent model updates and improvements.
[0069] Optionally, when updating the first, second, third, and fourth weights, a dynamic weight learning mechanism can be set. By constructing a lightweight neural network model, it receives various scores and metadata of the data samples (including sensor type, environmental conditions, target distance, etc.) and automatically outputs personalized fusion weights. This mechanism can adaptively adjust the attention given to the degree of confidence distribution anomaly, distribution deviation, inconsistency, and labeling conflict according to different application scenarios, thereby improving the system's sensitivity to detecting novel attacks and rare error types. For example, in a close-range obstacle detection scenario during parking, fisheye lenses may have image blurring and distortion, leading to feature shifts and easily generating out-of-distribution samples. Therefore, the third weight can be increased. When a low degree of confidence distribution anomaly is detected but the two models are consistent, it indicates that the data sample is truly blurred rather than a two-model adversarial situation, requiring a decrease in the first weight and an increase in the second weight. When the distance is close and there is a labeling conflict, it indicates that the vehicle is approaching the collision point, and any labeling error will cause danger. Therefore, the fourth weight should be increased. In high-speed target recognition scenarios while driving, adversarial disturbances may lead to misjudgments (such as tampering with traffic signs), requiring cross-validation between two models. Therefore, the second and fourth weights are increased. In multimodal input scenarios in fusion perception, if multimodal data is available and consistent, the first and third weights can be reduced to decrease the intensity of abnormal responses. If the quality of a single modality data is poor, such as high radar noise in rainy weather, the first weight corresponding to that modality needs to be reduced to prevent low-quality input from affecting model decisions.
[0070] The data filtering method provided in this embodiment calculates the average and standard deviation of the comprehensive score of each data sample, and updates the adaptive filtering threshold according to the average, the first adjustment coefficient corresponding to the average, the standard deviation, and the second adjustment coefficient corresponding to the standard deviation. This forms an adaptive threshold adjustment mechanism, which can adapt to different data distributions and business scenarios, and is not limited by specific fields. It can maintain stable harmful data recall performance in various tasks such as image classification, detection, segmentation, and anomaly detection. By sending each data sample that needs to be filtered to the manual review platform, and adjusting the relevant parameters in the data filtering loop task in response to the secondary review results, a closed-loop optimization can be formed, which can continuously improve the results. After several iterations, the accuracy can be improved while maintaining a high recall rate, forming a virtuous cycle.
[0071] As one or more specific application embodiments of the present invention, the optimal implementation scheme or the scheme that the inventors most want to embody is described in combination with the specific application scenario.
[0072] Figure 3 This is a flowchart of a data filtering method according to an embodiment of the present invention, such as... Figure 3As shown, the process begins by inputting manually labeled data samples. Next, multi-dimensional feature extraction is performed on the data samples, including confidence analysis to calculate confidence scores, consistency analysis to calculate consistency scores, distribution analysis to calculate distribution deviation scores, and conflict analysis to calculate label conflict scores. Then, a decision fusion center performs a weighted fusion of the confidence scores, consistency scores, distribution deviation scores, and label conflict scores to obtain a weighted total score. This weighted total score is then compared to a preset threshold. If the weighted total score exceeds the preset threshold, the data is deemed harmful and requires secondary manual review and confirmation. The model is then updated based on the confirmation result. If the weighted total score does not exceed the preset threshold, the data is deemed normal, and the training process continues.
[0073] This embodiment also provides a data filtering device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0074] This embodiment provides a data filtering device, such as... Figure 4 As shown, it includes: The data acquisition module 401 is used to acquire the data sample set and the manual annotation results of each data sample; The first prediction module 402 is used to extract features and predict the data sample set through the inference model to obtain the prediction confidence, the first prediction result and the intermediate layer features for each data sample. The second prediction module 403 is used to extract features and predict the data sample set through the true value model to obtain the second prediction result for each data sample. The scoring calculation module 404 is used to calculate a multi-index score for each data sample based on the prediction confidence, the first prediction result, the intermediate layer features, the manual annotation result, and the second prediction result, according to preset rules. The multi-index score includes at least the score for the degree of abnormality in the confidence distribution of the prediction confidence, the score for the inconsistency between the first prediction result and the second prediction result, the score for the degree of deviation of the distribution of the intermediate layer features relative to the historical data sample set, and the score for the annotation conflict between the first prediction result and the manual annotation result. The weighted fusion module 405 is used to weight and fuse the confidence distribution anomaly score, inconsistency score, distribution deviation score and annotation conflict score of each data sample to obtain the comprehensive score of each data sample. The filtering judgment module 406 is used to compare the comprehensive score of each data sample with the preset filtering conditions to obtain the filtering judgment result of each data sample.
[0075] In an optional implementation, the device further includes a confidence distribution anomaly scoring module, used for: The inference model is used to extract features and predict for each data sample in the data sample set, and the prediction confidence of each data sample is obtained; the prediction confidence includes sub-confidence corresponding to multiple prediction categories respectively; Calculate the sum of the confidence scores of each sub-confidence score except for the maximum predicted probability to obtain the total confidence score of each data sample; The sum of the confidence scores for each data sample is compared with a preset confidence threshold to determine the degree of anomaly in the confidence distribution of each data sample.
[0076] In an optional implementation, the device further includes a distribution deviation scoring module for: Obtain the feature distribution parameters of the historical data sample set; these feature distribution parameters include the mean and covariance of the intermediate layer features corresponding to each historical data sample in the historical data sample set; Extract the intermediate layer features of each data sample; Calculate the Mahalanobis distance between the intermediate layer features of each data sample and the feature distribution parameters of the historical training data to obtain the distribution deviation parameter of each data sample; The distribution deviation parameter of each data sample is compared with the preset distribution deviation threshold to obtain the distribution deviation score of each data sample.
[0077] In one optional implementation, the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score for each data sample are weighted and fused to obtain a comprehensive score for each data sample, including: For each data sample, a first weight is assigned to the confidence distribution anomaly score, a second weight is assigned to the inconsistency score, a third weight is assigned to the distribution deviation score, and a fourth weight is assigned to the annotation conflict score; wherein, the sum of the first, second, third, and fourth weights is 1. The sum of the products of the confidence distribution anomaly score and the first weight, the inconsistency score and the second weight, the distribution deviation score and the third weight, and the labeling conflict score and the fourth weight is calculated to obtain the comprehensive score for each data sample.
[0078] In one optional implementation, the filtering determination module is further configured to: The overall score of each data sample is compared with the adaptive filtering threshold; when the overall score of a data sample exceeds the adaptive filtering threshold, the filtering result of that data sample is that it needs to be filtered. The device also includes a threshold adjustment module for: Calculate the mean and standard deviation of the overall score for each data sample; The adaptive filtering threshold is updated based on the mean, the first adjustment factor corresponding to the mean, the standard deviation, and the second adjustment factor corresponding to the standard deviation, and the steps of obtaining the data sample set and the manually labeled results of each data sample are returned.
[0079] In an alternative implementation, the device further includes an update module for: Send the data samples that are determined to require filtering to the manual review platform; In response to the secondary review results of the manual review platform for each data sample that has been determined to require filtering, perform at least one of the following operations, and return the steps for obtaining the data sample set and the manual annotation results for each data sample to execute the next round of data filtering: Update the first, second, third, and fourth weights; Update the feature distribution parameters of the current data sample set to serve as the feature distribution parameters of the historical data sample set for the next round of data filtering; Update the first adjustment factor and the second adjustment factor; Add the data samples that require filtering based on the second review results to the negative sample library.
[0080] The data filtering device provided in this embodiment of the invention can execute the data filtering method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.
[0081] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.
[0082] The following is a detailed reference. Figure 5 This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0083] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0084] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the data filtering method of the embodiments of the present invention.
[0085] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0086] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded via a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data filtering method shown in the above embodiments is implemented.
[0087] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0088] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the invention.
Claims
1. A data filtering method, characterized in that, The method includes: Obtain the data sample set and the manually labeled results for each data sample; The data sample set is subjected to feature extraction and prediction by inference model to obtain the prediction confidence, first prediction result and intermediate layer features for each data sample. The data sample set is used to extract features and make predictions through a truth model to obtain a second prediction result for each data sample. Based on the prediction confidence, first prediction result, intermediate layer features, manual annotation result, and second prediction result of each data sample, a multi-index score is calculated for each data sample according to preset rules. The multi-index score includes at least the score for the degree of anomaly in the confidence distribution of the prediction confidence, the score for the inconsistency between the first prediction result and the second prediction result, the score for the degree of deviation of the distribution of the intermediate layer features relative to the historical data sample set, and the score for the annotation conflict between the first prediction result and the manual annotation result. The confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score of each data sample are weighted and fused to obtain the comprehensive score of each data sample. The overall score of each data sample is compared with the preset filtering conditions to obtain the filtering judgment result for each data sample.
2. The method according to claim 1, characterized in that, The method further includes: The inference model is used to extract features and predict for each data sample in the data sample set to obtain the prediction confidence of each data sample; the prediction confidence includes sub-confidence corresponding to multiple prediction categories respectively; The sum of the confidence scores of each sub-confidence level, excluding the maximum predicted probability, is calculated to obtain the total confidence score of each data sample, which is used as the score of the abnormality of the confidence score distribution for each data sample.
3. The method according to claim 1, characterized in that, The method further includes: Obtain the feature distribution parameters of the historical data sample set; the feature distribution parameters include the mean and covariance of the intermediate layer features corresponding to each historical data sample in the historical data sample set; Extract the intermediate layer features of each data sample; Calculate the Mahalanobis distance between the intermediate layer features of each data sample and the feature distribution parameters of the historical training data, and use it as a score for the degree of distribution deviation of each data sample.
4. The method according to any one of claims 1 to 3, characterized in that, The confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score for each data sample are weighted and fused to obtain a comprehensive score for each data sample, including: For each data sample, a first weight is assigned to the confidence distribution anomaly score, a second weight is assigned to the inconsistency score, a third weight is assigned to the distribution deviation score, and a fourth weight is assigned to the annotation conflict score; wherein, the sum of the first, second, third, and fourth weights is 1. The sum of the products of the confidence distribution anomaly score and the first weight, the inconsistency score and the second weight, the distribution deviation score and the third weight, and the labeling conflict score and the fourth weight is calculated to obtain the comprehensive score for each data sample.
5. The method according to claim 4, characterized in that, The step of comparing the comprehensive score of each data sample with preset filtering conditions to obtain the filtering determination result for each data sample includes: The overall score of each data sample is compared with the adaptive filtering threshold; when the overall score of a data sample exceeds the adaptive filtering threshold, the filtering determination result of the data sample is that it needs to be filtered. The method further includes: Calculate the mean and standard deviation of the overall score for each data sample; The adaptive filtering threshold is updated based on the mean, the first adjustment factor corresponding to the mean, the standard deviation, and the second adjustment factor corresponding to the standard deviation, and the steps of obtaining the data sample set and the manual annotation results of each data sample are returned.
6. The method according to claim 5, characterized in that, The method further includes: Send the data samples that are determined to require filtering to the manual review platform; In response to the secondary review results of the manual review platform for each data sample that has been determined to require filtering, perform at least one of the following operations, and return the steps for obtaining the data sample set and the manual annotation results for each data sample to execute the next round of data filtering: Update the first weight, second weight, third weight, and fourth weight; Update the feature distribution parameters of the current data sample set to serve as the feature distribution parameters of the historical data sample set for the next round of data filtering; Update the first adjustment factor and the second adjustment factor; Add the data samples that require filtering based on the second review results to the negative sample library.
7. A data filtering device, characterized in that, The device includes: The data acquisition module is used to acquire the data sample set and the manually labeled results for each data sample; The first prediction module is used to extract features and predict the data sample set through an inference model to obtain the prediction confidence, first prediction result and intermediate layer features for each data sample. The second prediction module is used to extract features and predict the data sample set through a truth model to obtain a second prediction result for each data sample. The scoring calculation module is used to calculate a multi-index score for each data sample based on the prediction confidence, the first prediction result, the intermediate layer features, the manual annotation result, and the second prediction result, according to preset rules. The multi-index score includes at least a score for the degree of anomaly in the confidence distribution of the prediction confidence, a score for the inconsistency between the first and second prediction results, a score for the degree of deviation of the distribution of the intermediate layer features relative to the historical data sample set, and a score for the annotation conflict between the first prediction result and the manual annotation result. The weighted fusion module is used to weight and fuse the confidence distribution anomaly score, inconsistency score, distribution deviation score, and annotation conflict score of each data sample to obtain the comprehensive score of each data sample. The filtering and determination module is used to compare the comprehensive score of each data sample with the preset filtering conditions to obtain the filtering and determination result of each data sample.
8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the data filtering method of any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the data filtering method according to any one of claims 1 to 6.
10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the data filtering method according to any one of claims 1 to 6.