A training data cleaning method, device, equipment and medium

Abstract

This application provides a training data cleaning method, apparatus, device, and medium, relating to the field of data processing technology. A corresponding cleaning rule base is pre-constructed for each model in an autonomous driving system. Using each cleaning rule base, the initial training data in the initial training dataset to be cleaned is differentially arbitrated, i.e., candidate cleaning labels are determined for the initial training data under each cleaning rule base. All candidate cleaning labels are combined to determine the target cleaning label for the initial training data. The initial training data is processed according to the target cleaning label until all initial training data has been processed, resulting in the target training dataset. In the case of multiple models sharing a training dataset, the target training dataset balances the training data requirements of each model in the autonomous driving system, aiming to ensure the overall performance of the autonomous driving system and avoid safety hazards.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, equipment and medium for cleaning training data. Background Technology

[0002] With the rapid development of artificial intelligence technology, machine learning models are being used more and more widely in the field of autonomous driving. Currently, multiple machine learning models with different functions (e.g., perception models, prediction models, and planning models) are typically set up in autonomous driving systems to collaboratively complete complex autonomous driving tasks. Furthermore, to reduce costs and meet the data consistency requirements of each machine learning model, the same training dataset is usually used to train all models in the autonomous driving system.

[0003] For machine learning models, the quality of training data is crucial. However, in practical applications, raw training data often suffers from numerous quality issues, such as blurred images, incorrect annotations, and missing annotations. Directly using such "dirty data" for model training can lead to decreased model accuracy and poor generalization ability, consequently degrading the performance of autonomous driving systems and posing safety hazards. Therefore, training data cleaning has become a critical preprocessing step to ensure model performance.

[0004] Current data cleaning methods mainly include rule-based cleaning methods and model-based cleaning methods. Rule-based cleaning methods rely on thresholds set based on human experience, such as removing images with image clarity below a preset threshold. Model-based methods, on the other hand, use pre-trained data quality assessment models (e.g., binary classification models) to automatically identify low-quality images and remove them.

[0005] However, current data cleaning methods tend to use a "one-size-fits-all" approach, resulting in datasets that fail to balance the training data needs of different models in autonomous driving systems. Therefore, using such datasets to train models in autonomous driving systems can negatively impact the overall performance of the system.

[0006] Therefore, there is a need to provide a training data cleaning method to meet the training requirements of different machine learning models in autonomous driving systems. Summary of the Invention

[0007] In view of this, this application provides a training data cleaning method, the method comprising: Obtain the initial training dataset to be cleaned; wherein, the initial training dataset is a dataset shared by various models in the autonomous driving system during pre-training, and the functions of each model are different; the initial training dataset consists of initial training data; The pre-built cleaning rule libraries are invoked; wherein, each cleaning rule library corresponds one-to-one with the model; the cleaning rule library is used to store data cleaning rules. For each of the cleaning rule bases, candidate cleaning labels for the initial training data under the cleaning rule bases are determined according to each of the data cleaning rules in the cleaning rule bases. Based on all the candidate cleaning labels, determine the target cleaning label for the initial training data; The initial training data is processed according to the target cleaning label until all the initial training data has been processed to obtain the target training dataset.

[0008] Optionally, the cleaning rule base is constructed, including: When it is determined that the autonomous driving system has malfunctioned, the abnormal model is located from among the models. Obtain the historical output results of the anomaly model; and analyze the historical output results to determine the target anomaly type corresponding to the anomaly model from each preset anomaly type; From the initial training dataset, erroneous training data is determined; wherein, the erroneous training data is the training data that causes the abnormal model to output abnormalities; Determine the common data features of the erroneous training data; and create target data cleaning rules corresponding to the target anomaly type based on the common data features. Based on the target data cleaning rules, a cleaning rule library corresponding to the anomaly model is constructed.

[0009] Optionally, identifying erroneous training data from the initial training dataset includes: Determine the data features of the initial training data to obtain the initial data features; Obtain a pre-constructed test dataset; wherein the test dataset consists of test data, and the test data is pre-labeled with real output results; The test data is input into the anomaly model to obtain the prediction output of the anomaly model; Data in the test data in which the consistency between the predicted output and the actual output does not meet the preset requirements are designated as erroneous test data; wherein, the consistency is used to characterize the degree of consistency between the predicted output and the actual output. Determine the data characteristics of the erroneous test data to obtain key data characteristics; The training data in the initial training dataset whose initial data features are consistent with the key data features are used as the erroneous training data.

[0010] Optionally, determining the data characteristics of the erroneous test data includes: Obtain the intended areas of interest that have been pre-annotated for the error test data; A heatmap of the anomaly model's attention to the erroneous test data is determined; wherein, the value of a pixel in the heatmap is used to characterize the degree of attention the anomaly model pays to that pixel; the value is positively correlated with the degree of attention. Based on the heatmap of interest, actual areas of interest that meet preset conditions are extracted from the erroneous test data; wherein, the preset condition is that the pixel value is higher than a preset threshold. By comparing the actual area of ​​interest with the expected area of ​​interest, the deviation area is determined. The data characteristics of the deviation region are used as the key data characteristics.

[0011] Optionally, based on the target data cleaning rules, a cleaning rule base corresponding to the anomaly model is constructed, including: Determine whether there are existing data cleaning rules in the current cleaning rule base that correspond to the target anomaly type; wherein, the current cleaning rule base is the cleaning rule base corresponding to the anomaly model; If it does not exist, the target data cleaning rule is added to the current cleaning rule library; If it exists, then perform the following steps: Based on the erroneous training data, the defect type corresponding to the existing data cleaning rules is determined; wherein, the defect type is either insufficient coverage or excessive coverage; the insufficient coverage type indicates insufficient cleaning of the erroneous training data; the excessive coverage type indicates excessive cleaning of the correct training data. Based on the defect type, adjust the existing data cleaning rules until the specified indicators of the anomaly model meet the requirements; wherein the specified indicators include at least one of recall rate and false positive rate.

[0012] Optionally, based on the data cleaning rules in the cleaning rule base, candidate cleaning labels for the initial training data under the cleaning rule base are determined, including: Determine the data features of the initial training data to obtain the initial data features; The initial data features are matched one by one with each of the data cleaning rules to determine whether the initial data features conform to any of the data cleaning rules. If so, then the candidate cleaning label is determined to be a cleaning label that represents cleaning the initial training data; If not, then determine the deviation degree between the initial data features and each of the data cleaning rules respectively; based on each deviation degree, determine the candidate cleaning label; wherein, the deviation degree is used to characterize the degree of deviation between the initial data features and the data cleaning rules, and the deviation degree is positively correlated with the degree of deviation.

[0013] Optionally, the candidate cleaning labels are determined based on each of the aforementioned deviation degrees, including: Determine whether each of the aforementioned deviations is greater than a preset deviation threshold; If so, then the candidate cleaning label is determined to be the retained label representing the retention of the initial training data; If not, proceed with the following steps: The initial data features are input into a pre-trained uncertainty assessment model to obtain a cleaning score generated by the uncertainty assessment model; wherein, the cleaning score is used to characterize the confidence level that the initial training data needs to be cleaned; the cleaning score is positively correlated with the confidence level; Determine whether the cleaning score is greater than the preset score; If it is greater than, then the candidate cleaning label is determined to be the cleaning label; If the value is not greater than the specified value, then the candidate cleaning label is determined to be a pending label that requires manual review.

[0014] Optionally, based on all the candidate cleaning labels, the target cleaning label for the initial training data is determined, including: Determine whether the candidate cleaning labels are consistent; If they match, the candidate cleaning label is taken as the target cleaning label; If there is a discrepancy, the target scenario corresponding to the initial training data is determined from each predefined scenario; the label score of the candidate cleaning label is determined based on the priority of the model corresponding to the candidate cleaning label in the target scenario; and the target cleaning label is determined from each candidate cleaning label based on the label score.

[0015] Optionally, based on the priority of the model corresponding to the candidate cleaning label in the target scenario, the label score of the candidate cleaning label is determined, including: Obtain the preset basic score corresponding to the model; Obtain a pre-constructed mapping relationship; wherein the mapping relationship is the correspondence between the predefined scenario and the preset weight list; the preset weight list includes the preset weights corresponding to the model; the preset weights are used to characterize the priority of the model in the corresponding predefined scenario, and the preset weights are positively correlated with the priority; Extract the target weight list corresponding to the target scene from the mapping relationship; and extract the target weights corresponding to the model from the target weight list. The product of the preset base score and the target weight is used as the tag score.

[0016] This application also provides a training data cleaning apparatus, the apparatus comprising: The dataset acquisition module is used to acquire the initial training dataset to be cleaned; wherein, the initial training dataset is a dataset shared by various models in the autonomous driving system during pre-training, and the functions of each model are different; the initial training dataset consists of initial training data; The rule base invocation module is used to invoke various pre-built cleaning rule bases; wherein, each cleaning rule base corresponds one-to-one with the model; the cleaning rule base is used to store data cleaning rules; The candidate label determination module is used to determine, for each of the cleaning rule bases, the candidate cleaning labels of the initial training data under the cleaning rule base according to each of the data cleaning rules in the cleaning rule base; The target label determination module is used to determine the target cleaning label of the initial training data based on all the candidate cleaning labels; The data processing module is used to process the initial training data according to the target cleaning label until all the initial training data has been processed to obtain the target training dataset.

[0017] This application also provides an electronic device, the device comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of any of the above training data cleaning methods.

[0018] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described training data cleaning methods.

[0019] In summary, this application provides a training data cleaning method, apparatus, device, and medium. A corresponding cleaning rule base is pre-constructed for each model in the autonomous driving system. Using each cleaning rule base, the initial training data in the initial training dataset to be cleaned is differentially arbitrated, i.e., candidate cleaning labels are determined for the initial training data under each cleaning rule base. All candidate cleaning labels are combined to determine the target cleaning label for the initial training data. The initial training data is processed according to the target cleaning label until all initial training data has been processed, resulting in the target training dataset. In the case of multiple models sharing a training dataset, the target training dataset balances the training data requirements of each model in the autonomous driving system, aiming to ensure the overall performance of the autonomous driving system and avoid safety hazards. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a training data cleaning method provided in this application.

[0021] Figure 2 This application provides a flowchart illustrating the process of constructing a cleaning rule base.

[0022] Figure 3 This is a flowchart illustrating a process for determining a target cleaning label, as provided in this application.

[0023] Figure 4 This is a schematic diagram of a training data cleaning device provided in this application.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0027] Please refer to Figure 1 , Figure 1 This application provides a flowchart illustrating a training data cleaning method, which includes: S101. Obtain the initial training dataset to be cleaned; wherein, the initial training dataset is the dataset shared by each model in the autonomous driving system during pre-training, and the functions of each model are different; the initial training dataset consists of initial training data.

[0028] To achieve complex autonomous driving tasks, autonomous driving systems need to be deployed on vehicles. These systems combine onboard sensors and other hardware with machine learning models and other software algorithms to achieve autonomous driving functionality. Autonomous driving systems typically include multiple machine learning models (hereinafter referred to as "models") with different functions, including but not limited to perception models, prediction models, planning models, and comfort models. The perception model is responsible for perceiving targets in the driving environment (such as pedestrians, vehicles, and traffic signs) and generating target perception data. The prediction model is responsible for predicting the future trajectory and intentions of targets based on the target perception data. The planning model is responsible for determining the vehicle's driving strategy and trajectory based on the outputs of the perception and prediction models to achieve autonomous driving. The comfort model is responsible for improving passenger comfort.

[0029] To improve the performance of autonomous driving systems, various models need to be trained. To reduce costs and meet data consistency requirements, the same training dataset is typically used to train all models. The initial training dataset in this application is the dataset shared by all models during training; the initial training dataset consists of initial training data, which can be images. For each training data cleaning process, the training dataset to be cleaned becomes the initial training dataset for that specific task.

[0030] S102. Call the pre-built cleaning rule bases; where each cleaning rule base corresponds one-to-one with a model; the cleaning rule base is used to store data cleaning rules.

[0031] As mentioned earlier, the functions of each model are different, and therefore their requirements for training data also differ. For example, for a perception model, blurry images in the training dataset need to be removed to ensure perception accuracy; however, for a prediction model, blurry images need to be used as difficult learning samples to improve prediction accuracy.

[0032] Based on this, this application pre-builds a dedicated cleaning rule library for each model. The training dataset obtained after cleaning the data using this cleaning rule library can meet the model's training data requirements. The specific process of building the cleaning rule library will be described in subsequent embodiments and will not be repeated here.

[0033] S103. For each cleaning rule base, determine the candidate cleaning labels for the initial training data under the cleaning rule base based on the data cleaning rules in the cleaning rule base.

[0034] It is important to emphasize that for each initial training data point in the initial training dataset, this application utilizes the cleaning rule base corresponding to each model to perform differential arbitration on the initial training data, thereby determining the candidate cleaning labels for the initial training data under each cleaning rule base. The cleaning labels are used to characterize whether the initial training data needs to be cleaned. It is understandable that for the same initial training data, its candidate cleaning labels may differ under different cleaning rule bases.

[0035] For each data cleaning rule set, the initial training data is matched against each data cleaning rule in that set. If the initial training data matches any data cleaning rule, a candidate cleaning label is determined as the label indicating that the initial training data should be cleaned. For example, if the data cleaning rule is "pedestrians are occluded by more than 70% and ambient light intensity is less than 50%", and the initial training data matches this rule, then its candidate cleaning label is determined as the label indicating that the data should be cleaned.

[0036] Finally, by combining all candidate cleaning labels, the target cleaning label for the initial training data is determined. The target cleaning label essentially balances the requirements of all models for the training data. Therefore, the target training dataset obtained by cleaning data using the target cleaning label can balance the requirements of all models for the training data, ensuring the overall performance of the autonomous driving system and reducing the possibility of safety hazards.

[0037] S104. Based on all candidate cleaning labels, determine the target cleaning label for the initial training data.

[0038] After obtaining candidate cleaning labels for the initial training data under various cleaning rule bases, the target cleaning label is determined by combining these candidate labels. The target cleaning label represents the final cleaning opinion for the initial training data. In essence, the target cleaning label integrates the arbitration results of all cleaning rule bases regarding the initial training data, or in other words, it integrates the cleaning opinions of all models on the initial training data, balancing the training data needs of various models in the autonomous driving system.

[0039] In other words, this application does not decide whether to clean the initial training data based on a single cleaning rule or the cleaning opinion given by a single model. Instead, it considers the cleaning opinions given by all models and decides whether to clean the initial training data from the perspective of improving the overall performance of the autonomous driving system.

[0040] This application does not impose specific limitations on the specific implementation of determining the target cleaning label. For example, when all candidate cleaning labels are consistent, the candidate cleaning label is used as the target cleaning label; when the candidate cleaning labels are inconsistent, the target cleaning label is selected from the candidate cleaning labels according to their importance. Subsequent embodiments will provide detailed descriptions, which will not be repeated here.

[0041] S105. Process the initial training data according to the target cleaning label until all initial training data has been processed and the target training dataset is obtained.

[0042] Finally, the initial training data is processed based on the target cleaning label corresponding to the initial training data. That is, when the target cleaning label indicates that the data needs to be cleaned, the data is removed from the initial training dataset; when the target cleaning label indicates that the data needs to be retained, the data does not need to be removed from the initial training dataset; when the target cleaning label indicates that it needs to be manually reviewed, the data is processed according to the results of the manual review.

[0043] After all the initial training data in the initial training dataset has been processed, the target training dataset is obtained. Using the target training dataset to train various models in the autonomous driving system can improve the overall performance of the system and prevent safety hazards.

[0044] In summary, this application provides a training data cleaning method that pre-constructs a corresponding cleaning rule base for each model in an autonomous driving system. Using these rule bases, the initial training data in the initial training dataset to be cleaned undergoes differential arbitration to determine candidate cleaning labels for the initial training data under each rule base. All candidate cleaning labels are then combined to determine the target cleaning label for the initial training data. The initial training data is processed according to the target cleaning label until all initial training data has been processed, resulting in the target training dataset. In the case of multiple models sharing a training dataset, the target training dataset balances the training data requirements of each model in the autonomous driving system, thereby ensuring the overall performance of the autonomous driving system and avoiding safety hazards.

[0045] Based on the above embodiments: Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the process of constructing a cleaning rule base as provided in this application. The process of constructing the cleaning rule base is explained below.

[0046] As an optional implementation, a cleaning rule base is constructed, including: S201. When it is determined that an abnormality has occurred in the autonomous driving system, locate the abnormal model from among the various models; S202. Obtain the historical output results of the anomaly model; and analyze the historical output results to determine the target anomaly type corresponding to the anomaly model from each preset anomaly type; S203. Identify erroneous training data from the initial training dataset; wherein, erroneous training data is the training data that causes the abnormal model to output abnormal data. S204. Determine the common data features of the erroneous training data; and create target data cleaning rules corresponding to the target anomaly type based on the common data features; S205. Based on the target data cleaning rules, construct a cleaning rule library corresponding to the anomaly model.

[0047] In this embodiment, the anomalies of the autonomous driving system are associated with the construction of the cleaning rule base. That is, when an anomaly occurs in the autonomous driving system, the abnormal model is located from each model, and new data cleaning rules are added or existing data cleaning rules are updated in the cleaning rule base corresponding to the abnormal model. This ensures that every adjustment to the cleaning rule base can directly contribute to the performance improvement of the autonomous driving system.

[0048] This embodiment does not specifically limit the method for determining the abnormality of the autonomous driving system.

[0049] As an optional implementation, performance metrics of the autonomous driving system are acquired during the autonomous driving testing phase (including tests conducted in a simulation environment and real-world road tests). These performance metrics include at least one of vehicle-level metrics and model-level metrics. Vehicle-level metrics include at least the number of safety incidents involving the vehicle during the autonomous driving testing phase; these incidents can be collisions or traffic violations, etc. Model-level metrics are used to characterize the performance of each model; for example, the recall rate of the perception model and the average displacement error or final displacement error of the prediction model can be used as model-level metrics. Furthermore, performance metrics may also include communication latency between functional modules in the autonomous driving system and the resource utilization rate of the autonomous driving system.

[0050] Based on this, it is determined whether the above performance indicators meet the corresponding preset conditions; if they do, the autonomous driving system is determined to be abnormal; otherwise, the autonomous driving system is determined to be normal. For example, if the performance indicator includes the number of safety accidents involving the vehicle, the corresponding preset condition can be that the total number of accidents is greater than a preset number; if the performance indicator includes the recall rate of the perception model, the corresponding preset condition can be that the recall rate is lower than a preset threshold. In short, the preset conditions corresponding to each performance indicator can be set according to actual needs, and this embodiment does not impose any special limitations on this.

[0051] Furthermore, the process of locating anomalous models differs depending on the performance metrics mentioned above. Specifically, if the performance metrics are vehicle-level metrics, the system logs of the autonomous driving system can be obtained; the scenario context at the time of the accident and the historical output results of each model can be extracted from the system logs; based on the historical output results of each model, a decision pipeline analysis is performed on each model, and the model with abnormal output is identified as the anomalous model that caused the accident. If the performance metrics are model-level metrics, their actual metrics are determined based on the model's historical output results, and models whose actual metrics do not meet the preset metrics are identified as anomalous models.

[0052] At this point, anomalies in the autonomous driving system have been associated with anomaly models, enabling targeted updates to the cleanup rule base corresponding to the anomaly models. The following section further explains the process of constructing the cleanup rule base for the anomaly models.

[0053] First, for each cleaning rule base, the data cleaning rules stored in the base correspond one-to-one with preset anomaly types. These preset anomaly types are predefined anomaly types. Taking the perception model in an autonomous driving system as an example, its corresponding preset anomaly types can include target missed detections and target false detections.

[0054] Once an anomaly occurs in the autonomous driving system and the anomaly model is located, on the one hand, it is necessary to determine the target anomaly type corresponding to the anomaly model from various preset anomaly types, and on the other hand, it is necessary to determine the erroneous training data that caused the anomaly model to output anomalies, and to construct target data cleaning rules based on the erroneous training data.

[0055] When determining the target anomaly type, the historical output results of the anomaly model are first obtained, and then statistical analysis is performed on the historical output results to determine the target anomaly type from various preset anomaly types. For example, when the anomaly model is a perception model, the historical output results generated by the perception model are compared with the manually determined actual output results to determine whether the target anomaly type corresponds to a missed detection or a false detection.

[0056] When determining the target data cleaning rules, first filter out the erroneous training data that causes the abnormal model output from the initial training dataset, then extract the common features of the erroneous training data, and finally fill the common data features into the pre-built rule template to obtain the target data cleaning rules.

[0057] Specifically, common data features of erroneous training data can be extracted from specified dimensions. For example, for image-type training data, specified dimensions include, but are not limited to, image quality (e.g., blurriness, brightness, and contrast), scene complexity (e.g., edge density and texture complexity), and semantic features. For point cloud-type training data, specified dimensions include, but are not limited to, point cloud density, reflectance distribution, and distance distribution. Of course, annotation quality features (e.g., metrics for characterizing annotation consistency and metrics for characterizing annotation integrity) and temporal consistency metrics of erroneous training data can also be extracted, and these can be set according to requirements.

[0058] Finally, based on the target anomaly type and its corresponding target data cleaning rules, a cleaning rule library corresponding to the anomaly model is constructed. When an existing data cleaning rule corresponding to the target anomaly type exists in the cleaning rule library, the existing data cleaning rule can be adjusted, or it can be directly replaced with the target data cleaning rule. The specific processing method will be explained in subsequent embodiments. When the aforementioned existing data cleaning rule does not exist in the cleaning rule library, the target data cleaning rule can be added to the cleaning rule library.

[0059] In summary, this embodiment achieves a closed-loop association between anomalies in the autonomous driving system and cleaning rules. Each cleaning action executed based on the cleaning rules can directly contribute to the improvement of the performance and safety of the autonomous driving system, greatly increasing the effectiveness of data cleaning.

[0060] The process of identifying erroneous training data in the initial training dataset is explained below.

[0061] As an optional implementation, identifying erroneous training data from the initial training dataset includes: Determine the data characteristics of the initial training data to obtain the initial data characteristics; Obtain a pre-built test dataset; wherein the test dataset consists of test data, and the test data is pre-labeled with real output results; The test data is input into the anomaly model to obtain the prediction output of the anomaly model; Data in each test set whose consistency between the predicted output and the actual output does not meet the preset requirements are considered as erroneous test data; where consistency is used to characterize the degree of consistency between the predicted output and the actual output. Determine the data characteristics of the erroneous test data to obtain key data features; In the initial training dataset, training data whose initial data features are consistent with the key data features are considered as erroneous training data.

[0062] For the aforementioned initial data features, features can be extracted from the initial training data according to the specified dimensions described above to obtain the initial data features. Furthermore, after initially determining the initial data features of the initial training data, the unique identifier corresponding to the initial training data and its initial data features can be stored in a feature database. This allows for quick retrieval and extraction of the initial data features directly from the feature database during subsequent data cleaning, avoiding redundant calculations and improving efficiency.

[0063] For key data features, it is necessary to first identify erroneous test data from the test dataset, and then extract features from the erroneous test data from the specified dimensions to obtain key data features.

[0064] Specifically, a test dataset is pre-constructed, and all test data in the dataset are pre-labeled with real output results. The test data is input into the anomaly model to obtain its predicted output. If the consistency between the predicted output and its corresponding real output does not meet a preset requirement, it indicates a defect in the anomaly model's performance on that test data, and therefore that test data is considered erroneous test data. The specific implementation of determining the key data features of erroneous test data will be explained below.

[0065] Finally, for each initial training data, its initial data features are compared with the key data features; if the two features are consistent, it can be considered that the abnormal model also has defects in the performance of the initial training data. Therefore, the initial training data is used as erroneous training data, which can accurately identify erroneous training data.

[0066] As an optional embodiment, determining the data characteristics of the error test data includes: Obtain the expected areas of interest that have been pre-labeled for the erroneous test data; Determine the heatmap of the anomaly model's attention to the erroneous test data; where the value of a pixel in the heatmap represents the degree of attention the anomaly model pays to that pixel; the value is positively correlated with the degree of attention. Based on the heatmap of interest, the actual areas of interest that meet the preset conditions are extracted from the erroneous test data; where the preset condition is that the pixel value is higher than the preset threshold. Compare the actual area of ​​focus with the expected area of ​​focus to identify the area of ​​deviation; The data characteristics of the deviation area are used as key data characteristics.

[0067] In this embodiment, in addition to labeling the test data with the actual output results, the expected region of interest is also labeled. For example, for a test image containing pedestrians in a nighttime scene, the area where the pedestrians are located is designated as the expected region of interest to determine whether the area actually focused on by the model deviates from the expected region of interest.

[0068] The process of determining the actual area of ​​interest for the anomaly model in erroneous test data is explained below.

[0069] First, a heatmap of the anomaly model's attention to the erroneous test data is determined. The pixel values ​​in this heatmap represent the degree of attention the anomaly model pays to that pixel. A larger pixel value indicates greater attention from the anomaly model, thus contributing more to the model's output. Conversely, a smaller pixel value indicates less attention from the anomaly model, contributing less to the model's output. Therefore, the regions in the heatmap where pixel values ​​exceed a preset threshold are considered the actual areas of attention for the anomaly model to the erroneous test data. This preset threshold can be set according to actual needs; this embodiment does not impose any specific limitations on it.

[0070] As an optional implementation, the aforementioned attention heatmap can be determined using backpropagation saliency analysis or attention mechanism visualization methods. Specifically, for backpropagation saliency analysis, the absolute value of the gradient of the output of the anomalous model with respect to the input is determined, that is, the absolute value of the gradient of the true output of the anomalous model to the erroneous test data with respect to the erroneous test data is determined to obtain the aforementioned attention heatmap. For attention mechanism visualization analysis, the erroneous test data is segmented into image patches; the image patches are input into a pre-constructed Transformer encoder to obtain the attention weight matrix corresponding to the image patch; and an attention heatmap is constructed based on each attention weight matrix.

[0071] Based on this, the expected and actual regions of interest in the erroneous test data are compared to determine the areas of deviation, and the data characteristics of these deviation areas are used as key data features. The deviation areas are the regions that the abnormal model incorrectly focuses on or ignores. The initial training data with the same data characteristics as these deviation areas are precisely the erroneous training data that needs to be added to or removed.

[0072] In summary, this embodiment uses the erroneous test data of the abnormal model in the test dataset as failure cases, extracts the key data features of the failure cases, and uses the key data features to back match the erroneous training data with the same features in the initial training dataset, which can accurately locate the training data that causes the model to output abnormalities.

[0073] The process of building a cleaning rule base is explained in detail below.

[0074] As an optional implementation, based on the target data cleaning rules, a cleaning rule base corresponding to the anomaly model is constructed, including: Determine whether there are existing data cleaning rules in the current cleaning rule base that correspond to the target anomaly type; where the current cleaning rule base is the cleaning rule base corresponding to the anomaly model; If it does not exist, add the target data cleaning rule to the current cleaning rule library; If it exists, then perform the following steps: Based on erroneous training data, the defect types corresponding to existing data cleaning rules are determined; among them, the defect types are insufficient coverage or excessive coverage; insufficient coverage indicates insufficient cleaning of erroneous training data; excessive coverage indicates excessive cleaning of correct training data. Adjust existing data cleaning rules according to the defect type until the specified indicators of the anomaly model meet the requirements; the specified indicators include at least one of recall rate and false positive rate.

[0075] In this embodiment, the data cleaning rules in the cleaning rule base correspond one-to-one with preset anomaly types. After determining the target anomaly type and target data cleaning rule based on the anomaly occurring in the autonomous driving system, it is checked whether there is an existing data cleaning rule in the current cleaning rule base that corresponds to the same target anomaly type. If not, the target data cleaning rule is directly added to the current cleaning rule base. If it exists, the current cleaning rule base is updated by adjusting the existing data cleaning rules.

[0076] The following section focuses on the process of updating the current cleaning rule base by adjusting the existing data cleaning rules, which already exist in the current cleaning rule base.

[0077] It is understandable that the model in the autonomous driving system has been trained using the initial training dataset. If the autonomous driving system subsequently experiences anomalies again, it indicates that the overall performance of the autonomous driving system is still insufficient. Furthermore, it indicates that there is erroneous training data in the initial training dataset. Even further, it indicates that the data cleaning rules in the current cleaning rule base are insufficient, especially the existing data cleaning rules corresponding to the target anomaly type.

[0078] Therefore, this embodiment determines the defect type of existing data cleaning rules based on erroneous training data. The defect type of existing data cleaning rules may be either insufficient coverage or excessive coverage. Insufficient coverage refers to existing data cleaning rules that are too lenient, resulting in insufficient cleaning of erroneous training data when using them for data cleaning. Excessive coverage refers to existing data cleaning rules that are too strict, resulting in a large amount of valid, correct training data being incorrectly cleaned when using them for data cleaning.

[0079] As an optional embodiment, the target data defect that the erroneous training data conforms to is determined from the predefined preset data defects; a pre-constructed mapping relationship is obtained, wherein the mapping relationship is the correspondence between preset data defects and preset defect types; from the mapping relationship, the preset defect type corresponding to the target data defect is extracted as the defect type corresponding to the existing data cleaning rules.

[0080] For example, the presupposed data defects include low labeling quality and training data scenarios that are rare. Labeling quality issues usually correspond to insufficient cleaning; therefore, in the mapping relationship, a correspondence is established between low labeling quality and insufficient coverage. Rare scenarios usually correspond to excessive cleaning; therefore, in the mapping relationship, a correspondence is established between rare scenarios and excessive coverage. If the erroneous training data meets the data defect of low labeling quality, it can be determined that the existing data cleaning rules correspond to insufficient coverage, and the existing data cleaning rules need to be adjusted to be more stringent.

[0081] After identifying the aforementioned defect types, the existing data cleaning rules are adjusted to update the current cleaning rule base, with the goal of meeting the specified indicators of the anomaly model. The method of adjusting the rules will differ depending on the form of the existing data cleaning rules, but as long as the goal is to meet the specified indicators of the anomaly model, the specific adjustment method of the rules in this embodiment is not particularly limited.

[0082] For example, the existing data cleaning rule is "the degree of pedestrian occlusion is >70%", but the common data feature of the erroneous training data is "the degree of pedestrian occlusion is between 60% and 70%".

[0083] If the defect type is insufficient coverage, the cleaning rules need to be relaxed. Specifically, the degree of pedestrian occlusion can be used as the decision variable, with [60%, 70%] as the decision interval. Using a Bayesian optimization algorithm, the thresholds corresponding to the decision variables are gradually reduced until the recall rate of the anomaly model reaches the target recall rate and the false positive rate is controllable. This threshold is then used as the target threshold. Afterward, the thresholds in the existing data cleaning rules are updated to the target threshold, resulting in the updated data cleaning rules.

[0084] If the defect type is over-coverage, the cleaning rules need to be tightened. Specifically, a reinforcement learning reward mechanism can be used to gradually increase the threshold corresponding to the decision variables, thereby narrowing the scope of the cleaning rules.

[0085] In addition, all adjustments to the above data cleaning rules can be recorded in the log, supporting rollback and auditing.

[0086] In summary, this embodiment achieves automated updates to the data cleaning rule base. It can accurately analyze the defects of existing data cleaning rules and, with the goal of improving the performance of the anomaly model, differentiate and adjust existing data cleaning rules to ensure that the optimization of cleaning rules serves to improve model performance.

[0087] The process of determining candidate clean labels for the initial training data is explained below.

[0088] As an optional implementation, candidate cleaning labels for the initial training data are determined based on the data cleaning rules in the cleaning rule base, including: Determine the data characteristics of the initial training data to obtain the initial data characteristics; The initial data features are matched one by one with each data cleaning rule to determine whether the initial data features meet any of the data cleaning rules. If so, then the candidate cleaning label is determined to be the cleaning label that represents the initial training data being cleaned; If not, then determine the deviation between the initial data features and each data cleaning rule; based on each deviation, determine the candidate cleaning labels; wherein, the deviation is used to characterize the degree of deviation between the initial data features and the data cleaning rules, and the deviation is positively correlated with the degree of deviation.

[0089] In this embodiment, the data cleaning rule uses data features as the judgment object, specifying the conditions that the judgment object must meet; its expression can be the magnitude relationship between data features and preset feature values, etc. For example, the data cleaning rule is "the target is a pedestrian, the pedestrian is occluded by more than 70% and the ambient light intensity is less than 50 lux". Based on this, it is first necessary to determine the initial data features of the initial training data. Specifically, features of the initial training data can be extracted from the specified dimensions mentioned above to obtain the initial data features; correspondingly, the data cleaning rule also sets judgment conditions for the data features of the specified dimensions.

[0090] Next, the initial data features are matched one by one with the data cleaning rules in the cleaning rule base. If an initial data feature is found to match any data cleaning rule, it is determined that the initial training data corresponding to that initial data feature needs to be cleaned, and thus a candidate cleaning label is determined as the cleaning label.

[0091] If the initial data features do not conform to all data cleaning rules, this embodiment does not directly set the candidate cleaning labels as the retained labels. Instead, it determines the deviation between the initial data features and each data cleaning rule, and then determines the final candidate cleaning labels based on each deviation.

[0092] To determine the degree of deviation between the initial data features and the data cleaning rules, a tolerance range can be preset. Specifically, if the data cleaning rule is the relationship between the data features and preset feature values, a tolerance range corresponding to the preset feature value can be set. If the initial data features fall within this tolerance range, the deviation between the initial data features and the data cleaning rules is determined to be a first preset deviation, which is less than a preset deviation threshold, indicating that the deviation between the initial data features and the data cleaning rules is small. If the initial data features exceed the tolerance range, the deviation is determined to be a second preset deviation, which is greater than the preset deviation threshold, indicating that the deviation between the initial data features and the data cleaning rules is large.

[0093] Furthermore, for data cleaning rules that contain multiple sub-rules, a pre-set limit on the number of allowed conflicts can be set. If the number of sub-rules that do not conform to the initial data characteristics is less than the allowed number of conflicts, the deviation is set to the first preset deviation; otherwise, the deviation is set to the second preset deviation.

[0094] For example, the data cleaning rule is "the target is a pedestrian, the pedestrian is occluded by more than 70% and the ambient light intensity is less than 50 lux". This data cleaning rule actually includes the following three rules: the target is a pedestrian (hereinafter referred to as the first rule), the pedestrian is occluded by more than 70% (hereinafter referred to as the second rule), and the ambient light intensity is less than 50 lux (hereinafter referred to as the third rule). As mentioned above, the permissible deviation range for the pedestrian occlusion level can be preset as [65%, 70%]; the permissible deviation range for the ambient light intensity can be preset as [50, 55]; and the permissible number of conflict entries can be preset as 1. Based on each permissible deviation range and the permissible number of conflict entries, the deviation between the initial data features and the data cleaning rule is determined.

[0095] As an optional embodiment, candidate cleaning labels are determined based on various deviation degrees, including: Determine whether each deviation is greater than the preset deviation threshold; If so, then the candidate cleaning label is determined as the retained label representing the preservation of the initial training data; If not, proceed with the following steps: The initial data features are input into a pre-trained uncertainty assessment model to obtain a cleaning score generated by the uncertainty assessment model; the cleaning score is used to characterize the confidence level that the initial training data needs to be cleaned; the cleaning score is positively correlated with the confidence level. Determine if the cleaning score is greater than the preset score; If it is greater than 1, then the candidate cleaning label is determined to be a cleaning label; If the value is not greater than the specified value, then the candidate cleaning label is determined as a pending label that requires manual review.

[0096] In this embodiment, if the deviation between the initial data feature and all data cleaning rules is greater than the preset deviation threshold, it is determined that the initial data feature does not conform to each data cleaning rule and the deviation is large. Therefore, it is determined that the initial training data corresponding to the initial data feature does not need to be cleaned, that is, the candidate cleaning label is set as the retention label for retaining the initial training data.

[0097] For cases where not all deviations exceed the preset deviation threshold—that is, where there is ambiguity in the matching between the initial data features and some cleaning rules, or where the initial data features only conflict with some cleaning rules—a pre-trained lightweight uncertainty assessment model is used to determine candidate cleaning labels. The uncertainty assessment model takes the initial data features as input and outputs their corresponding cleaning scores. The cleaning score characterizes the confidence level that the initial training data needs to be cleaned, and the cleaning score is positively correlated with the confidence level. If the cleaning score is greater than the preset score, the candidate cleaning label is determined as a cleaning label; otherwise, the candidate cleaning label is determined as a pending label indicating that manual review is required. It should be noted that the aforementioned preset score is pre-set, and this embodiment does not specifically limit its value.

[0098] In summary, this embodiment provides a dual label determination mechanism based on rules and models. First, candidate cleaning labels corresponding to the initial training data are determined based on the data cleaning rules themselves. For cases where rule matching is ambiguous or partially conflicting, an uncertainty assessment model is then used to determine candidate cleaning labels to ensure the reliability of the candidate cleaning labels.

[0099] Furthermore, for each cleaning rule base, the candidate cleaning labels for the initial training data under that rule base need to be determined according to the above process. To improve efficiency, an independent arbitration instance can be created in advance for each cleaning rule base. The core of the arbitration instance is a decision system based on a high-performance rule engine (such as Drools), while also integrating a lightweight machine learning model (such as a small random forest or XGBoost model) to complete uncertain decision-making, forming a hybrid decision architecture with "rules as the main approach and models as the auxiliary approach".

[0100] Please refer to Figure 3 , Figure 3 This application provides a flowchart for determining target cleaning labels. The process of determining target cleaning labels for initial training data is described below.

[0101] As an optional implementation, the target cleaning label for the initial training data is determined based on all candidate cleaning labels, including: Determine if the cleaning labels for each candidate are consistent; If they match, the candidate cleaning label will be used as the target cleaning label. If there is a discrepancy, the target scenario corresponding to the initial training data is determined from each predefined scenario; the label score of the candidate cleaning label is determined based on the priority of the model corresponding to the candidate cleaning label in the target scenario; and the target cleaning label is determined from each candidate cleaning label based on the label score.

[0102] In this embodiment, it is first determined whether the candidate cleaning labels are consistent. If the candidate cleaning labels are consistent, the candidate cleaning label is directly used as the target cleaning label. If the candidate cleaning labels are inconsistent, the target cleaning label needs to be selected from the candidate cleaning labels. The process of determining the target cleaning label when the candidate cleaning labels are inconsistent is described below.

[0103] The predefined scenarios mentioned above include, but are not limited to, highway cruising, urban congestion, nighttime rain, and tunnels / low light conditions, etc., which can be set according to actual needs. Within the same scenario, the priority of each model differs. For the same model, its priority may also differ in different scenarios. That is, the priority of each model is not fixed but dynamically changes according to the scenario. The priority of a model in a particular scenario characterizes the degree of its impact on the overall performance of the autonomous driving system in that scenario. For the same model, the higher its priority, the higher its candidate cleaning label score. Therefore, when determining the label score of candidate cleaning labels, the impact of the training data's corresponding scenario on the model priority needs to be fully considered.

[0104] Based on this, this embodiment first determines the target scene corresponding to the initial training data. Then, based on the model's priority in the target scene, the label score of the candidate cleaning label corresponding to the model is determined. When the model's priority is higher, the label score of its corresponding candidate cleaning label is higher, and the candidate cleaning label is more likely to be identified as the target cleaning label.

[0105] The following describes the process of determining the label scoring for candidate cleaning labels.

[0106] As an optional implementation, the label score of the candidate cleaning label is determined based on the priority of the model corresponding to the candidate cleaning label in the target scenario, including: Obtain the preset base score corresponding to the model; Obtain the pre-built mapping relationship; where the mapping relationship is the correspondence between predefined scenarios and preset weight list; the preset weight list includes the preset weights corresponding to the model; the preset weights are used to characterize the priority of the model in the corresponding predefined scenario, and the preset weights are positively correlated with the priority; Extract the target weight list corresponding to the target scene from the mapping relationship; and extract the target weights corresponding to the model from the target weight list. The product of the preset base score and the target weight is used as the tag score.

[0107] In this embodiment, a corresponding preset basic score is set for each model in advance. The preset basic score is positively correlated with the basic priority of the model, and the basic priority is independent of the scene.

[0108] Taking an autonomous driving system comprising a perception model, a prediction model, a planning model, and a comfort model as an example, the priority of the perception model and the planning model is predefined as P0, the priority of the prediction model is P1, and the priority of the comfort model is P2; the basic priority decreases sequentially from P0 to P2. Based on the basic priority, a preset basic score is set for each model. For example, the preset basic score for the perception model and the planning model is 3, the preset basic score for the prediction model is 2, and the preset basic score for the comfort model is 1.

[0109] The priority of the model in the target scenario is reflected by the preset weights corresponding to the model. Specifically, this embodiment pre-constructs a mapping relationship; wherein, the mapping relationship is the correspondence between predefined scenarios and a preset weight list; the preset weight list includes the preset weights corresponding to each model. Taking the predefined scenarios including highway cruising, urban congestion, nighttime rainy days, and tunnels / low light, and the autonomous driving system including perception models, prediction models, and planning models as an example, the mapping relationship can be referred to Table 1 below.

[0110] Table 1. Mapping Relationship Diagram Taking high-speed cruising as the target scenario as an example, the pre-set weights for the planning model are the highest, followed by the perception model, and the pre-set weights for the prediction model are the lowest. Therefore, for the initial training data corresponding to the high-speed cruising scenario, the candidate cleaning labels given by the planning model are more likely to be selected as the target cleaning labels. In other words, when determining whether to clean the initial training data, the focus is more on the data requirements of the planning model.

[0111] Based on this, a list of target weights corresponding to the target scene is extracted from the mapping relationship. For each model, a preset weight corresponding to that model is extracted from the target weight list as the target weight; the product of the target weight and the preset base score corresponding to that model is used as the label score corresponding to that model.

[0112] At this point, by combining the basic priorities of each model and the dynamic priorities of each model in the target scenario, the label scores of the candidate cleaning labels corresponding to each model are determined. The following explains the process of determining the target cleaning label from the candidate cleaning labels based on the label scores.

[0113] Specifically, first, the sum of all tag scores is determined to obtain the total score. Then, for each tag score, the proportion of that tag score to the total score is determined. It is then determined whether any tag score has a proportion greater than a preset proportion. If so, the candidate cleansing tag corresponding to that score is selected as the target cleansing tag. If not, the score with the highest corresponding proportion among all tag scores is selected as the target score, and the candidate cleansing tag corresponding to the target score is selected as the target cleansing tag. The preset proportion can be set according to actual needs, for example, to 60%.

[0114] Of course, a veto right can also be provided based on the basic priority of each model. For example, if it is determined that any P0-level model provides a clean label (that is, a label representing the initial training data that needs to be cleaned), and its label score accounts for more than 30% of the total score, then the target clean label is also forcibly determined to be a clean label to ensure a safety baseline.

[0115] The following example, taking a safety incident caused by missed pedestrian detection in an urban nighttime scene, illustrates in detail the processing procedure of the training data cleaning method provided in this application.

[0116] During nighttime testing of the vehicle, the module monitoring the autonomous driving system detected a pedestrian sudden appearance, which was recorded as an anomaly in the autonomous driving system. Subsequently, the system logs of the autonomous driving system were retrieved, and the historical output results of each model in the system were extracted. Analysis revealed that the perception model in the system did not report the presence of a pedestrian in the current driving environment before the pedestrian sudden appearance, and the confidence level of the perception model in detecting the pedestrian remained below a preset threshold before and after the incident. Therefore, the perception model was determined to be an abnormal model.

[0117] By using backpropagation saliency analysis and attention visualization techniques, the region where the perception model failed to accurately focus on the pedestrian was identified. Therefore, the pedestrian region was designated as the bias region, and the data characteristics of this bias region were identified as key data features. Backtracking analysis was performed on the initial training dataset, and the initial training data with key data features was identified as erroneous training data. Common data features of the erroneous training data were extracted from specified dimensions, identifying them as: "target is a pedestrian, pedestrian occlusion is between 30% and 70%, ambient light intensity is less than 50 lux, and the scene is a nighttime rainy day."

[0118] Based on this, the following data cleaning rules are created in the cleaning rule base corresponding to the perception model: conditions: -time_of_day: night / / Nighttime scene; -occlusion_level: [0.3, 0.7] / / Pedestrian occlusion is between 30% and 70%; -illumination: <50 lux / / Ambient light intensity is less than 50 lux; -weather: [rainy] / / The weather is rainy; action: human_review / / Candidate cleaning labels are pending labels.

[0119] When cleaning the initial training dataset, the initial data features are first obtained. These features include the scene corresponding to the training data, the occlusion ratio of the target, and the ambient light intensity. The initial training data is then arbitrated using the cleaning rule bases corresponding to the perception model, prediction model, and planning model. Candidate cleaning labels for the initial training data under the cleaning rule base corresponding to the perception model are designated as pending labels, candidate cleaning labels under the cleaning rule base corresponding to the prediction model are designated as retained labels, and candidate cleaning labels under the cleaning rule base corresponding to the planning model are designated as pending labels.

[0120] Since the candidate cleaning labels are inconsistent, the label scores for each model's candidate cleaning labels are determined based on the model's basic priority and the preset weights for each model in rainy nighttime conditions. Calculations show that the label score for the undetermined label corresponding to the perception model is 4.5, the label score for the retained label corresponding to the prediction model is 2.2, and the label score for the undetermined label corresponding to the planning model is 3.9. Therefore, the label score for the undetermined label is 8.4, the label score for the retained label is 2.2, and the sum of all label scores is 10.6. Following the judgment logic of the aforementioned embodiment, the target cleaning label is determined to be an undetermined label. Therefore, a manual review is conducted to decide whether to clean the initial training data.

[0121] Please refer to Figure 4 , Figure 4 A schematic diagram of a training data cleaning apparatus provided in this application. This application also provides a training data cleaning apparatus, comprising: The dataset acquisition module 401 is used to acquire the initial training dataset to be cleaned; wherein, the initial training dataset is a dataset shared by various models in the autonomous driving system during pre-training, and the functions of each model are different; the initial training dataset consists of initial training data; The rule base invocation module 402 is used to invoke various pre-built cleaning rule bases; each cleaning rule base corresponds one-to-one with a model; the cleaning rule base is used to store data cleaning rules. The candidate label determination module 403 is used to determine the candidate cleaning labels of the initial training data under the cleaning rule base for each cleaning rule base based on the data cleaning rules in the cleaning rule base. The target label determination module 404 is used to determine the target cleaning label of the initial training data based on all candidate cleaning labels. The data processing module 405 is used to process the initial training data according to the target cleaning label until all the initial training data has been processed and the target training dataset is obtained.

[0122] For a detailed description of the training data cleaning apparatus provided in this application, please refer to the embodiments of the training data cleaning method described above; further details will not be repeated here.

[0123] Based on the above embodiments: As an optional embodiment, the device further includes: The anomaly model localization module is used to locate the anomaly model from among the various models when an anomaly is determined to occur in the autonomous driving system. The anomaly type determination module is used to obtain the historical output results of the anomaly model and analyze the historical output results to determine the target anomaly type corresponding to the anomaly model from each preset anomaly type. The erroneous training data determination module is used to determine erroneous training data from the initial training dataset; wherein, erroneous training data is the training data that causes the abnormal model to output abnormalities. The cleaning rule creation module is used to determine the common data features of erroneous training data; and to create target data cleaning rules corresponding to the target anomaly type based on the common data features; The rule base construction module is used to build a cleaning rule base corresponding to the anomaly model based on the target data cleaning rules.

[0124] As an optional embodiment, the error training data determination module includes: The initial feature determination module is used to determine the data features of the initial training data and obtain the initial data features. The test set acquisition module is used to acquire a pre-built test dataset; the test dataset consists of test data, and the test data is pre-labeled with real output results; The prediction result determination module is used to input test data into the anomaly model and obtain the prediction output result of the anomaly model; The error test data determination module is used to identify data in each test data set where the consistency between the predicted output and the actual output does not meet preset requirements as error test data; where consistency is used to characterize the degree of consistency between the predicted output and the actual output. The key feature determination module is used to determine the data characteristics of the erroneous test data and obtain key data features. The Error Data Determination Submodule is used to identify training data in the initial training dataset whose initial data features are consistent with the key data features as error training data.

[0125] As an optional embodiment, the key feature determination module includes: The expected area of ​​interest acquisition module is used to acquire the expected areas of interest that have been pre-labeled for the erroneous test data; The heatmap determination module is used to determine the heatmap of the abnormal model's attention to the erroneous test data; the value of the pixel in the attention heatmap is used to characterize the degree of attention the abnormal model pays to the pixel; the value is positively correlated with the degree of attention. The actual area of ​​interest determination module is used to extract actual areas of interest that meet preset conditions from erroneous test data based on the heatmap of interest; wherein the preset condition is that the pixel value is higher than a preset threshold. The deviation area determination module is used to compare the actual area of ​​concern with the expected area of ​​concern to determine the deviation area; The key feature determination submodule is used to identify the data features of the deviation region as key data features.

[0126] As an optional implementation, the rule base construction module includes: The first determination module is used to determine whether there are existing data cleaning rules in the current cleaning rule base that correspond to the target anomaly type; wherein, the current cleaning rule base is the cleaning rule base corresponding to the anomaly model; if it does not exist, the rule addition module is triggered; if it exists, the rule adjustment module is triggered. The rule addition module is used to add target data cleaning rules to the current cleaning rule library; The rule adjustment module is used to determine the defect type of the existing data cleaning rules based on the erroneous training data. The defect type is either under-coverage or over-coverage. Under-coverage indicates insufficient cleaning of erroneous training data, while over-coverage indicates excessive cleaning of correct training data. The existing data cleaning rules are adjusted according to the defect type until the specified indicators of the abnormal model meet the requirements. The specified indicators include at least one of recall rate and false positive rate.

[0127] As an optional embodiment, the candidate label determination module 403 includes: The initial feature determination module is used to determine the data features of the initial training data and obtain the initial data features. The second determination module is used to match the initial data features with each data cleaning rule one by one to determine whether the initial data features meet any data cleaning rule; if yes, the first label determination module is triggered; if no, the second label determination module is triggered. The first label determination module is used to determine the candidate cleaning labels as cleaning labels that represent the initial training data being cleaned. The second label determination module is used to determine the deviation between the initial data features and each data cleaning rule; based on each deviation, candidate cleaning labels are determined; wherein, the deviation is used to characterize the degree of deviation between the initial data features and the data cleaning rules, and the deviation is positively correlated with the degree of deviation.

[0128] As an optional embodiment, the second label determination module includes: The deviation determination module is used to determine the deviation between the initial data features and each data cleaning rule; The third determination module is used to determine whether each deviation degree is greater than the preset deviation threshold; if so, the third label determination module is triggered; if not, the cleaning scoring module is triggered. The third label determination module is used to determine the candidate cleaned labels as retained labels that represent the preservation of the initial training data; The cleaning score module is used to input the initial data features into a pre-trained uncertainty assessment model to obtain a cleaning score generated by the uncertainty assessment model; determine whether the cleaning score is greater than a preset score; if it is greater, determine the candidate cleaning label as a cleaning label; if it is not greater, determine the candidate cleaning label as a pending label that needs to be manually reviewed; wherein, the cleaning score is used to represent the confidence level that the initial training data needs to be cleaned; the cleaning score is positively correlated with the confidence level.

[0129] As an optional embodiment, the target label determination module 404 includes: The fourth determination module is used to determine whether the candidate cleaning labels are consistent; if they are consistent, the first target label determination module is triggered; if they are inconsistent, the second target label determination module is triggered. The first target label determination module is used to use candidate cleaning labels as target cleaning labels; The second target label determination module is used to determine the target scene corresponding to the initial training data from each predefined scene; determine the label score of the candidate cleaning label based on the priority of the model corresponding to the candidate cleaning label in the target scene; and determine the target cleaning label from each candidate cleaning label based on the label score.

[0130] As an optional embodiment, the second target label determination module includes: The target scene determination module is used to determine the target scene corresponding to the initial training data from each predefined scene; The basic score acquisition module is used to obtain the preset basic score corresponding to the model; The mapping relationship acquisition module is used to acquire pre-built mapping relationships; wherein, the mapping relationship is the correspondence between predefined scenarios and preset weight lists; the preset weight list includes preset weights corresponding to the model; the preset weights are used to characterize the priority of the model in the corresponding predefined scenario, and the preset weights are positively correlated with the priority; The target weight extraction module is used to extract a list of target weights corresponding to the target scene from the mapping relationship; and to extract the target weights corresponding to the model from the list of target weights. The tag scoring determination module is used to take the product of the preset base score and the target weight as the tag score; The target label determination submodule is used to determine the target cleaning label from each candidate cleaning label based on the label score.

[0131] Please refer to Figure 5 , Figure 5 This application provides a schematic diagram of the structure of an electronic device. This application also provides an electronic device, which includes: Memory 501 is used to store computer programs; The processor 502 is used to implement the steps of any of the training data cleaning methods described above when executing a computer program.

[0132] The aforementioned memory 501 includes all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0133] In some embodiments, the electronic device may further include a display screen, input / output interfaces, communication interfaces, a power supply, and a communication bus, etc. Those skilled in the art will understand that... Figure 5 The structures shown do not constitute a limitation on the electronic device and may include more components than those shown.

[0134] For a detailed description of the electronic device provided in this application, please refer to the embodiments of the training data cleaning method described above; further details will not be repeated here.

[0135] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described training data cleaning methods.

[0136] For a detailed description of the storage medium provided in this application, please refer to the embodiments of the training data cleaning method described above; this application will not repeat the details here.

[0137] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0138] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0139] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A training data cleaning method, characterized in that, The method includes: Obtain the initial training dataset to be cleaned; wherein, the initial training dataset is a dataset shared by various models in the autonomous driving system during pre-training, and the functions of each model are different; the initial training dataset consists of initial training data; The pre-built cleaning rule libraries are invoked; wherein, each cleaning rule library corresponds one-to-one with the model; the cleaning rule library is used to store data cleaning rules. For each of the cleaning rule bases, candidate cleaning labels for the initial training data under the cleaning rule bases are determined according to each of the data cleaning rules in the cleaning rule bases. Based on all the candidate cleaning labels, determine the target cleaning label for the initial training data; The initial training data is processed according to the target cleaning label until all the initial training data has been processed to obtain the target training dataset.

2. The training data cleaning method as described in claim 1, characterized in that, Constructing the cleaning rule base includes: When it is determined that the autonomous driving system has malfunctioned, the abnormal model is located from among the models. Obtain the historical output results of the anomaly model; and analyze the historical output results to determine the target anomaly type corresponding to the anomaly model from each preset anomaly type; From the initial training dataset, erroneous training data is determined; wherein, the erroneous training data is the training data that causes the abnormal model to output abnormalities; Determine the common data features of the erroneous training data; and create target data cleaning rules corresponding to the target anomaly type based on the common data features. Based on the target data cleaning rules, a cleaning rule library corresponding to the anomaly model is constructed.

3. The training data cleaning method as described in claim 2, characterized in that, From the initial training dataset, identify erroneous training data, including: Determine the data features of the initial training data to obtain the initial data features; Obtain a pre-constructed test dataset; wherein the test dataset consists of test data, and the test data is pre-labeled with real output results; The test data is input into the anomaly model to obtain the prediction output of the anomaly model; Data in the test data in which the consistency between the predicted output and the actual output does not meet the preset requirements are designated as erroneous test data; wherein, the consistency is used to characterize the degree of consistency between the predicted output and the actual output. Determine the data characteristics of the erroneous test data to obtain key data characteristics; The training data in the initial training dataset whose initial data features are consistent with the key data features are used as the erroneous training data.

4. The training data cleaning method as described in claim 3, characterized in that, Determining the data characteristics of the erroneous test data includes: Obtain the intended areas of interest that have been pre-annotated for the error test data; A heatmap of the anomaly model's attention to the erroneous test data is determined; wherein, the value of a pixel in the heatmap is used to characterize the degree of attention the anomaly model pays to that pixel; the value is positively correlated with the degree of attention. Based on the heatmap of interest, actual areas of interest that meet preset conditions are extracted from the error test data; wherein, the preset condition is that the pixel value is higher than a preset threshold. By comparing the actual area of ​​interest with the expected area of ​​interest, the deviation area is determined; The data characteristics of the deviation region are used as the key data characteristics.

5. The training data cleaning method as described in claim 2, characterized in that, Based on the target data cleaning rules, a cleaning rule library corresponding to the anomaly model is constructed, including: Determine whether there are existing data cleaning rules in the current cleaning rule base that correspond to the target anomaly type; wherein, the current cleaning rule base is the cleaning rule base corresponding to the anomaly model; If it does not exist, the target data cleaning rule is added to the current cleaning rule library; If it exists, then perform the following steps: Based on the erroneous training data, the defect type corresponding to the existing data cleaning rules is determined; wherein, the defect type is either insufficient coverage or excessive coverage; the insufficient coverage type indicates insufficient cleaning of the erroneous training data; the excessive coverage type indicates excessive cleaning of the correct training data. Based on the defect type, adjust the existing data cleaning rules until the specified indicators of the anomaly model meet the requirements; wherein the specified indicators include at least one of recall rate and false positive rate.

6. The training data cleaning method as described in claim 1, characterized in that, Based on the data cleaning rules in the cleaning rule base, candidate cleaning labels for the initial training data under the cleaning rule base are determined, including: Determine the data features of the initial training data to obtain the initial data features; The initial data features are matched one by one with each of the data cleaning rules to determine whether the initial data features conform to any of the data cleaning rules. If so, then the candidate cleaning label is determined to be a cleaning label that represents cleaning the initial training data; If not, then determine the deviation degree between the initial data features and each of the data cleaning rules respectively; based on each deviation degree, determine the candidate cleaning label; wherein, the deviation degree is used to characterize the degree of deviation between the initial data features and the data cleaning rules, and the deviation degree is positively correlated with the degree of deviation.

7. The training data cleaning method as described in claim 6, characterized in that, Based on each of the aforementioned deviation degrees, the candidate cleaning labels are determined, including: Determine whether each of the aforementioned deviations is greater than a preset deviation threshold; If so, then the candidate cleaning label is determined to be the retained label representing the retention of the initial training data; If not, proceed with the following steps: The initial data features are input into a pre-trained uncertainty assessment model to obtain a cleaning score generated by the uncertainty assessment model; wherein, the cleaning score is used to characterize the confidence level that the initial training data needs to be cleaned; the cleaning score is positively correlated with the confidence level; Determine whether the cleaning score is greater than a preset score; If it is greater than, then the candidate cleaning label is determined to be the cleaning label; If the value is not greater than the specified value, then the candidate cleaning label is determined to be a pending label that requires manual review.

8. The training data cleaning method as described in claim 1, characterized in that, Based on all the candidate cleaning labels, the target cleaning label for the initial training data is determined, including: Determine whether the candidate cleaning labels are consistent; If they match, the candidate cleaning label is taken as the target cleaning label; If there is a discrepancy, the target scenario corresponding to the initial training data is determined from each predefined scenario; the label score of the candidate cleaning label is determined based on the priority of the model corresponding to the candidate cleaning label in the target scenario; and the target cleaning label is determined from each candidate cleaning label based on the label score.

9. The training data cleaning method as described in claim 8, characterized in that, Based on the priority of the model corresponding to the candidate cleaning label in the target scenario, the label score of the candidate cleaning label is determined, including: Obtain the preset basic score corresponding to the model; Obtain a pre-constructed mapping relationship; wherein the mapping relationship is the correspondence between the predefined scenario and the preset weight list; the preset weight list includes the preset weights corresponding to the model; the preset weights are used to characterize the priority of the model in the corresponding predefined scenario, and the preset weights are positively correlated with the priority; Extract the target weight list corresponding to the target scene from the mapping relationship; and extract the target weights corresponding to the model from the target weight list. The product of the preset base score and the target weight is used as the tag score.

10. A training data cleaning apparatus, characterized in that, The device includes: The dataset acquisition module is used to acquire the initial training dataset to be cleaned; wherein, the initial training dataset is a dataset shared by various models in the autonomous driving system during pre-training, and the functions of each model are different; the initial training dataset consists of initial training data; The rule base invocation module is used to invoke various pre-built cleaning rule bases; wherein, each cleaning rule base corresponds one-to-one with the model; the cleaning rule base is used to store data cleaning rules; The candidate label determination module is used to determine, for each of the cleaning rule bases, the candidate cleaning labels of the initial training data under the cleaning rule base according to each of the data cleaning rules in the cleaning rule base; The target label determination module is used to determine the target cleaning label of the initial training data based on all the candidate cleaning labels; The data processing module is used to process the initial training data according to the target cleaning label until all the initial training data has been processed to obtain the target training dataset.

11. An electronic device, characterized in that, The device includes: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the training data cleaning method as described in any one of claims 1 to 9.

12. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the training data cleaning method as described in any one of claims 1 to 9.

Images