End-to-end model training data anomaly detection method, device and electronic equipment

Abstract

The present disclosure provides an end-to-end model training data anomaly detection method and device and electronic equipment, relating to the technical fields of automatic driving, large models, model training, anomaly detection, etc. The specific implementation scheme is: obtaining a loss type corresponding to a training task in each training step in an end-to-end model training process; performing the training task based on multiple frames of training data included in the training step, and determining a loss value corresponding to each frame of training data based on the loss type; determining candidate abnormal data existing in the multiple frames of training data based on the loss value; and performing anomaly detection on the candidate abnormal data based on a data attribute of the candidate abnormal data to determine target abnormal data in the training step.

Description

Technical Field

[0001] This disclosure relates to the technical fields of autonomous driving, large models, model training, and anomaly detection, and in particular to an anomaly detection method, apparatus, and electronic device for end-to-end model training data. Background Technology

[0002] In end-to-end model training, the quality of training data directly affects the model's performance and security. End-to-end models typically employ a multi-task joint training approach, simultaneously optimizing the loss functions for multiple sub-tasks such as perception, prediction, and planning.

[0003] Current training processes use large-scale training data. When abnormal data such as labeling errors, sensor failures, and extreme scenarios are mixed into the large-scale training data, the model has difficulty detecting abnormal data during training due to the large scale of the training data, and the efficiency of problem investigation is also low. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for anomaly detection in end-to-end model training data.

[0005] According to one aspect of this disclosure, an anomaly detection method for end-to-end model training data is provided, comprising: obtaining the loss type corresponding to the training task in each training step during end-to-end model training; executing the training task based on multiple frames of training data included in the training step, and determining the loss value corresponding to each frame of training data based on the loss type; determining candidate anomaly data with anomalies from the multiple frames of training data based on the loss value; and performing anomaly detection on the candidate anomaly data based on the data attributes of the candidate anomaly data to determine the target anomaly data in the training step.

[0006] According to another aspect of this disclosure, an anomaly detection device for end-to-end model training data is provided, comprising: an acquisition module, configured to acquire the loss type corresponding to the training task in each training step during end-to-end model training; a first determination module, configured to execute the training task based on multiple frames of training data included in the training step, and determine the loss value corresponding to each frame of training data based on the loss type; a second determination module, configured to determine candidate abnormal data with anomalies from the multiple frames of training data based on the loss value; and an anomaly detection module, configured to perform anomaly detection on the candidate abnormal data based on the data attributes of the candidate abnormal data, so as to determine the target abnormal data in the training step.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the anomaly detection method for end-to-end model training data as described in one aspect of the above embodiment.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein computer programs / instructions are stored thereon, the computer instructions being used to cause the computer to execute the anomaly detection method for end-to-end model training data described in one aspect of the above-described embodiment.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the anomaly detection method for end-to-end model training data as described in one aspect of the above-described embodiment.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 A flowchart illustrating an anomaly detection method for end-to-end model training data provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating another method for anomaly detection in end-to-end model training data provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating another method for anomaly detection in end-to-end model training data provided in this embodiment of the disclosure; Figure 4 A flowchart illustrating another method for anomaly detection in end-to-end model training data provided in this embodiment of the disclosure; Figure 5 This is a flowchart illustrating the process of filtering candidate abnormal data provided in an embodiment of this disclosure; Figure 6 A schematic diagram of the flow of detection training data provided in the embodiments of this disclosure; Figure 7 A flowchart illustrating another method for anomaly detection in end-to-end model training data provided in this embodiment of the disclosure; Figure 8This is a schematic diagram of the structure of an anomaly detection system for end-to-end model training data provided in an embodiment of this disclosure; Figure 9 A schematic diagram of the structure of an anomaly detection device for end-to-end model training data provided in this embodiment of the disclosure; Figure 10 This is a block diagram of an electronic device used to implement the anomaly detection method for end-to-end model training data according to embodiments of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] The following description, with reference to the accompanying drawings, outlines an anomaly detection method, apparatus, electronic device, and storage medium for end-to-end model training data according to embodiments of the present disclosure.

[0015] Figure 1 This is a flowchart illustrating an anomaly detection method for end-to-end model training data provided in an embodiment of this disclosure.

[0016] like Figure 1 As shown, the anomaly detection method for end-to-end model training data may include: S101, obtain the loss type corresponding to the training task in each training step during the end-to-end model training process.

[0017] It should be noted that the execution entity of the anomaly detection method for end-to-end model training data in this disclosure embodiment can be a hardware device with data processing capabilities and / or the necessary software to drive the hardware device. Optionally, the execution entity may include a server, a user terminal, and other smart devices. Optionally, the user terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, etc. Optionally, the server includes, but is not limited to, a network server, an application server, or a server of a distributed system, or a server combined with blockchain, etc. This disclosure embodiment does not impose specific limitations.

[0018] It should be noted that the end-to-end model is a deep learning architecture that uses a unified neural network to directly complete the mapping from the original input to the final output, without the need for manual division or independent optimization of multiple intermediate modules.

[0019] In some embodiments, the training process of an end-to-end model is divided into multiple training steps. Each training step may involve multiple training tasks, and each training task may correspond to one or more loss types. The corresponding loss type can be determined in advance based on the different training task types.

[0020] In other words, for a training task, multiple loss values ​​corresponding to that loss task can be calculated, thereby enabling multi-dimensional monitoring of a training task.

[0021] In some embodiments, the loss type includes, but is not limited to: regression type, diffusion trajectory type, prediction type, etc.

[0022] Taking an end-to-end autonomous driving model as an example, the training task type in a training step is the autonomous driving task type. The autonomous driving task can include, but is not limited to: self-vehicle planning and imitation, other vehicle prediction and imitation, assisted imitation, self-vehicle motion lexical probability, diffusion trajectory refinement, Euclidean Signed Distance Field (ESDF) construction, traffic light decision, assisted traffic light decision, etc. The loss type is the autonomous driving loss type, which can include, but is not limited to, the following loss types.

[0023] For example: For autonomous vehicle planning and imitation, the corresponding losses may include: planning trajectory classification loss, regression coefficient loss, regression exponential loss, heading angle loss, etc., and the corresponding loss type is regression type.

[0024] For predicting and imitating other vehicles, the corresponding losses may include: classification loss for different target categories (vehicles, pedestrians), position regression loss for different target categories, vehicle heading loss, etc., and the corresponding loss type is regression type.

[0025] For assisted imitation, the corresponding loss may include: regression loss of the assisted task, and the corresponding loss type is regression type.

[0026] For the probability of vehicle motion words, the corresponding loss can include: probability distribution loss of motion words, and the corresponding loss type is prediction type.

[0027] For the refinement of diffusion trajectories, the corresponding loss may include: diffusion model trajectory refinement loss (including a weighted sum of sub-items such as position, heading, velocity, acceleration, and curvature), and the corresponding loss type is diffusion trajectory type.

[0028] For ESDF construction, the corresponding loss can include: Euclidean signature distance field loss, and the corresponding loss type is prediction type.

[0029] For traffic light decisions, the corresponding losses may include: traffic light passage decision probability loss, and the corresponding loss type is prediction type.

[0030] For traffic light decision support, the corresponding loss may include: traffic light decision support loss, and the corresponding loss type is prediction type.

[0031] In this disclosure, there are no restrictions on the end-to-end model, the training task, or the loss type corresponding to the training task.

[0032] S102, perform the training task based on the multi-frame training data included in the training step, and determine the loss value corresponding to each frame of training data based on the loss type.

[0033] In some embodiments, different training steps may include different training tasks, and different training tasks may use different training data. For a single training step, the training task can be performed using multiple batches of training data. Each batch may contain multiple frames of training data.

[0034] For example, a training step includes Task 1 and Task 2. Task 1 is executed using training data from batch 1, and Task 2 is executed using training data from batch 2. The training data from batch 1 can be training data from frame 1 to frame 100, and the training data from batch 2 can be training data from frame 500 to frame 1000.

[0035] In some embodiments, a training task can be performed using multiple frames of training data in each training step. Since the training task corresponds to multiple loss types, the loss value of each frame of training data under each loss type can be calculated.

[0036] In some embodiments, different loss types correspond to different loss calculation methods, so the loss value of each frame of training data can be calculated based on the loss calculation method corresponding to any loss type.

[0037] One training task can correspond to multiple loss values.

[0038] Continuing with the example above, if the loss type corresponding to task 1 is type 1 and type 2, then for the training data from frame 1 to frame 100, the loss value can be calculated using the loss calculation method of type 1 and the loss value can be calculated using the loss calculation method of type 2, thereby obtaining the loss value of type 1 and the loss value of type 2 for each frame of training data in the training data from frame 1 to frame 100.

[0039] S103, based on the loss value, identifies candidate abnormal data with anomalies from multiple frames of training data.

[0040] In some embodiments, loss thresholds corresponding to different loss types can be obtained to determine the magnitude of the loss value of any loss type and the corresponding loss threshold, and the training data corresponding to the loss value greater than the loss threshold can be regarded as candidate abnormal data with anomalies.

[0041] In other words, if the loss value of any frame of training data is greater than the corresponding loss threshold, it can be determined that there is an anomaly, and the training data of that frame is a candidate anomaly data.

[0042] In some embodiments, once candidate anomalous data is identified, structured anomalous records can be generated based on the data attributes of the candidate anomalous data. For example, annomalous records may include training steps, scene type labels, anomalous categories, loss types, loss values, data file paths, frame numbers, etc.

[0043] The data file path and frame number can precisely locate a specific frame of data in the training dataset.

[0044] It should be noted that the model training disclosed herein can be distributed training, that is, the model can be trained on multiple graphics processing units (GPUs). Each GPU can generate anomaly records of candidate anomaly data. In order to avoid the input and output operations of anomaly records blocking the GPU training main thread, an asynchronous writing mechanism can be used to aggregate the anomaly records.

[0045] In some embodiments, anomaly records on each GPU can be written to a separate comma-separated values ​​(CSV) file.

[0046] S104, based on the data attributes of the candidate abnormal data, perform anomaly detection on the candidate abnormal data to determine the target abnormal data in the training step.

[0047] In some embodiments, data tuples can be constructed based on the data attributes of candidate anomaly data, and aggregation operations can be performed on the data tuples to achieve TOPK aggregation of the loss values ​​of candidate anomaly data based on the data tuples, thereby identifying the TOPK candidate anomaly data with the largest loss values ​​from the candidate anomaly data as target anomaly data.

[0048] In other words, anomaly detection of candidate anomaly data is also based on the loss value of the candidate anomaly data, in order to identify the data with the largest loss value from the candidate anomaly data as the target anomaly data.

[0049] In some embodiments, at least one data tuple can be constructed based on data attributes, wherein the number of elements included in the at least one data tuple is different. In this disclosure, the number of elements refers to the number of data attributes. For example, data tuple 1 may include three data attributes, such as task type, loss type, and the data file name of the data file to which the candidate anomaly data belongs. Each data attribute has its corresponding value. For example, the task type is type a, and the loss type is the loss value of type A.

[0050] For example, data tuple 2 can include two data attributes, such as task type and loss type.

[0051] In some embodiments, candidate anomalous data can be filtered based on different data tuples to achieve anomaly detection. For example, candidate anomalous data can be filtered first based on data tuple 1 to obtain the filtering results, and then the filtering results can be filtered a second time based on data tuple 2 to obtain the target anomalous data.

[0052] In some embodiments, screening candidate abnormal data can be performed by setting screening conditions. During the first screening, it is determined whether the candidate abnormal data meets the first screening conditions based on data tuple 1, and the candidate abnormal data that meets the first screening conditions can be used as the screening result.

[0053] During the second screening, based on data tuple 2, it is determined whether the screening results meet the second screening conditions, and candidate abnormal data that meet the second screening conditions are identified from the screening results as target abnormal data.

[0054] The first filtering criterion can be the K1 candidate outliers with the largest loss values, and the second filtering criterion can be the K2 candidate outliers with the largest maximum loss values. K1 and K2 are both natural numbers, with K1 being less than K2.

[0055] In some embodiments, data tuple 1 can be a triple and data tuple 2 can be a binary tuple. The triple has an additional data file name compared to the binary tuple. The data file name is the name of the data file to which the candidate abnormal data belongs.

[0056] In other words, filtering candidate anomaly data based on triples can yield filtering results for different data files, with each data file corresponding to one filtering result. Filtering the filtering results based on binary tuples can identify target anomaly data within different data files. This allows for initial filtering of each data file, narrowing the filtering scope, followed by further filtering within that narrowed scope. This reduces the computational load of anomaly detection and improves its efficiency.

[0057] According to the anomaly detection method for end-to-end model training data provided in this disclosure, by obtaining the loss type corresponding to the training task in each training step during the end-to-end model training process, and executing the training task based on the multi-frame training data included in the training step, the loss value corresponding to each frame of training data can be determined based on the loss type. Further, based on the loss value, candidate abnormal data with anomalies is identified from the multi-frame training data, and anomaly detection is performed on the candidate abnormal data based on its data attributes to identify the target abnormal data in the training step. Therefore, this solution can detect abnormal training data in real time during model training, avoiding resource waste caused by data anomalies during training and improving training efficiency. Moreover, anomaly detection through data attributes can improve anomaly localization efficiency and accelerate the anomaly investigation process, further improving anomaly detection efficiency. Correspondingly, when all solutions of this application are applied to autonomous driving scenarios, the resulting effects become the corresponding effects for autonomous driving scenarios, such as detecting abnormal training data for autonomous driving, avoiding resource waste caused by data anomalies during training, and improving the efficiency of autonomous driving model training. Furthermore, by using the data attributes of autonomous driving for anomaly detection, the efficiency of anomaly localization in autonomous driving can be improved, the anomaly investigation process can be accelerated, and the efficiency of anomaly detection in autonomous driving can be further improved.

[0058] Similarly, all the solutions in this application can be applied to autonomous driving scenarios to achieve the corresponding improvements in autonomous driving.

[0059] Figure 2 This is a flowchart illustrating another method for anomaly detection in end-to-end model training data provided in an embodiment of this disclosure.

[0060] like Figure 2 As shown, the anomaly detection method for end-to-end model training data may include: S201, obtain the loss type corresponding to the training task in each training step during the end-to-end model training process.

[0061] The details of step S201 can be found in the above embodiments and will not be repeated here.

[0062] S202, determine the loss calculation method corresponding to the loss type of at least one training task in the training step.

[0063] In some embodiments, a training step includes at least one training task, and each training task may correspond to at least one loss type. The loss types include, but are not limited to, regression type, diffusion trajectory type, prediction type, etc.

[0064] In some embodiments, different loss types correspond to different loss calculation methods. For example, regression type uses calculation method 1 to calculate the loss value; diffusion trajectory type uses calculation method 2 to calculate the loss value, etc.

[0065] In some embodiments, the correspondence between different loss types and loss calculation methods can be preset. Once the loss type of the training task is determined, the loss calculation method for calculating the loss value of the training task can be determined by querying the correspondence.

[0066] In this disclosure, by setting different calculation methods to calculate the loss value of different loss types, the accurate calculation of the loss value can be achieved.

[0067] S203, determine the training data to be used to perform at least one training task from the multi-frame training data included in the training step.

[0068] In some embodiments, a training step includes at least one training task, and the training data used to perform the at least one training task may be the same or different. Multiple frames of training data in a training step may be executed in batches for the at least one training task.

[0069] In other words, it is possible to determine a batch for performing at least one training task, and to determine at least one set of training data from that batch as training data for performing at least one training task.

[0070] For example, a training step includes Task 1 and Task 2. Task 1 is executed using training data from batch 1, and Task 2 is executed using training data from batch 2. The training data from batch 1 can be training data from frame 1 to frame 100, and the training data from batch 2 can be training data from frame 500 to frame 1000.

[0071] Among them, the training data of frames 1 to 100 are the training data used to perform task 1; the training data of frames 500 to 1000 are the training data used to perform task 2.

[0072] In some embodiments, each frame of training data has a frame number, and the training data used to perform at least one training task can be determined based on the frame number.

[0073] S204, Based on the loss calculation method, perform loss calculation on the training data used to perform at least one training task to obtain the loss value corresponding to each frame of training data.

[0074] In some embodiments, for the loss type being regression, the effective frames in the training step can be determined by the training step dimension, and the effective frames can be weighted and summed and then divided by the number of effective frames to obtain the loss value corresponding to each frame of training data, which is the regression loss value.

[0075] The formula for calculating the regression loss is shown below: (1) in, Indicates the first The regression loss value of the training data frames. Indicates the first Frame training data in the th The original loss value for each training step. Indicates a valid frame. This represents a constant that is divisible by zero.

[0076] In some embodiments, for a loss type of diffusion trajectory, the errors associated with the diffusion trajectory can be weighted and summed to obtain the loss value corresponding to each frame of training data, which is the diffusion trajectory loss value. The errors associated with the diffusion trajectory include, but are not limited to: position error, heading error, velocity error, acceleration error, curvature error, etc.

[0077] The formula for calculating the diffusion trajectory loss value is shown below: (2) in, This represents the diffusion trajectory loss value. Indicates positional error. The weighting coefficients representing the position error. Indicates heading error. The weighting coefficients representing the heading error. Indicates speed error, The weighting coefficients representing the speed error. Indicates acceleration error. The weighting coefficients representing the acceleration error. Indicates curvature error. The weighting coefficient represents the curvature error.

[0078] In some embodiments, for a loss type of prediction, a pre-calculated loss value can be obtained, the training data corresponding to the loss value can be determined, and the loss values ​​of different numbers from the same training data can be aggregated to obtain the predicted loss value of the training data.

[0079] S205, based on the loss value, identifies candidate anomalous data from multiple frames of training data.

[0080] In some embodiments, candidate anomalous data can be identified from multi-frame training data based on the magnitude of the loss value. By pre-setting loss thresholds for different loss types and comparing the loss values ​​of the training data with the loss thresholds under the same loss type, candidate anomalous data can be identified. This method can simply and efficiently identify anomalous training data from large-scale training data, reducing the computational load for identifying target anomalous data.

[0081] In some embodiments, the loss threshold can be a positive number or a negative number.

[0082] In some embodiments, by determining the loss threshold corresponding to the loss type, the loss value of any frame of training data is compared with the loss threshold. In response to the loss value of any frame of training data being greater than the loss threshold, any frame of training data is determined to be candidate abnormal data.

[0083] In some embodiments, by comparing the loss values ​​for each loss type and each frame of training data, candidate anomalous data can be identified, and an anomalous set can be generated based on the candidate anomalous data. The anomalous set is shown below: (3) in, Represents a set of exceptions. Indicates the first The loss value of the training data frames. Indicates the type of loss The corresponding loss threshold.

[0084] S206, Based on the data attributes of the candidate abnormal data, perform anomaly detection on the candidate abnormal data to determine the target abnormal data in the training step.

[0085] The details of step S206 can be found in the above embodiments and will not be repeated here.

[0086] According to the anomaly detection method for end-to-end model training data provided in this disclosure, by determining the loss calculation method corresponding to the loss type of at least one training task in the training step, loss calculation is performed on the training data used to perform at least one training task based on the loss calculation method, obtaining the loss value corresponding to each frame of training data. After obtaining the loss value, candidate abnormal data is determined from the training data according to the magnitude of the loss value, enabling quantitative evaluation and automatic identification of abnormal training data. Accordingly, when detecting abnormal training data for autonomous driving, the loss is calculated through different loss calculation methods, further improving the efficiency and accuracy of anomaly detection for autonomous driving.

[0087] Figure 3This is a flowchart illustrating another method for anomaly detection in end-to-end model training data provided in an embodiment of this disclosure.

[0088] like Figure 3 As shown, the anomaly detection method for end-to-end model training data may include: S301, obtain the loss type corresponding to the training task in each training step during the end-to-end model training process.

[0089] S302, perform the training task based on the multi-frame training data included in the training step, and determine the loss value corresponding to each frame of training data based on the loss type.

[0090] S303, based on the loss value, identifies candidate anomalous data from multiple frames of training data.

[0091] The relevant content of steps S301-S303 can be found in the above embodiments, and will not be repeated here.

[0092] S304, Based on data attributes, determine at least one data tuple corresponding to the candidate abnormal data.

[0093] In some embodiments, the elements in a data tuple are data attributes. Each data tuple includes a different number of elements, that is, each data tuple includes different data attributes and a different number of data attributes. Moreover, each data tuple includes at least one data tuple.

[0094] In some embodiments, the number of different types of data tuples can be related to data attributes. For example, for a triple with 3 elements, if its data attribute can be a data file name, then the number of triples can be equal to the number of data files.

[0095] In some embodiments, filtering candidate anomalous data through data tuples can enable precise location of anomalies, accelerate the anomaly screening process, and improve the efficiency of detecting whether training data is anomalous.

[0096] In some embodiments, triples and tuples corresponding to candidate anomaly data can be constructed based on data attributes. The triples include at least the task type, loss type, and data file name of the data file to which the candidate anomaly data belongs.

[0097] For example, the format of a triple is (task type, loss type, data file name). A single data file can correspond to at least one triple. That is, the training data from the same data file can be used to perform training tasks of at least one task type and also have loss values ​​of at least one loss type.

[0098] In some embodiments, the tuple includes at least the task type and loss type corresponding to the candidate anomaly data. For example, the format of the tuple is (task type, loss type).

[0099] S305, determine the loss value corresponding to the data tuple, and filter the candidate abnormal data based on the loss value corresponding to the data tuple to determine the target abnormal data.

[0100] In some embodiments, each candidate anomaly data has its corresponding triples and tuples. For the triples, the loss values ​​of the candidate anomaly data in each data file can be filtered to determine a first set number of candidate anomaly data with large loss values ​​in each data file.

[0101] For binary pairs, the already filtered data files can be further filtered. Since the data files contain at least one frame of candidate abnormal data, and the loss values ​​of at least one frame of candidate abnormal data may be different, the maximum loss value corresponding to a data file can be determined first, and the data files can be further filtered based on the maximum loss value to obtain a second set number of data files with the maximum loss value. The candidate abnormal data in the data file is then taken as the target abnormal data.

[0102] In some embodiments, determining a first predetermined number of candidate outliers with large loss values ​​in each data file can be achieved by performing TOPK aggregation on the candidate outliers based on the loss values ​​to determine the first predetermined number of candidate outliers with large loss values ​​in each data file.

[0103] In other words, the candidate anomaly data filtered based on triples is aggregated according to a data file. The aggregated data file includes a first predetermined number of candidate anomaly data.

[0104] In some embodiments, for each triple, a min-heap can be maintained to perform TOPK aggregation on the candidate anomaly data. Each triple corresponds to a min-heap, and the candidate anomaly data within the min-heap are from the same data file, with the same loss type and task type.

[0105] In some embodiments, the min-heap contains a first predetermined number of candidate anomaly data, and the candidate anomaly data are sorted in ascending order of loss value, wherein the top of the min-heap is the candidate anomaly data with the smallest loss value.

[0106] In some embodiments, after determining the min-heap corresponding to each triplet, the min-heap can be further filtered based on the pairs of candidate outlier data in the min-heap, so as to further filter the already filtered data files.

[0107] In some embodiments, a second set number of min-heaps can be determined from the min-heaps, and the min-heap with the smallest maximum loss value can be determined from the second set number of min-heaps. For example, if there are min-heap 1 and min-heap 2, the maximum loss value 1 of min-heap 1 and the maximum loss value 2 of min-heap 2 can be determined, and the loss value with the smaller maximum loss value can be determined from the maximum loss value 1 and the maximum loss value 2.

[0108] If the maximum loss value 1 is less than the maximum loss value 2, then the min-heap with the smallest maximum loss value is min-heap 1; if the maximum loss value 2 is less than the maximum loss value 1, then the min-heap with the smallest maximum loss value is min-heap 2.

[0109] In some embodiments, determining the second predetermined number of min-heaps from the min-heaps can be done randomly, by determining the magnitude of the maximum loss value of the other min-heaps and the loss value of the min-heap with the smallest maximum loss value, and filtering different min-heaps based on the magnitude comparison result. The other min-heaps are those other than the second predetermined number of min-heaps.

[0110] For ease of understanding, we consider the min-heap with the smallest maximum loss value as the target min-heap. Then, the loss value of the min-heap with the smallest maximum loss value, which is also the loss value of the target min-heap, is loss value 1, and the maximum loss value of the other min-heaps is loss value 2.

[0111] By comparing the magnitudes of loss value 1 and loss value 2, the size comparison result can be determined. If the size comparison result is that loss value 1 is less than loss value 2, the target min-heap can be filtered out; if the size comparison result is that loss value 1 is greater than loss value 2, other min-heaps can be filtered out.

[0112] In some embodiments, by filtering different min-heaps, it is possible to achieve a second set number of min-heaps as the filtering result based on tuples, and the loss values ​​of the second set number of min-heaps are all larger than the loss values ​​of other min-heaps of the same loss type.

[0113] In some embodiments, candidate anomaly data can be obtained from a second predetermined number of min-heaps as target anomaly data.

[0114] The anomaly detection method for end-to-end model training data provided in this disclosure can determine at least one data tuple corresponding to candidate anomaly data based on data attributes. By determining the loss value corresponding to the data tuple, candidate anomaly data can be filtered based on the loss value to identify target anomaly data. Anomaly detection through data tuples can transform anomaly detection into targeted screening of TOPK anomaly data, improving the efficiency of anomaly detection. Rapidly detecting anomaly training data is beneficial to improving the efficiency of end-to-end model training. Accordingly, when detecting anomaly training data for autonomous driving, transforming anomaly detection into targeted screening of TOPK anomaly data can improve the detection efficiency of anomaly data for autonomous driving, thereby further improving the training efficiency of the end-to-end autonomous driving model.

[0115] In the above embodiments, the process of filtering candidate abnormal data can be combined with... Figure 4 To understand further, Figure 4 This is a flowchart illustrating another method for anomaly detection in end-to-end model training data provided in an embodiment of this disclosure.

[0116] like Figure 4 As shown, the anomaly detection method for end-to-end model training data may include: S401, based on the loss value of the loss type in the triplet, filters the candidate outlier data in the same batch to determine the min-heap corresponding to each data file.

[0117] In some embodiments, a min-heap can be maintained to filter candidate anomaly data in the same batch based on the loss value of the loss type in the triplet. The min-heap has an upper limit; that is, it can store a first predetermined number of candidate anomaly data. This first predetermined number can be a first threshold number.

[0118] In some embodiments, a single data file can correspond to multiple triples, and each triple corresponds to a min-heap, so a single data file can correspond to multiple min-heaps.

[0119] In some embodiments, the min-heap sorts the candidate anomaly data according to the size of the loss value, and the top of the min-heap is the candidate anomaly data with the smallest loss value.

[0120] In some embodiments, the first frame of candidate abnormal data that has been filtered can be regarded as the first candidate abnormal data that has not been filtered. At this time, the min-heap does not contain any data, and the first candidate abnormal data can be directly written into the min-heap until the number of data in the min-heap reaches a first quantity threshold.

[0121] In some embodiments, a min-heap can be maintained for each triple. For the first candidate anomaly data that has not been filtered, the min-heap corresponding to the first candidate anomaly data is determined based on the triples of the first candidate anomaly data.

[0122] Whether to directly write the first candidate anomaly data into the min-heap is determined by checking whether the first number of the second candidate anomaly data in the min-heap reaches a first number threshold. If the first number of the second candidate anomaly data in the min-heap does not reach the first number threshold, the first candidate anomaly data can be written into the min-heap.

[0123] In some embodiments, in response to the first number of second candidate anomaly data in the min-heap reaching a first number threshold, it is determined whether the loss value of the first candidate anomaly data is greater than the loss value at the top of the heap. In response to the loss value of the first candidate anomaly data being greater than the loss value at the top of the heap, the second candidate anomaly data corresponding to the loss value at the top of the heap is replaced based on the first candidate anomaly data to obtain the min-heap corresponding to each data file.

[0124] In some embodiments, by performing the above filtering steps on multiple triples corresponding to a data file, multiple min-heaps can be obtained, and the multiple min-heaps constitute a min-heap corresponding to a data file.

[0125] For example, a triplet consists of group a, group b, group c, and group d, where the data file names of groups a, b, and c are the same. By maintaining the min-heaps corresponding to groups a, b, c, and d, we can obtain min-heaps a, b, c, and d, where min-heaps a, b, and c are min-heaps corresponding to the same data file.

[0126] In some embodiments, candidate anomaly data written into the min-heap can be used as second candidate anomaly data, and candidate anomaly data that has not been filtered can be used as first candidate anomaly data. Note that first candidate anomaly data is not written into the min-heap, but candidate anomaly data not written into the min-heap is not necessarily first candidate anomaly data.

[0127] For example, if there are data 1, data 2, and data 3, and data 1 and data 2 have already been filtered, and it is determined that data 1 is written to a min-heap, then data 2 and data 3 are also written to a min-heap. However, data 2 is data that has already been filtered but is not written to a min-heap, while data 3 is data that has not been filtered.

[0128] In some embodiments, after replacing the second candidate anomaly data corresponding to the top loss value based on the first candidate anomaly data, the original order of the loss values ​​in the min-heap may change. The second candidate anomaly data in the min-heap can be re-sorted based on the size of the loss values ​​to obtain the replaced top of the heap.

[0129] The replaced top of the heap can be either the first candidate outlier with a loss value greater than the loss value of the top of the heap, or it can be another second candidate outlier in the min-heap, whose loss value is the smallest in the min-heap.

[0130] For example, if the first candidate outlier with a loss value greater than the loss value of the top of the heap is data 1, and the other second candidate outliers in the min-heap are data a and data b, then if the loss value of data 1 is less than the loss values ​​of data a and data b, then the top of the heap after replacement is data 1. If the loss value of data 1 is greater than the loss value of data b, which is greater than the loss value of data a, then the top of the heap after replacement is data a.

[0131] In some embodiments, for the first candidate abnormal data that has not been filtered, the loss value comparison continues. In response to the existence of first candidate abnormal data whose loss value is greater than the replacement heap top loss value, the second candidate abnormal data of the replacement heap top is replaced until there is no first candidate abnormal data that has not been filtered, and the min-heap corresponding to each data file is obtained.

[0132] In this disclosure, candidate anomaly data is filtered through a min-heap. It is not necessary to store all candidate anomaly data; only one min-heap needs to be maintained, which greatly reduces memory overhead and improves computational efficiency. It is suitable for anomaly detection in large-scale end-to-end model training data.

[0133] S402, based on the loss value of the loss type in the tuple, filter the min-heap to identify the target outlier data from the min-heap.

[0134] In some embodiments, after obtaining the min-heap corresponding to the data file, further filtering can be performed on the min-heap based on the tuples to identify target anomalous data from the min-heap. Since the tuples do not contain the data file names, the min-heaps of all data files can be filtered.

[0135] In other words, triples perform filtering within a file, retaining candidate outliers with the highest loss values. Doubles, on the other hand, perform filtering at a macro level, aiming to limit the number of data files for each loss type, thus achieving a min-heap that retains the largest possible loss value.

[0136] By retaining the candidate outlier data with the largest loss value as a min-heap, and retaining the min-heap with the largest maximum loss value, the target outlier data can be determined from this min-heap. This allows for two-level filtering of candidate outlier data, enabling the rapid identification of the worst-performing candidate outlier data, i.e., the data with the largest loss value, as the target outlier data, thus achieving precise positioning of the target outlier data.

[0137] In some embodiments, since the determination of the min-heap corresponding to the data file has a certain time sequence, and the min-heap of all data files is not determined at the same time, the min-heap can be continuously tracked after it is determined, and the number of min-heaps can be counted to achieve the maintenance of a second set number of min-heaps.

[0138] In other words, by determining a second number of min-heaps, and in response to the second number reaching a second threshold, the maximum loss value of each min-heap is determined based on the loss value of the loss type in the tuple. The second threshold can be a second predetermined number.

[0139] In some embodiments, min-heaps can be sorted based on their maximum loss value according to the loss type to obtain a target min-heap with the smallest maximum loss value. That is, min-heaps of the same loss type can be sorted by their maximum loss value to determine different target min-heaps corresponding to different loss types.

[0140] In some embodiments, the min-heap is continuously tracked. If a new min-heap is tracked, the maximum loss value of the min-heap is compared with the maximum loss value of the target min-heap. If the maximum loss value of the min-heap is less than the maximum loss value of the target min-heap, the min-heap is filtered out, that is, the min-heap is discarded.

[0141] If the maximum loss value of the min-heap is greater than the maximum loss value of the target min-heap, then the target min-heap is filtered out, that is, the target min-heap is discarded, so that the maximum loss value of the retained min-heap is greater than the maximum loss value of the unretained min-heap.

[0142] In other words, in response to the existence of a min-heap whose maximum loss value is less than that of the target min-heap, any min-heap is filtered out; otherwise, the target min-heap is filtered out to obtain the min-heap selection results. The selection results include a first min-heap with a second quantity threshold. The maximum loss value of the first min-heap is greater than that of the second min-heap. The second min-heap is the filtered-out min-heap.

[0143] In some embodiments, anomaly detection on training data can be achieved by determining a min-heap of a second quantity threshold from the filtering results and using the candidate anomaly data corresponding to the min-heap of the second quantity threshold as the target anomaly data.

[0144] In some embodiments, since the anomaly records of candidate anomaly data are written to a CSV file, statistical information such as file frequency count, loss type frequency count, phase frequency count, phase total count, and scenario frequency count can be statistically analyzed based on the CSV file.

[0145] By filtering candidate anomaly data on a single GPU, a min-heap and target anomaly data are obtained. These, along with statistical information, are then serialized to produce a serialized file. Since multiple GPUs exist, the serialized files from these GPUs can be collected and merged according to statistical information to obtain an initial merged result.

[0146] Furthermore, based on the min-heap and the target anomalous data, the initial merge result is truncated to obtain the target merge result, thereby identifying the target anomalous data on each GPU.

[0147] The anomaly detection method for end-to-end model training data provided in this disclosure can filter candidate anomaly data in the same batch based on the loss value of the loss type in the triplet to determine the min-heap corresponding to each data file. Then, based on the loss value of the loss type in the binary tuple, the min-heap is further filtered to determine the target anomaly data. This method allows for first filtering candidate anomaly data based on the min-heap, eliminating the need to store all candidate anomaly data; only one min-heap needs to be maintained, significantly reducing memory overhead and improving computational efficiency. Finally, the target anomaly data is determined from the min-heap. This two-level filtering of candidate anomaly data improves the efficiency of anomaly data monitoring. Correspondingly, when detecting anomaly training data for autonomous driving, this two-level filtering can improve the efficiency and accuracy of anomaly detection, thereby reducing the false negative and false positive rates.

[0148] Figure 5 This is a schematic diagram of the process for filtering candidate abnormal data provided in an embodiment of this disclosure. Figure 5 The process includes a first-level filter (triplets) and a second-level filter (binaries). After obtaining candidate anomaly data, it is input into the first-level filter. Based on the triples of the candidate anomaly data, a min-heap is selected. After obtaining the min-heap, it is input into the second-level filter. Based on the binary pairs, the min-heap is further filtered to select the min-heap with a second quantity threshold.

[0149] In addition, the min-heap of the data file can be pre-filtered and optimized. If the maximum loss value of the min-heap is less than the maximum loss value of the target min-heap, the min-heap is filtered out.

[0150] For specific implementation methods of filtering candidate abnormal data, please refer to any of the above embodiments, which will not be repeated here.

[0151] Figure 6 This is a schematic diagram of the process for providing detection training data in an embodiment of this disclosure.

[0152] For each training step and the corresponding loss type for the training task within that step, the loss value for the training data of different loss types can be calculated by iterating through the loss types. By comparing the loss value with a loss threshold, candidate outliers can be identified. Simultaneously, candidate outliers from different GPUs can be asynchronously written to a CSV file.

[0153] Furthermore, the candidate abnormal data is screened in two stages to identify the target abnormal data, and the target abnormal data on different GPUs is updated and serialized to obtain a serialized file.

[0154] Figure 7 This is a flowchart illustrating another method for anomaly detection in end-to-end model training data provided in an embodiment of this disclosure.

[0155] like Figure 7 As shown, the anomaly detection method for end-to-end model training data may include: S701, obtain the loss type corresponding to the training task in each training step during the end-to-end model training process.

[0156] S702 executes the training task based on the multi-frame training data included in the training step, and determines the loss value corresponding to each frame of training data based on the loss type.

[0157] S703 identifies candidate anomalous data from multiple frames of training data based on the loss value.

[0158] S704 performs anomaly detection on candidate anomaly data based on the data attributes of the candidate anomaly data in order to determine the target anomaly data in the training step.

[0159] The relevant content of steps S701-S704 can be found in the above embodiments, and will not be repeated here.

[0160] S705 provides multi-dimensional alerts based on target anomaly data.

[0161] In some embodiments, after identifying the target abnormal data, multi-dimensional alerts can be issued based on the target abnormal data. By determining the alert triggering conditions corresponding to each alert dimension, in response to the target abnormal data satisfying any alert triggering condition, the alert information corresponding to the alert dimension of any alert triggering condition is determined, thereby enabling alerts to be issued for the target abnormal data based on the alert information.

[0162] The alarm information may include the number of GPUs, training task, current training step, and abnormal loss type.

[0163] In other words, alarm information can indicate the training tasks and training steps that are abnormal, as well as which type of loss is abnormal.

[0164] In some embodiments, multi-dimensional alerts may include comparison alerts with a baseline version and convergence trend alerts.

[0165] The comparison alarm with the baseline version can be based on the anomaly statistics of a training task with validated quality (the baseline version). For each loss type of the target anomalous data, the average number of anomaly triggers at any given stage is calculated, and an alarm is triggered based on the number of anomaly triggers. Any given stage can be a training step stage.

[0166] In other words, the alarm triggering conditions for the comparison alarms with the baseline version are related to the number of times the anomaly is triggered.

[0167] In some embodiments, the ratio of the number of anomaly triggers of the target abnormal data to the baseline number of anomaly triggers can be calculated. If the ratio is greater than the alarm multiplier threshold, the alarm triggering condition is determined to be met. In other words, the alarm triggering condition for the comparison alarm with the baseline version is that the ratio of the number of anomaly triggers to the baseline number of anomaly triggers is greater than the alarm multiplier threshold.

[0168] In some embodiments, the formula for determining the number of exception triggers is as follows: (4) in, Indicates the type of loss The average number of exceptions triggered at any given stage. Indicates the total number of stages. Indicates the number of GPUs.

[0169] The formula for calculating the ratio of the number of exception triggers to the baseline number of exception triggers is shown below: (5) in, Indicates the baseline number of exceptions triggered. This indicates the alarm multiplier threshold.

[0170] Among them, the convergence trend alarm can perform linear regression fitting on the number of anomaly triggers at each stage for each loss type of the target abnormal data to determine the convergence trend, and determine whether to trigger an alarm based on the convergence trend.

[0171] In other words, the alarm triggering conditions for convergence trend alerts are related to the convergence trend.

[0172] In some embodiments, the slope of the convergence trend can be determined, and if the slope is greater than a slope threshold, it can be determined that the alarm triggering condition is met. That is, the alarm triggering condition for a convergence trend alarm is that the slope of the convergence trend is greater than a slope threshold.

[0173] In some embodiments, the formula for determining the convergence trend is as follows: (6) in, Indicates the type of loss The convergence trend Indicates the slope. Indicates the current stage. Represents a constant.

[0174] In some embodiments, alarm intervals can be set to trigger alarms to avoid frequent alarms. Alternatively, when multiple loss types trigger alarms, an alarm can be triggered only for the loss type with the highest loss value.

[0175] In some embodiments, after identifying the target abnormal data, the original training data can be used for further investigation to locate the cause of the abnormality, repair the sample, or optimize the model accordingly, thereby improving the overall quality of the training data and the generalization ability of the model.

[0176] In some embodiments, the cause of the anomaly is determined by tracing the source of the target anomalous data. This can be done by locating the training step of the target anomalous data and determining its contextual information, thereby tracing the cause of the anomaly based on the contextual information.

[0177] For example, tracing data to create a chain and checking for operations that may introduce abnormalities at each stage.

[0178] In some embodiments, the target anomalous data can also be located to determine at least one of the training task, training step, and frame number in which the target anomalous data is located, thereby enabling the anomalous data to be accurately identified from the training data, which is beneficial for cleaning or repairing the anomalous data.

[0179] In some embodiments, anomaly statistics during training with different data versions can be compared to assess changes in data quality. For example, comparing changes in the frequency of anomaly triggering for different loss types; a decrease in frequency indicates improved data quality. Identifying phased trend changes; faster convergence speed indicates improved data quality. Identifying changes in the list of high-frequency anomaly data files: newly added / disappeared high-frequency anomaly data files indicate the scope of impact of data changes. Identifying changes in scene distribution: changes in the proportion of anomalies for a specific scene type reflect changes in the quality of data in that scene.

[0180] In some embodiments, visualization charts and graphs can also be generated based on the target anomaly data, such as pie charts showing the distribution of the number of anomaly triggers for each loss type, line charts showing the change of the number of anomaly triggers for each loss type with the training phase, bar charts showing the distribution of anomalies for each scenario type, and ranking tables of high-frequency anomaly data files.

[0181] According to the end-to-end model training data anomaly detection method provided in this disclosure, after identifying target abnormal data, if the target abnormal data meets any alarm triggering condition, the alarm information corresponding to the alarm dimension of any alarm triggering condition can be determined, and an alarm can be issued for the target abnormal data based on the alarm information. This achieves real-time alarming of abnormal data during the training process, avoiding the continuous use of abnormal data to train the model and improving training efficiency. Accordingly, when detecting abnormal training data for autonomous driving, issuing alarms by detecting abnormal data can promptly identify data quality issues, prevent abnormal data from polluting model training, and thus ensure the performance and safety of the autonomous driving model.

[0182] Based on any of the above embodiments, an anomaly detection system for end-to-end model training data can be determined based on the anomaly detection method for end-to-end model training data disclosed herein. Figure 8 This is a schematic diagram of the structure of an anomaly detection system for end-to-end model training data provided in an embodiment of this disclosure.

[0183] Figure 8 It includes an end-to-end model training system, a distributed aggregation module, and an analysis and alerting module.

[0184] The end-to-end model training system includes a loss anomaly detection module. By loading training data and performing model forward propagation, the system can calculate loss values ​​for different loss types for different training tasks. The loss anomaly detection module performs threshold comparisons to identify candidate anomalies and then applies a two-level TOPK filtering process to determine the target anomaly.

[0185] The distributed aggregation module can merge the target anomaly data on each GPU and input it into the analysis and alarm module to analyze and alarm on the target anomaly data on each GPU.

[0186] Alarm methods include baseline comparison alarms and convergence trend alarms. Once the target abnormal data meets the alarm triggering conditions for baseline comparison alarms or convergence trend alarms, corresponding alarm information can be generated, and alarms can be issued based on the alarm information.

[0187] It should be noted that this disclosure can be applied to the training process of end-to-end models for autonomous driving, with specific application scenarios as follows: Application Scenario 1: Daily Training Monitoring During the daily training of the model, this system runs automatically and continuously monitors various loss anomalies. When training instability is detected (such as a surge in the frequency of a certain type of loss anomaly or an abnormal convergence trend), it automatically sends an alert through instant messaging tools, enabling R&D personnel to intervene in a timely manner and avoid wasting training resources.

[0188] Application Scenario 2: Troubleshooting Data Issues When a model is found to perform poorly in specific scenarios (such as intersection collisions, lane change scenario control anomalies, etc.), you can: 1. Select the appropriate loss type based on the problem type; 2. View the exported data file of the TOPK outlier data for this loss type; 3. Locate the specific training data based on the file path and frame number; 4. Analyze the data to identify any issues such as labeling errors or sensor malfunctions; 5. Repair abnormal data.

[0189] Application Scenario 3: Data Version Iteration Evaluation After data cleaning or relabeling, the effect of data improvement can be quickly evaluated by comparing the abnormal statistical indicators during training with the old and new data versions, providing a quantitative basis for data iteration.

[0190] Application Scenario 4: Abnormal Data Cleaning Based on the ranking of high-frequency abnormal data files, batch identification and cleaning of abnormal data can be performed to improve the overall quality of the training dataset.

[0191] Corresponding to the anomaly detection methods for end-to-end model training data provided in the above embodiments, an embodiment of this disclosure also provides an anomaly detection device for end-to-end model training data. Since the anomaly detection device for end-to-end model training data provided in this disclosure corresponds to the anomaly detection methods for end-to-end model training data provided in the above embodiments, the implementation methods of the above-mentioned anomaly detection methods for end-to-end model training data are also applicable to the anomaly detection device for end-to-end model training data provided in this disclosure, and will not be described in detail in the following embodiments.

[0192] Figure 9 This is a schematic diagram of the structure of an anomaly detection device for end-to-end model training data provided in an embodiment of this disclosure.

[0193] like Figure 9 As shown, the end-to-end model training data anomaly detection device 900 of this disclosure includes an acquisition module 901, a first determination module 902, a second determination module 903 and an anomaly detection module 904.

[0194] Module 901 is used to obtain the loss type corresponding to the training task in each training step during the end-to-end model training process. The first determining module 902 is used to execute the training task based on the multi-frame training data included in the training step, and to determine the loss value corresponding to each frame of training data based on the loss type. The second determining module 903 is used to determine candidate abnormal data with anomalies from the multi-frame training data based on the loss value; Anomaly detection module 904 is used to perform anomaly detection on the candidate anomaly data based on the data attributes of the candidate anomaly data, so as to determine the target anomaly data in the training step.

[0195] In one embodiment of this disclosure, the first determining module 902 is further configured to: determine the loss calculation method corresponding to the loss type of at least one training task in the training step; determine the training data used to perform the at least one training task from the multi-frame training data included in the training step; and perform loss calculation on the training data used to perform the at least one training task based on the loss calculation method to obtain the loss value corresponding to each frame of training data.

[0196] In one embodiment of this disclosure, the second determining module 903 is further configured to: determine a loss threshold corresponding to the loss type; and determine the any frame of training data as the candidate abnormal data in response to the loss value of any frame of training data being greater than the loss threshold.

[0197] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: determine at least one data tuple corresponding to the candidate anomaly data based on the data attributes; wherein each data tuple includes a different number of elements, and each data tuple includes at least one data tuple; determine the loss value corresponding to the data tuple; and filter the candidate anomaly data based on the loss value corresponding to the data tuple to determine the target anomaly data.

[0198] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: construct triples and tuples corresponding to the candidate anomaly data based on the data attributes; wherein the triples include at least the task type, loss type, and data file name of the data file to which the candidate anomaly data belongs; and the tuples include at least the task type and loss type corresponding to the candidate anomaly data.

[0199] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: filter candidate anomaly data in the same batch based on the loss value of the loss type in the triplet to determine the min-heap corresponding to each data file; wherein the min-heap sorts the candidate anomaly data according to the size of the loss value, and the top of the min-heap is the candidate anomaly data with the smallest loss value; and filter the min-heap based on the loss value of the loss type in the binary tuple to determine the target anomaly data from the min-heap.

[0200] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: for first candidate anomaly data that has not been filtered, determine a min-heap corresponding to the first candidate anomaly data based on the triples of the first candidate anomaly data; in response to a first number of second candidate anomaly data in the min-heap reaching a first number threshold, determine whether the loss value of the first candidate anomaly data is greater than the top loss value of the heap; in response to the loss value of the first candidate anomaly data being greater than the top loss value of the heap, replace the second candidate anomaly data corresponding to the top loss value of the heap based on the first candidate anomaly data to obtain the min-heap corresponding to each data file.

[0201] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: re-sort the second candidate anomaly data in the min-heap based on the magnitude of the loss value to obtain a replaced heap top; in response to the existence of first candidate anomaly data with a loss value greater than the loss value of the replaced heap top, replace the second candidate anomaly data of the replaced heap top until there is no first candidate anomaly data that has not been filtered, thereby obtaining the min-heap corresponding to each data file.

[0202] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: determine a second number of the min-heaps; in response to the second number reaching a second number threshold, determine a maximum loss value for each min-heap based on the loss value of the loss type in the tuples; sort the min-heaps based on the maximum loss value to obtain a target min-heap with the smallest maximum loss value; and in response to the existence of a maximum loss value of any min-heap being less than the maximum loss value of the target min-heap, filter any min-heap; otherwise, filter the target min-heap to obtain the filtering result of the min-heaps.

[0203] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: determine the min-heap of the second quantity threshold from the filtering results; and use the candidate anomaly data corresponding to the min-heap of the second quantity threshold as the target anomaly data.

[0204] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to: determine the alarm triggering condition corresponding to each alarm dimension; in response to the target abnormal data satisfying any alarm triggering condition, determine the alarm information of the alarm dimension corresponding to the any alarm triggering condition; and issue an alarm to the target abnormal data based on the alarm information.

[0205] In one embodiment of this disclosure, the anomaly detection module 904 is further configured to include at least one of the following operations: tracing the source of the target anomaly data to determine the cause of the anomaly; and locating the target anomaly data to determine at least one of the training task, training step, and frame number in which the target anomaly data is located.

[0206] According to the anomaly detection device for end-to-end model training data provided in this disclosure, by acquiring the loss type corresponding to the training task in each training step during the end-to-end model training process, and executing the training task based on the multi-frame training data included in the training step, the loss value corresponding to each frame of training data can be determined based on the loss type. Further, based on the loss value, candidate abnormal data with anomalies is identified from the multi-frame training data, and anomaly detection is performed on the candidate abnormal data based on its data attributes to identify the target abnormal data in the training step. Therefore, this solution can detect abnormal training data in real time during model training, avoiding resource waste caused by data anomalies during training and improving training efficiency. Moreover, anomaly detection through data attributes can improve anomaly localization efficiency and accelerate the anomaly investigation process, further improving anomaly detection efficiency. Correspondingly, when all solutions of this application are applied to autonomous driving scenarios, the resulting effects become the corresponding effects for autonomous driving scenarios, such as detecting abnormal training data for autonomous driving, avoiding resource waste caused by data anomalies during training, and improving the efficiency of autonomous driving model training. Furthermore, by using the data attributes of autonomous driving for anomaly detection, the efficiency of anomaly localization in autonomous driving can be improved, the anomaly investigation process can be accelerated, and the efficiency of anomaly detection in autonomous driving can be further improved.

[0207] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0208] The examples in this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and provisions. In the examples of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.

[0209] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0210] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0211] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0212] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes based on computer programs / instructions stored in read-only memory (ROM) 1002 or loaded into random access memory (RAM) 1003 from storage unit 1006. RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0213] Multiple components in device 1000 are connected to I / O interface 1005, including: input units 1006 such as keyboard, mouse, etc.; output units 1007 such as various types of displays, speakers, etc.; storage units 1008 such as disks, optical discs, etc.; and communication units 1009 such as network cards, modems, wireless transceivers, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0214] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as an anomaly detection method for end-to-end model training data. For example, in some embodiments, the anomaly detection method for end-to-end model training data can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1006. In some embodiments, part or all of the computer program / instructions can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program / instructions are loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the anomaly detection method for end-to-end model training data described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other suitable manner (e.g., by means of firmware) to perform an anomaly detection method for end-to-end model training data.

[0215] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0216] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0217] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0219] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0220] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. The client-server relationship is created by computer programs / instructions running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0221] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in the disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this document does not impose any restrictions.

[0222] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An anomaly detection method for end-to-end model training data, wherein, The method includes: Obtain the loss type corresponding to the training task in each training step during the end-to-end model training process; The training task is performed based on the multi-frame training data included in the training step, and the loss value corresponding to each frame of training data is determined based on the loss type. Based on the loss value, candidate abnormal data with anomalies are determined from the multi-frame training data; Based on the data attributes of the candidate abnormal data, anomaly detection is performed on the candidate abnormal data to determine the target abnormal data in the training step.

2. The method according to claim 1, wherein, The step of determining the loss value corresponding to each frame of training data based on the loss type includes: Determine the loss calculation method corresponding to the loss type of at least one training task in the training step; From the multi-frame training data included in the training step, determine the training data used to perform the at least one training task; Based on the loss calculation method, loss calculation is performed on the training data used to perform the at least one training task to obtain the loss value corresponding to each frame of training data.

3. The method according to claim 1, wherein, The step of determining candidate abnormal data with anomalies from the multi-frame training data based on the loss value includes: Determine the loss threshold corresponding to the loss type; If the loss value of any frame of training data is greater than the loss threshold, the any frame of training data is determined to be the candidate abnormal data.

4. The method according to claim 1, wherein, The step of detecting anomalies in the candidate anomaly data based on its data attributes to determine the target anomaly data in the training step includes: Based on the data attributes, at least one data tuple corresponding to the candidate abnormal data is determined; wherein, each data tuple includes a different number of elements, and each data tuple includes at least one data tuple. Determine the loss value corresponding to the data tuple, and filter the candidate abnormal data based on the loss value corresponding to the data tuple to determine the target abnormal data.

5. The method according to claim 4, wherein, The step of determining at least one data tuple corresponding to the candidate abnormal data based on the data attributes includes: Based on the data attributes, construct the triples and tuples corresponding to the candidate abnormal data; The triplet includes at least the task type, loss type, and data file name of the data file to which the candidate anomaly data belongs; The tuple includes at least the task type and loss type corresponding to the candidate abnormal data.

6. The method according to claim 5, wherein, The step of filtering the candidate outlier data based on the loss value corresponding to the data tuple includes: Based on the loss values ​​of the loss types in the triplet, candidate outlier data in the same batch are filtered to determine the min-heap corresponding to each data file; wherein, the min-heap sorts the candidate outlier data according to the size of the loss value, and the top of the min-heap is the candidate outlier data with the smallest loss value; Based on the loss values ​​of the loss types in the tuples, the min-heap is filtered to determine the target abnormal data from the min-heap.

7. The method according to claim 6, wherein, The step of filtering candidate outlier data in the same batch based on the loss value of the loss type in the triplet to determine the min-heap corresponding to each data file includes: For the first candidate abnormal data that has not been filtered, a min-heap corresponding to the first candidate abnormal data is determined based on the triples of the first candidate abnormal data. In response to the first number of second candidate anomaly data in the min-heap reaching a first number threshold, it is determined whether the loss value of the first candidate anomaly data is greater than the loss value at the top of the heap. In response to the first candidate anomaly data having a loss value greater than the heap top loss value, the second candidate anomaly data corresponding to the heap top loss value is replaced based on the first candidate anomaly data to obtain the min-heap corresponding to each data file.

8. The method according to claim 7, wherein, The step of replacing the second candidate anomaly data corresponding to the top loss value of the heap with the first candidate anomaly data to obtain the min-heap corresponding to each data file includes: The second candidate outlier data in the min-heap are re-sorted based on the magnitude of the loss value to obtain the replaced heap top; In response to the existence of first candidate abnormal data with a loss value greater than the replacement heap top loss value, the second candidate abnormal data of the replacement heap top is replaced until there is no first candidate abnormal data that has not been filtered, thus obtaining the min-heap corresponding to each data file.

9. The method according to claim 7, wherein, The filtering of the min-heap based on the loss value of the loss type in the binary tuple includes: Determine a second number of the min-heaps, and in response to the second number reaching a second number threshold, determine the maximum loss value of each min-heap based on the loss value of the loss type in the tuples; Sort the min-heaps based on the maximum loss value to obtain the target min-heap with the minimum maximum loss value; In response to the existence of a minimum heap whose maximum loss value is less than that of the target minimum heap, the minimum heap is filtered; otherwise, the target minimum heap is filtered to obtain the filtering result of the minimum heap.

10. The method according to claim 9, wherein, Determining the target anomalous data from the min-heap includes: Determine the min-heap of the second quantity threshold from the filtering results; The candidate anomaly data corresponding to the min-heap of the second quantity threshold is taken as the target anomaly data.

11. The method according to any one of claims 1-10, wherein, After determining the target anomalous data in the training step, the process includes: Determine the alarm triggering conditions corresponding to each alarm dimension; In response to the target abnormal data satisfying any alarm triggering condition, the alarm information corresponding to the alarm dimension of the any alarm triggering condition is determined; An alert is issued for the abnormal target data based on the alarm information.

12. The method according to any one of claims 1-10, wherein, After determining the target anomalous data in the training step, the method further includes at least one of the following operations: The source of the target abnormal data is traced to determine the cause of the abnormality. The target anomalous data is located to determine at least one of the following: the training task, the training step, and the frame number where the target anomalous data is located.

13. An anomaly detection device for end-to-end model training data, wherein, The device includes: The acquisition module is used to acquire the loss type corresponding to the training task in each training step during the end-to-end model training process. The first determining module is used to execute the training task based on the multi-frame training data included in the training step, and to determine the loss value corresponding to each frame of training data based on the loss type. The second determining module is used to determine candidate abnormal data with anomalies from the multi-frame training data based on the loss value; An anomaly detection module is used to perform anomaly detection on the candidate anomaly data based on the data attributes of the candidate anomaly data, so as to determine the target anomaly data in the training step.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-12.

Images