Sample selection method and device for end-to-end algorithm, equipment and medium
Patent Information
- Application Number
- CN202411395281.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-02-28
AI Technical Summary
The samples of end-to-end algorithms in the prior art are incomplete, resulting in high data labeling costs and low training efficiency. The existing difficult sample selection strategies cannot be scientifically and comprehensively evaluated, resulting in the data set exceeding the boundaries of end-to-end algorithms.
The two-stage high-value sample selection process is adopted. First, the original sensor data is divided into in-service data and out-of-service data through VLM large model and multi-modal scene classification model, and the out-of-service data is abandoned; then, the evaluation indicators such as 3D dynamic target object perception, multi-object tracking, open-loop test evaluation and closed-loop test evaluation are comprehensively considered, and the overall evaluation indicators of the end-to-end algorithm are calculated and high-value data are screened as the training set.
It improves the robustness of end-to-end algorithms within the business boundaries, reduces data labeling costs, improves training efficiency, and ensures that the data set is within the boundaries of end-to-end algorithms, avoiding the coarse-grained classification of invalid samples and effective samples.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a sample selection method, device, equipment and medium for an end-to-end algorithm. Background Art
[0002] The field of autonomous driving focuses on using modular / non-modular deep learning models to achieve end-to-end intelligent driving tasks from perception to prediction to decision-making. Deep learning models are often data-driven, that is, they rely on a large amount of data to train the model, so that the end-to-end algorithm based on the deep learning model can have high accuracy and high robustness in different scenarios. In other words, the quality and quantity of data determine the upper limit of deep learning models in the field of autonomous driving, and then determine the overall performance of intelligent driving functions. Large-scale, high-quality data often means huge annotation costs, especially for end-to-end algorithms. Since end-to-end algorithms require multi-modal sensor input, the cost of data collection is relatively high. At the same time, the amount of data required to train end-to-end algorithms is also huge compared to a single task (perception / prediction / planning). And since the data that can really improve the performance of end-to-end algorithms often accounts for a small proportion, how to select the most helpful data for the end-to-end algorithm is a very valuable research direction.
[0003] The core issue of how to select difficult samples is how to establish a scientific and comprehensive evaluation system to select data that can truly improve the performance of end-to-end algorithms from a large number of repeated data sets, thereby reducing the cost of data annotation and improving the training efficiency of end-to-end algorithms. Especially for end-to-end algorithms, since they need to directly generate the planning results of the vehicle from the original sensor data through deep learning model design, although the environmental perception results and trajectory prediction results are used as intermediate features of the algorithm and are transmitted through implicit model results, if there is a demand to generate environmental perception results and trajectory prediction results, the implicit intermediate features need to be output, and this intermediate result needs to be considered in the evaluation system.
[0004] Existing difficult sample selection strategies often only design evaluation indicators for a single task and select difficult samples based on the indicators. For example, environmental perception tasks often only need to evaluate the detection accuracy of dynamic / static targets, while planning tasks only consider factors such as the safety and comfort of predicted trajectories. Since the end-to-end task will include environmental perception tasks, trajectory prediction tasks, and self-vehicle planning tasks integrated into the same model, the evaluation indicators for a single task are often not scientific and comprehensive. Therefore, this patent designs an evaluation indicator that comprehensively considers dynamic target perception, multi-target tracking, open-loop test evaluation, and closed-loop test evaluation.
[0005] If the data is selected based only on a set of evaluation indicators, it will often be classified into invalid samples and valid samples based only on the evaluation indicators of the model on different data. The granularity of data classification is coarse. At the same time, if the data is selected only based on the evaluation indicators, the selected data set may be helpful for model improvement, but it exceeds the boundaries of the end-to-end algorithm. Summary of the invention
[0006] In view of this, an embodiment of the present invention provides a sample selection method for an end-to-end algorithm to solve the technical problem of incomplete samples of the end-to-end algorithm in the prior art. The method includes:
[0007] Input the raw sensor data into the VLM large model, extract scene description words, input the scene description words, raw sensor data, local map data and vehicle posture data into the multimodal scene classification model, and divide the raw sensor data into business data and non-business data;
[0008] Calculate the NDS evaluation index of the 3D dynamic target perception task in the end-to-end algorithm, calculate the association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm, calculate the open-loop test evaluation index of the target trajectory prediction task in the end-to-end algorithm, and calculate the closed-loop test evaluation index of the planning task in the end-to-end algorithm;
[0009] The overall evaluation index of the end-to-end algorithm is calculated by NDS evaluation index, association accuracy, association quality, open-loop test evaluation index and closed-loop test evaluation index. The data in the business data that meets the overall evaluation index is regarded as high-value data, and the high-value data is used as the training set of the end-to-end algorithm.
[0010] The embodiment of the present invention also provides a sample selection device for an end-to-end algorithm to solve the technical problem of incomplete samples of the end-to-end algorithm in the prior art. The device includes:
[0011] The initial sample selection module is used to input the raw sensor data into the VLM large model, extract scene description words, input the scene description words, raw sensor data, local map data and ego vehicle posture data into the multimodal scene classification model, and divide the raw sensor data into business data and non-business data;
[0012] The evaluation index calculation module is used to calculate the NDS evaluation index of the 3D dynamic target perception task in the end-to-end algorithm, calculate the association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm, calculate the open-loop test evaluation index of the target trajectory prediction task in the end-to-end algorithm, and calculate the closed-loop test evaluation index of the planning task in the end-to-end algorithm;
[0013] The secondary sample selection module is used to calculate the overall evaluation index of the end-to-end algorithm through the NDS evaluation index, association accuracy, association quality, open-loop test evaluation index and closed-loop test evaluation index, and regard the data in the business data that meets the overall evaluation index as high-value data, and use the high-value data as the training set of the end-to-end algorithm.
[0014] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements any of the above-mentioned sample selection methods for the end-to-end algorithm to solve the technical problem of imperfect samples of the end-to-end algorithm in the prior art.
[0015] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned sample selection methods for end-to-end algorithms to solve the technical problem of imperfect samples of end-to-end algorithms in the prior art.
[0016] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:
[0017] Through a two-stage high-value sample selection process and combined with a multimodal business boundary data selection strategy based on large model scene description words, the end-to-end algorithm is comprehensively evaluated from the dimensions of perception, tracking, prediction, and planning, thereby improving the robustness of the end-to-end algorithm for data within the business boundary. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 is a flow chart of a sample selection method for an end-to-end algorithm provided by an embodiment of the present invention;
[0020] Figure 2 is a flow chart of a sample selection method for implementing the above end-to-end algorithm provided by an embodiment of the present invention;
[0021] Figure 3 is a structural block diagram of a computer device provided by an embodiment of the present invention;
[0022] Figure 4 It is a structural block diagram of a sample selection device for an end-to-end algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0024] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.
[0025] In an embodiment of the present invention, a sample selection method for an end-to-end algorithm is provided, such as Figure 1 and Figure 2 As shown, the method includes:
[0026] Step S101: inputting the raw sensor data into the VLM large model, extracting scene description words, inputting the scene description words, raw sensor data, local map data and vehicle posture data into the multimodal scene classification model, and dividing the raw sensor data into in-service data and out-of-service data;
[0027] Step S102: Calculate the NDS evaluation index of the 3D dynamic target perception task in the end-to-end algorithm, calculate the association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm, calculate the open-loop test evaluation index of the target trajectory prediction task in the end-to-end algorithm, and calculate the closed-loop test evaluation index of the planning task in the end-to-end algorithm;
[0028] Step S103: The overall evaluation index of the end-to-end algorithm is calculated by using the NDS evaluation index, association accuracy, association quality, open-loop test evaluation index and closed-loop test evaluation index. The data in the business data that meets the overall evaluation index is regarded as high-value data, and the high-value data is used as the training set of the end-to-end algorithm.
[0029] Specifically, the sample selection method of the embodiment of the present application is a two-stage data selection strategy. First, the scene description words are generated from the original sensor data combined with the VLM large model, and then the generated scene description words, original sensor data, local map data and self-vehicle posture data are input into the first-stage multimodal scene selection model, so that the original sensor data can be divided into data within the business boundary and data outside the business boundary, and the data outside the business boundary is discarded. The data outside the business boundary is input into the end-to-end algorithm, and then evaluated using the comprehensive evaluation indicators designed by this patent. Data with low scores indicates that the end-to-end algorithm performs poorly on this data. This data is likely to be a scene that is not covered by the end-to-end algorithm. After targeted manual annotation and adding it to the end-to-end algorithm training set, it is very likely that the accuracy of the end-to-end algorithm can be improved. Therefore, this type of algorithm is defined as high-value data, and vice versa.
[0030] In specific implementation, the following steps are used to calculate the NDS evaluation index for the 3D dynamic target perception task in the end-to-end algorithm:
[0031] Calculate the average detection rate mAP of each category on different distance measurement benchmarks, where Among them, C is the target category set, D is the different distance threshold sets, AP c,d is the accuracy of the target category c at different distance thresholds d; the NDS evaluation index is calculated by the positioning error between the target 3D box and the true value box, the size error between the target 3D box and the true value box, the orientation error between the target 3D box and the true value box, the speed error between the target 3D box and the true value box, the category error between the target 3D box and the true value box, and the average detection rate mAP of each category on different distance measurement benchmarks, where
[0032] mATE is the positioning error between the target object 3D box and the true value box, mASE is the size error between the target object 3D box and the true value box, mAOE is the orientation error between the target object 3D box and the true value box, mAVE is the velocity error between the target object 3D box and the true value box, mAAE is the category error between the target object 3D box and the true value box, and a, b, c, d, e, f and g are all real numbers that can be adjusted dynamically.
[0033] Specifically, the evaluation index definition of the second stage includes the NDS evaluation index used for 3D dynamic target perception tasks (derived from the nuscene open source dataset), the AssA+AssQ evaluation index designed for 3D multi-target tracking tasks, the open-loop test evaluation index used for trajectory prediction tasks, and the (interactive) closed-loop test evaluation index used for planning tasks.
[0034] For 3D dynamic target perception tasks, NDS is used as a comprehensive evaluation indicator for 3D target detection results. mAP calculates the detection rate of each category on different distance measurement benchmarks. C is the target category set, D is a set of different distance thresholds, mATE refers to the positioning error between the target 3D box and the true value box, mASE refers to the size error between the target 3D box and the true value box, mAE refers to the orientation error between the target 3D box and the true value box, mAE refers to the speed error between the target 3D box and the true value box, and mAAE refers to the category error between the target 3D box and the true value box. a, b, c, d, e, f and g are real numbers and can be dynamically adjusted according to the key optimization indicators.
[0035] In specific implementation, the following steps are used to calculate the association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm:
[0036] Scale Parameters Where c is a matching pair, A(c) is the association accuracy between the true trajectory and the predicted trajectory of the matching pair c, TPA(c) is the complete consistency between the true trajectory frame and the tracking trajectory frame of the matching pair c, FNA(c) is the absence of the tracking trajectory frame on the true trajectory frame of the matching pair c or the matching with the tracking trajectory frame of other targets, and FPA(c) is the absence of the true trajectory point on the tracking trajectory frame of the matching pair c or the matching with the true trajectory frame of other targets. The association accuracy AssA is calculated by the scale parameter A(c), where the association accuracy TP is the set of matching pairs; generalized intersection-over-union loss Where GIoU(c) is the generalized intersection-over-union loss, T(c) is the tracking trajectory box in the matching pair, G(c) is the true value trajectory box in the matching pair, K is the minimum enclosing rectangle between T(c) and G(c), and U is the intersection between T(c) and G(c); the association quality AssQ is calculated by the generalized intersection-over-union loss GIoU(c), where the association quality
[0037] Specifically, for the 3D multi-target tracking task, the association accuracy AssA (AssociateAccuracy) and the association quality AssQ (AssociateQuality) are considered at the same time.
[0038] For all matched prediction boxes and detection boxes, they form a TP (True Positive) matching pair set and the matching pair c in it. For any true value trajectory, TPA (True Positive Associations) means that the true value trajectory box and the tracking trajectory box are completely consistent, and FNA (False Negative Association) means that there is no tracking trajectory box on the true value trajectory box or it matches the tracking trajectory box of other targets. For any tracking trajectory, FPA (False Positive Associations) means that there is no true value trajectory point on the tracking trajectory box or it matches the true value trajectory box of other targets. T(c) is the tracking trajectory box in the matching pair, G(c) is the true value trajectory box in the matching pair, K is the minimum enclosing rectangle between T(c) and G(c), and U is the intersection between T(c) and G(c).
[0039] In specific implementation, the following steps are used to calculate the open-loop test evaluation index of the target trajectory prediction task in the end-to-end algorithm:
[0040] The open-loop test evaluation index OLS_predict = (W1*ADEWB*M1+W2*AHEWB*M2+W3*FDEWB*M3+W4*FHEWB*M4)*MRWB, where ADEWB is the average displacement error within the limit, AHEWB is the average heading error within the limit, FDEWB is the final displacement error within the limit, FHEWB is the final heading error within the limit, MRWB is the error rate within the limit, W1, W2, W3 and W4 are weight coefficients, and M1, M2, M3 and M4 are multiplication coefficients for each indicator.
[0041] Specifically, W1, W2, W3, and W4 are weight coefficients (constants), and M1 to M4 are multiplication coefficients of each indicator (values are 0 or 1). The detailed parameter definitions are:
[0042] ADEWB (Average Displacement Error Within Bound) is the average displacement error within the boundary, AHEWB (Average heading error Within Bound) is the average heading error within the boundary, FDEWB (Final Displacement Error Within Bound) is the final displacement error within the boundary, and FHEWB (Final heading error (FHE) Within Bound) is the final heading error within the boundary.
[0043] Miss Rate Within Bound: Within a period of time, the number of times the error between the planner and the expert exceeds the maximum error threshold is 0 if it exceeds the number of misses threshold, otherwise it is 1;
[0044] M1: If the average position error between the planner and the expert results exceeds the maximum error threshold within a period of time, it is 0, otherwise it is 1;
[0045] M2: Within a period of time, if the final position error between the planner and the expert results exceeds the maximum error threshold, it is 0, otherwise it is 1;
[0046] M3: If the average orientation error between the planner and the expert results exceeds the maximum error threshold within a period of time, it is 0, otherwise it is 1;
[0047] M4: Within a period of time, if the final orientation error between the planner and the expert results exceeds the maximum error threshold, it is 0, otherwise it is 1.
[0048] In specific implementation, the closed-loop test evaluation index of the planning task in the end-to-end algorithm is calculated through the following steps:
[0049] The closed-loop test evaluation index OLS = (W1*DDC*M1+W2*TTCWB+W3*SLC+W4*EPAER+W5*EIC)*NEAFC*DAC*EIMP, where DDC is the score for whether to drive in the opposite direction, TTCWB is the score for the collision time within the limit, SLC is the speed limit score, EPAER is the expert trajectory completion ratio, EIC is the comfort score, NEAFC is the own-vehicle fault collision score, DAC is the drivable area crossing score, EIMP is the expert trajectory completion score, and W1, W2, W3, W4 and W5 are weight coefficients.
[0050] Specifically, W1~W5 are weight coefficients (constants), and other indicators are defined in detail as follows:
[0051] no_ego_at_fault_collisions Self-car at fault collision score: A collision is defined as an event where the bounding box of the ego intersects the bounding box of another target object. If a collision lasts for multiple frames, it is still considered a single collision, and the kinetic energy transfer at the time of the collision is calculated based on the first frame of the collision, which represents the intensity of the collision between the ego and the other agent. All target object tracks that collide will be removed when evaluating the indicator in future frames;
[0052] drivable_area_compliance: The ego vehicle should always drive within the mapped drivable area. This metric identifies frames where the ego vehicle drives outside the drivable area. If the ego vehicle size is estimated accurately, we allow it to slightly exceed the drivable area. If there is a frame where the maximum distance from the corner of the ego vehicle's bounding box to the nearest drivable area exceeds the selected threshold, the compliance score is 0, otherwise it is 1;
[0053] driving_direction_compliance is the score for driving in the opposite direction: This indicator is used to penalize the situation where the vehicle "drives into the opposite lane". It calculates the movement of the center of the vehicle in the driving direction within a 1-second time frame based on the baseline of the vehicle's lane or lane connection line. If the vehicle does not drive in the opposite direction beyond the compliance threshold, the score is 1; if the vehicle drives in the opposite direction beyond the violation threshold, the score is 0; otherwise, it is 0.5;
[0054] time_to_collision_within_bound: TTC is defined as the time until the ego vehicle and other tracks will collide if they continue at their current speed and heading. We only calculate TTC for tracks in front of the ego vehicle and cross traffic, as well as side tracks (when the ego vehicle footprint is not entirely in a single lane or lane connector, such as during a lane change) or when the ego vehicle is in an area marked as an intersection, where lanes may merge or expand into multiple lanes. If the predicted bounding boxes do not intersect within the selected time range in the future, TTC is set to infinity;
[0055] speed_limit_compliance: This metric evaluates whether the ego vehicle's speed exceeds the speed limit associated in the map. The speed limit is queried from the lane the ego vehicle is associated with. If the ego vehicle is associated with a lane connector, the speed limit is set to the larger of the inbound and outbound lane speed limits of that lane connector.
[0056] If the speed is exceeded, it is 0; if the speed is not exceeded, it is 1;
[0057] ego_progress_along_expert_routeExpert trajectory completion ratio: Evaluates the trajectory of the ego vehicle by comparing the proportion of its completion along the expert route in the scene. The expert route is a sequence of lanes and lane connectors that the expert moves through in the scene. The progress of each frame is calculated based on the progress along the expert route baseline when the ego vehicle is on the corresponding section of the expert route. The overlap of the ego vehicle and the expert in each frame in the scene is calculated to define the ego vehicle to expert progress ratio;
[0058] ego_is_making_progress Expert trajectory completion score: When the expert trajectory completion ratio is less than the (percentage) threshold, the score is 0, otherwise it is 1;
[0059] Ego_is_comfortable comfort score: If the speed, lateral acceleration, angular velocity, angular acceleration, longitudinal jerk, and lateral jerk are all within the threshold range during the driving distance, the score is 1, otherwise it is 0.
[0060] In the specific implementation, the following steps are performed to calculate the overall evaluation index of the end-to-end algorithm through the NDS evaluation index, association accuracy, association quality, open-loop test evaluation index and closed-loop test evaluation index:
[0061] Overall evaluation indicators Among them, NDS is the NDS evaluation index, AssA is the association accuracy, AssQ is the association quality, OLS_prediction is the open-loop test evaluation index, CLS is the closed-loop test evaluation index, k1 is the first weight coefficient, k2 is the second weight coefficient, and k3 is the third weight coefficient.
[0062] In specific implementation, the weight coefficient is adjusted according to customer needs through the following steps:
[0063] The first weight coefficient, the second weight coefficient and the third weight coefficient can be adjusted according to customer needs, where the customer demand is to obtain the weights of the 3D dynamic target perception task, 3D multi-target tracking task, target trajectory prediction task and planning task in the end-to-end algorithm when the customer uses the end-to-end algorithm; the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1.
[0064] In this embodiment, a computer device is provided, such as Figure 3 As shown, it includes a memory 301, a processor 302, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned sample selection methods for the end-to-end algorithm is implemented.
[0065] Specifically, the computer device may be a computer terminal, a server or a similar computing device.
[0066] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned sample selection methods for the end-to-end algorithm.
[0067] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0068] Based on the same inventive concept, a sample selection device for an end-to-end algorithm is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the sample selection device for the end-to-end algorithm is similar to that of the sample selection method for the end-to-end algorithm, the implementation of the sample selection device for the end-to-end algorithm can refer to the implementation of the sample selection method for the end-to-end algorithm, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0069] Figure 4 is a structural block diagram of a sample selection device for an end-to-end algorithm according to an embodiment of the present invention. Figure 4 As shown, it includes: a primary sample selection module 401, an evaluation index calculation module 402 and a secondary sample selection module 403. The structure is described below.
[0070] The initial sample selection module 401 is used to input the raw sensor data into the VLM large model, extract scene description words, input the scene description words, the raw sensor data, the local map data and the ego vehicle posture data into the multimodal scene classification model, and divide the raw sensor data into in-service data and out-of-service data;
[0071] Evaluation index calculation module 402, used to calculate the NDS evaluation index of the task of acquiring 3D dynamic target perception in the end-to-end algorithm, calculate the association accuracy and association quality of the task of 3D multi-target tracking in the end-to-end algorithm, calculate the open-loop test evaluation index of the task of target trajectory prediction in the end-to-end algorithm, and calculate the closed-loop test evaluation index of the planning task in the end-to-end algorithm;
[0072] The secondary sample selection module 403 is used to calculate the overall evaluation index of the end-to-end algorithm through the NDS evaluation index, association accuracy, association quality, open-loop test evaluation index and closed-loop test evaluation index, and regard the data in the business data that meets the overall evaluation index as high-value data, and use the high-value data as the training set of the end-to-end algorithm.
[0073] In one embodiment, the evaluation index calculation module includes:
[0074] The detection rate calculation unit is used to calculate the average detection rate mAP of each category on different distance measurement benchmarks, where: Among them, C is the target category set, D is the different distance threshold sets, AP c,d is the accuracy of the target category c at different distance thresholds d;
[0075] The NDS evaluation index calculation unit is used to calculate the NDS evaluation index through the positioning error between the target 3D frame and the true value frame, the size error between the target 3D frame and the true value frame, the orientation error between the target 3D frame and the true value frame, the speed error between the target 3D frame and the true value frame, the category error between the target 3D frame and the true value frame, and the average detection rate mAP of each category on different distance measurement benchmarks, where:
[0076]
[0077] is the positioning error between the target object 3D box and the true value box, mASE is the size error between the target object 3D box and the true value box, mAOE is the orientation error between the target object 3D box and the true value box, mAVE is the velocity error between the target object 3D box and the true value box, mAAE is the category error between the target object 3D box and the true value box, a, b, c, d, e, f and g are all real numbers that can be adjusted dynamically.
[0078] In one embodiment, the evaluation index calculation module further includes:
[0079] Calculate the scale parameter unit, used for scale parameter Wherein, c is a matching pair, A(c) is the correlation accuracy between the true trajectory and the predicted trajectory of the matching pair c, TPA(c) is the complete consistency between the true trajectory frame and the tracking trajectory frame of the matching pair c, FNA(c) is the absence of the tracking trajectory frame on the true trajectory frame of the matching pair c or the matching with the tracking trajectory frame of other targets, and FPA(c) is the absence of the true trajectory point on the tracking trajectory frame of the matching pair c or the matching with the true trajectory frame of other targets;
[0080] The association unit is used to calculate the association accuracy AssA by using the proportional parameter A(c), where the association accuracy TP is the set of matching pairs;
[0081] Calculates generalized intersection-over-union unit for generalized intersection-over-union loss Among them, GIoU(c) is the generalized intersection-over-union loss, T(c) is the tracking trajectory box in the matching pair, G(c) is the true value trajectory box in the matching pair, K is the minimum enclosing rectangle between T(c) and G(c), and U is the intersection between T(c) and G(c);
[0082] The association quality calculation unit is used to calculate the association quality AssQ through the generalized intersection-over-union loss GIoU(c), where the association quality
[0083] In one embodiment, the evaluation index calculation module further includes:
[0084] Open-loop test evaluation index calculation unit, used for open-loop test evaluation index OLS_predict = (W1*ADEWB*M1+W2*AHEWB*M2+W3*FDEWB*M3+W4*
[0085] FHEWB*M4)*MRWB, where ADEWB is the average displacement error within the limit, AHEWB is the average heading error within the limit, FDEWB is the final displacement error within the limit, FHEWB is the final heading error within the limit, MRWB is the error rate within the limit, W1, W2, W3 and W4 are weight coefficients, and M1, M 2, M 3 and M 4 are multiplication coefficients for each indicator.
[0086] In one embodiment, the evaluation index calculation module further includes:
[0087] Closed-loop test evaluation index calculation unit, used for closed-loop test evaluation index OLS = (W1*DDC*M1+W2*TTCWB+W3*SLC+W4*EPAER+W5*EIC)*NEAFC*
[0088] DAC*EIMP, where DDC is the score for driving in the opposite direction, TTCWB is the score for the time to collision within the limit, SLC is the speed limit score, EPAER is the expert trajectory completion ratio, EIC is the comfort score, NEAFC is the self-vehicle fault collision score, DAC is the drivable area crossing score, EIMP is the expert trajectory completion score, and W1, W2, W3, W4 and W5 are weight coefficients.
[0089] In one embodiment, the secondary sample selection module further includes:
[0090] Overall evaluation index calculation unit, used for overall evaluation index Among them, NDS is the NDS evaluation index, AssA is the association accuracy, AssQ is the association quality, OLS_prediction is the open-loop test evaluation index, CLS is the closed-loop test evaluation index, k1 is the first weight coefficient, k2 is the second weight coefficient, and k3 is the third weight coefficient.
[0091] In one embodiment, the overall evaluation index calculation unit is also used for the first weight coefficient, the second weight coefficient and the third weight coefficient to be adjusted according to customer needs, wherein the customer demand is to obtain the weights of the 3D dynamic target perception task, the 3D multi-target tracking task, the target trajectory prediction task and the planning task in the end-to-end algorithm when the customer uses the end-to-end algorithm; the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1.
[0092] The embodiments of the present invention achieve the following technical effects:
[0093] Existing difficult sample selection strategies often only design evaluation indicators for a single task and select difficult samples based on the indicators. For example, environmental perception tasks often only need to evaluate the detection accuracy of dynamic / static targets, while planning tasks only consider factors such as the safety and comfort of predicted trajectories. Since the end-to-end task will include environmental perception tasks, trajectory prediction tasks and self-vehicle planning tasks integrated into the same model, the evaluation indicators for a single task often cannot be evaluated scientifically and comprehensively. The sample selection method of the embodiment of the present invention comprehensively considers evaluation indicators such as dynamic target perception, multi-target tracking, open-loop test evaluation and closed-loop test evaluation; if only one set of evaluation indicators is used for selection, the data will often be classified into invalid samples and valid samples based only on the evaluation indicators of the model on different data, and the granularity of data classification is relatively coarse. At the same time, the selected data set may be helpful for model improvement, but it exceeds the boundary of the end-to-end algorithm. A two-stage selection process (initial sample selection and secondary sample selection) is introduced. In the first stage, the data set is first screened and classified based on the actual scene boundary, and only the business scenes that the end-to-end algorithm needs to cover are retained to converge the business boundary. Then, the comprehensive evaluation index designed in this patent is applied to the data within the business scope for sample selection; the scene classification strategy of the first stage (initial sample selection) of the embodiment of the present invention, the existing strategy often only classifies the scene according to the posture information of the vehicle, and there are too few screening considerations. There is a high possibility that two scenes with large differences will be classified as the same scene. The multimodal scene classification network based on the deep learning network of the embodiment of the present application uses the original sensor data, local map information, and the scene description words generated by LLM (language large model) as the input of the multimodal scene classification network, outputs the classification of the original data, and then can filter out the data outside the business boundary; through the two-stage high-value sample selection process and the multimodal business boundary data selection strategy combined with the large model scene description words, the end-to-end algorithm is comprehensively evaluated from the dimensions of perception, tracking, prediction and planning, and the robustness of the end-to-end algorithm within the business boundary data is improved.
[0094] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.
[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A sample selection method for an end-to-end algorithm, characterized in that: include: Input the raw sensor data into the VLM large model, extract scene description words, input the scene description words, the raw sensor data, local map data and vehicle posture data into a multimodal scene classification model, and divide the raw sensor data into in-service data and out-of-service data; Calculate the NDS evaluation index of the 3D dynamic target perception task in the end-to-end algorithm, calculate the association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm, calculate the open-loop test evaluation index of the target trajectory prediction task in the end-to-end algorithm, and calculate the closed-loop test evaluation index of the planning task in the end-to-end algorithm; The overall evaluation index of the end-to-end algorithm is calculated by using the NDS evaluation index, the association accuracy, the association quality, the open-loop test evaluation index and the closed-loop test evaluation index, and the data in the business data that meets the overall evaluation index is used as high-value data, and the high-value data is used as a training set for the end-to-end algorithm.
2. The sample selection method for the end-to-end algorithm according to claim 1, characterized in that: The NDS evaluation index of the 3D dynamic target perception task in the end-to-end algorithm is calculated, including: Calculate the average detection rate mAP of each category on different distance measurement benchmarks, where Among them, C is the target category set, D is the different distance threshold sets, AP c,d is the accuracy of the target category c at different distance thresholds d; The NDS evaluation index is calculated by the positioning error between the target 3D frame and the true value frame, the size error between the target 3D frame and the true value frame, the orientation error between the target 3D frame and the true value frame, the speed error between the target 3D frame and the true value frame, the category error between the target 3D frame and the true value frame, and the average detection rate mAP of each category on different distance measurement benchmarks, where: mATE is the positioning error between the target object 3D frame and the true value frame, mASE is the size error between the target object 3D frame and the true value frame, mAOE is the orientation error between the target object 3D frame and the true value frame, mAVE is the velocity error between the target object 3D frame and the true value frame, mAAE is the category error between the target object 3D frame and the true value frame, and a, b, c, d, e, f and g are all real numbers that can be dynamically adjusted.
3. The sample selection method for the end-to-end algorithm according to claim 1, characterized in that: The association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm are calculated, including: Scale Parameters Wherein, c is a matching pair, A(c) is the correlation accuracy between the true trajectory and the predicted trajectory of the matching pair c, TPA(c) is the complete consistency between the true trajectory frame and the tracking trajectory frame of the matching pair c, FNA(c) is the absence of the tracking trajectory frame on the true trajectory frame of the matching pair c or the matching with the tracking trajectory frame of other targets, and FPA(c) is the absence of the true trajectory point on the tracking trajectory frame of the matching pair c or the matching with the true trajectory frame of other targets; The association accuracy AssA is calculated by the scale parameter A(c), where TP is the set of matching pairs; Generalized Intersection-Union Loss Among them, GIoU(c) is the generalized intersection-over-union loss, T(c) is the tracking trajectory box in the matching pair, G(c) is the true value trajectory box in the matching pair, K is the minimum enclosing rectangle between T(c) and G(c), and U is the intersection between T(c) and G(c); The association quality AssQ is calculated by the generalized intersection-over-union loss GIoU(c), where 4. The sample selection method for the end-to-end algorithm according to claim 1, characterized in that: The open-loop test evaluation indicators of the target trajectory prediction task in the end-to-end algorithm are calculated, including: The open-loop test evaluation index OLS_predict = (W1*ADEWB*M1+W2*AHEWB*M2+W3*FDEWB*M3+W4*FHEWB*M4)*MRWB, where ADEWB is the average displacement error within the limit, AHEWB is the average heading error within the limit, FDEWB is the final displacement error within the limit, FHEWB is the final heading error within the limit, MRWB is the error rate within the limit, W1, W2, W3 and W4 are weight coefficients, and M1, M2, M3 and M4 are multiplication coefficients for each indicator.
5. The sample selection method for the end-to-end algorithm according to claim 1, characterized in that: The closed-loop test evaluation index of the planning task in the end-to-end algorithm is calculated, including: The closed-loop test evaluation index OLS = (W1*DDC*M1+W2*TTCWB+W3*SLC+W4*EPAER+W5*EIC)*NEAFC*DAC*EIMP, where DDC is the score for whether to drive in the opposite direction, TTCWB is the score for the collision time within the limit, SLC is the speed limit score, EPAER is the expert trajectory completion ratio, EIC is the comfort score, NEAFC is the own-vehicle fault collision score, DAC is the drivable area crossing score, EIMP is the expert trajectory completion score, and W1, W2, W3, W4 and W5 are weight coefficients.
6. The sample selection method for the end-to-end algorithm according to claim 1, characterized in that: The overall evaluation index of the end-to-end algorithm is calculated by the NDS evaluation index, the association accuracy, the association quality, the open-loop test evaluation index, and the closed-loop test evaluation index, including: Overall evaluation indicators Among them, NDS is the NDS evaluation index, AssA is the association accuracy, AssQ is the association quality, OLS_prediction is the open-loop test evaluation index, CLS is the closed-loop test evaluation index, k1 is the first weight coefficient, k2 is the second weight coefficient, and k3 is the third weight coefficient.
7. The sample selection method for the end-to-end algorithm according to claim 6, characterized in that: Also includes: The first weight coefficient, the second weight coefficient and the third weight coefficient can be adjusted according to customer needs, wherein the customer needs are the weights of the 3D dynamic target perception task, the 3D multi-target tracking task, the target trajectory prediction task and the planning task in the end-to-end algorithm when the customer uses the end-to-end algorithm; The sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1.
8. A sample selection device for an end-to-end algorithm, characterized in that: include: The initial sample selection module is used to input the raw sensor data into the VLM large model, extract scene description words, input the scene description words, the raw sensor data, local map data and ego vehicle posture data into the multimodal scene classification model, and divide the raw sensor data into in-service data and out-of-service data; An evaluation index calculation module is used to calculate the NDS evaluation index of the task of acquiring 3D dynamic target perception in the end-to-end algorithm, calculate the association accuracy and association quality of the 3D multi-target tracking task in the end-to-end algorithm, calculate the open-loop test evaluation index of the target trajectory prediction task in the end-to-end algorithm, and calculate the closed-loop test evaluation index of the planning task in the end-to-end algorithm; The secondary sample selection module is used to calculate the overall evaluation index of the end-to-end algorithm through the NDS evaluation index, the association accuracy, the association quality, the open-loop test evaluation index and the closed-loop test evaluation index, and take the data in the business data that meets the overall evaluation index as high-value data, and use the high-value data as the training set of the end-to-end algorithm.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the sample selection method for the end-to-end algorithm according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the sample selection method for an end-to-end algorithm according to any one of claims 1 to 7.