Evaluation method for automatic driving planning module

Abstract

The invention relates to the technical field of automatic driving, and particularly discloses an evaluation method for an automatic driving planning module, and the method comprises the steps: obtaining evaluation data of a plurality of different scenes; each piece of evaluation data has a plurality of labels; evaluating the evaluation data according to preset indexes to obtain index scores of the preset indexes; obtaining a weight coefficient corresponding to each preset index according to the scene; and obtaining an evaluation score of the evaluation data according to the weight coefficient and the evaluation score corresponding to each preset index.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an evaluation method for an autonomous driving planning module. Background Art

[0002] The planning module directly or indirectly obtains sensor information and outputs the future movement trajectory of the vehicle, which is a key link in the entire autonomous driving system. Accurate offline evaluation of the planning module is the basis for ensuring the safe online operation of the autonomous driving system, and is also an important means to discover planning algorithm problems and improve algorithm performance.

[0003] Existing technologies can provide a basic evaluation of autonomous driving planning modules, but there are still several areas that need improvement.

[0004] The most common evaluation method is to score based on the average Euclidean distance between the planning result and the true trajectory. The smaller the distance, the better the planning algorithm performance, and vice versa. However, the evaluation strategy based solely on the mathematical similarity of two trajectories is not enough to comprehensively evaluate the quality of the trajectory. On the one hand, this indicator does not evaluate the details of a single trajectory itself, nor does it fully consider the interaction with the surrounding environment, and lacks the comprehensiveness of performance evaluation; on the other hand, two different trajectories to be evaluated may be close in mathematical similarity to the true trajectory, but very different in actual use, which also makes this method not objective enough.

[0005] Subsequent methods have added other evaluation indicators such as "smoothness and collision" on the basis of "proximity", which has improved the evaluation effect to a certain extent. However, in the same scene, the influence of different evaluation indicators may be mutually exclusive, and the same set of weight coefficients cannot be well applied to multiple evaluation scenes. Therefore, the performance evaluation of planning algorithms in different scenes is still not accurate enough. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide an evaluation method for an autonomous driving planning module, which aims to perform multi-scenario, multi-indicator offline evaluation of the planning module of an autonomous driving system to ensure its safe online operation, and to improve the overall performance of the planning module by discovering potential algorithm defects, thereby achieving continuous optimization and iteration of the algorithm.

[0007] The present invention provides an evaluation method for an autonomous driving planning module, comprising:

[0008] Acquire evaluation data of multiple different scenarios; each of the evaluation data has multiple labels;

[0009] Evaluate the evaluation data according to preset indicators to obtain the indicator scores of each preset indicator;

[0010] Obtaining weight coefficients corresponding to each preset indicator according to the scenario;

[0011] The evaluation score of the evaluation data is obtained according to the weight coefficients and evaluation scores corresponding to the preset indicators.

[0012] In a possible implementation, obtaining evaluation data of multiple different scenarios includes:

[0013] Get the real car dataset;

[0014] Screening the real vehicle data set according to the scenario to obtain evaluation data for multiple different scenarios;

[0015] The label of the evaluation data is set according to the road type, traffic participant type, and vehicle status type in the evaluation data.

[0016] In one possible implementation, the preset indicators include: a proximity indicator that characterizes the difference between the evaluated trajectory planning and the actual trajectory, a centrality indicator that characterizes the degree of deviation between the planned trajectory and the center line of the lane, a smoothness indicator that characterizes the smoothness of the trajectory, a safety indicator that characterizes the collision risk, and a stability indicator that characterizes the inter-frame stability of the planning result.

[0017] In a possible implementation, the evaluating the evaluation data according to the preset indicators to obtain the indicator scores of the preset indicators includes:

[0018] The proximity index speed f is calculated according to the following formula g (t):

[0019] f g (t) = θ v |v(t)-v gt (t)|+θ d |d(t)|

[0020] In the formula, θ v Represents the evaluation weight of the speed approach, θ d Represent the evaluation weights of horizontal differences, v gt (t) represents the velocity value corresponding to the true trajectory at time t, v(t) represents the velocity value corresponding to the evaluated trajectory at time t, and d(t) represents the lateral difference between the true trajectory and the evaluated trajectory at time t.

[0021] In a possible implementation, the method further includes:

[0022] The centering index f is calculated according to the following formula l (t):

[0023] f l (t)=|l(t)|

[0024] Where l(t) represents the lateral deviation of the vehicle relative to the center line of the lane at time t.

[0025] In a possible implementation, the method further includes:

[0026] The smoothness index f is calculated according to the following formula s (t):

[0027] f s (t) = θ s |j(t) s |+θ l |(j(t) l |

[0028] In the formula, θ s represents the evaluation weight of longitudinal smoothness, θ l Represents the evaluation weight of lateral smoothness, j(t) s represents the longitudinal acceleration at time t, j(t) l represents the lateral acceleration at time t.

[0029] In a possible implementation, the method further includes:

[0030] The safety index f is calculated according to the following formula c (e(t), o(t)):

[0031]

[0032] Where e(t) represents the position of the vehicle at time t, o(i, t) represents the position of the i-th obstacle at time t, and α is an auxiliary coefficient less than 0.

[0033] In a possible implementation, the method further includes:

[0034] The stability index f is calculated according to the following formula a :

[0035] f a =θ a |c a |+θ t |c t |+θ b |c b |

[0036] In the formula, θ w Represents the influence weight of the steering wheel angle on the stability index, θ a represents the weight of the accelerator pedal on the stability index, θ b represents the influence weight of the brake pedal on the stability index, c wrepresents the absolute average change in steering wheel angle, c a represents the absolute average change in the accelerator pedal, c b Represents the absolute average change in brake pedal pressure.

[0037] In a possible implementation manner, before obtaining the weight coefficient corresponding to each preset indicator according to the scenario, the method further includes:

[0038] A fuzzy table is used to establish a corresponding relationship between the scene and the weight coefficient of the preset indicator;

[0039] The weight coefficient w of the preset indicator is determined according to the following formula: f，g :

[0040]

[0041] In the formula, σ i represents the correction gain of the i-th label for the preset index, w g represents the initial value of the preset indicator, and n represents the number of tags.

[0042] In a possible implementation, the method further includes:

[0043] The evaluation scores are output and the evaluation scores are visualized.

[0044] The evaluation method for the autonomous driving planning module provided by the present invention is used to perform multi-scenario and multi-indicator evaluation on the output results of the autonomous driving planning module, and the weight of each indicator is correlated with the scenario characteristics, so as to evaluate the characteristics of the trajectory with emphasis, thereby solving the problem that the traditional trajectory evaluation method is not comprehensive and objective enough in evaluating the performance of the planning module. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of an evaluation method provided by an embodiment of the present invention;

[0046] Figure 2 An architectural diagram of an evaluation method provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of solving lateral deviation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0049] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0050] Figure 1 A schematic diagram of a flow chart of an evaluation method provided by an embodiment of the present invention, Figure 2 The architecture diagram of the evaluation method provided by the embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the present invention provides an evaluation method for an autonomous driving planning module, comprising:

[0051] Step S1, obtaining evaluation data of multiple different scenarios; each evaluation data has multiple labels;

[0052] In a possible implementation, obtaining evaluation data for multiple different scenarios includes:

[0053] Get the real car dataset;

[0054] Filter the real vehicle data set according to the scenario to obtain evaluation data for multiple different scenarios;

[0055] The evaluation data is labeled according to the road type, traffic participant type, and vehicle status type in the evaluation data.

[0056] In one example, a specific data set is screened according to the scene in the real vehicle data set, and exported in .db slice format, and the duration of a single evaluation data is set to 20s. In order to ensure the scene generalization and scene representativeness of the evaluation data, labels are set according to road type, traffic participant type, and vehicle status type when constructing the evaluation data set. Different scene categories are shown in Tables 1 to 3. Table 1 is the road type, Table 2 is the traffic participant type, and Table 3 is the vehicle status type.

[0057] Table 1

[0058]

[0059] Table 2

[0060]

[0061]

[0062] Table 3

[0063]

[0064] Step S2, evaluating the evaluation data according to the preset indicators to obtain the indicator scores of each preset indicator;

[0065] In a possible implementation, in order to comprehensively evaluate the output trajectory of the planning module, the present invention introduces multiple indicators for comprehensive evaluation. The preset indicators include: a proximity indicator that characterizes the difference between the planned evaluation trajectory and the actual trajectory, a centrality indicator that characterizes the degree of deviation between the planned trajectory and the center line of the lane, a smoothness indicator that characterizes the smoothness of the trajectory, a safety indicator that characterizes the risk of collision, and a stability indicator that characterizes the inter-frame stability of the planning result. After solving and obtaining the value of each preset indicator, the values ​​of each preset indicator are weighted averaged to obtain the final evaluation score. The calculation details of each indicator are as follows:

[0066] Evaluate the test data according to the preset indicators, and obtain the indicator scores of each preset indicator, including:

[0067] The proximity index speed f is calculated according to the following formula g (t):

[0068] f g (t) = θ v |v(t)-v gt (t)|+θ d |d(t)|

[0069] In the formula, θ v Represents the evaluation weight of the speed approach, θ d Represent the evaluation weights of horizontal differences, v gt (t) represents the velocity value corresponding to the true trajectory at time t, v(t) represents the velocity value corresponding to the evaluated trajectory at time t, and d(t) represents the lateral difference between the true trajectory and the evaluated trajectory at time t, which is calculated using the vector cross product, as follows: Figure 3 shown.

[0070] Figure 3 In the equation, the upper arc is the true trajectory, the lower arc is the evaluation trajectory, O represents the position of the true trajectory at time t-1, B represents the position of the evaluation trajectory at time t, and C represents the position of the true trajectory at time t. The horizontal difference between B and C, BD, is d(t) in the above formula, and the calculation formula is as follows:

[0071]

[0072] The centrality index f is calculated according to the following formula l (t):

[0073] f l (t)=|l(t)|

[0074] Where l(t) represents the lateral deviation of the vehicle relative to the center line of the lane at time t.

[0075] The smoothness index f is calculated according to the following formula s (t):

[0076] f s (t) = θ s |j(t) s |+θ l |j(t) l |

[0077] In the formula, θ s represents the evaluation weight of longitudinal smoothness, θ l Represents the evaluation weight of lateral smoothness, j(t) s represents the longitudinal acceleration at time t, j(t) l represents the lateral acceleration at time t.

[0078] The safety index f is calculated according to the following formula c (e(t), o(t)):

[0079]

[0080] Where e(t) represents the position of the vehicle at time t, o(i, t) represents the position of the i-th obstacle at time t, and α is an auxiliary coefficient less than 0.

[0081] The stability index f is calculated according to the following formula a :

[0082] f a =θ a |c a |+θ t |c t |+θ b |c b |

[0083] In the formula, θ w Represents the influence weight of the steering wheel angle on the stability index, θ a represents the weight of the accelerator pedal on the stability index, θ b represents the influence weight of the brake pedal on the stability index, c w represents the absolute average change in steering wheel angle, c a represents the absolute average change in the accelerator pedal, c b Represents the absolute average change in brake pedal pressure.

[0084] Step S3, obtaining the weight coefficient corresponding to each preset indicator according to the scenario;

[0085] In a possible implementation, before obtaining the weight coefficient corresponding to each preset indicator according to the scenario, the following is also included:

[0086] A fuzzy table is used to establish the corresponding relationship between the scene and the weight coefficient of the preset indicator;

[0087] Determine the weight coefficient w of the preset indicator according to the following formula f，g :

[0088]

[0089] In the formula, σ i represents the correction gain of the i-th label for the preset indicator, and the artificially specified value range is [0,2], w g Represents the initial value of the preset indicator, and n represents the number of labels.

[0090] In order to focus on evaluating certain characteristics of the planned trajectory in different test scenarios, it is necessary to set different evaluation index weight coefficients according to different test scenarios. For example, in a high-speed straight-driving scenario where there are motor vehicles in adjacent lanes that are close to each other, it is usually expected that the planned trajectory will show a certain degree of avoidance tendency to ensure driving safety. In this case, the weight of the safety index should be increased and the weight of the neutrality index should be reduced. Each serial number label corresponds to a set of correction gains for the weight coefficients. For the same test scenario, it may have multiple serial number labels, such as the vehicle is on a straight highway at a speed of 80km / h, there is a motor vehicle merging in front and there is a motor vehicle in the adjacent lane behind that is close.

[0091] Step S4, obtaining the evaluation score of the evaluation data according to the weight coefficient and evaluation score corresponding to each preset indicator.

[0092] In a possible implementation, a weighted average is performed on the basis of the initial value of the preset indicator to obtain the final evaluation score f of the evaluation data. t , the formula is as follows:

[0093] f t =ω l f l +w g f g +w c f c +w s f s +w a f a

[0094] In the formula, w l 、w g 、w c 、w s 、w aThey are the weight coefficients corresponding to the preset indicators, f l 、f g 、f c 、f s 、f a They are the initial values ​​corresponding to the preset indicators.

[0095] In a possible implementation, the evaluation score is output and visualized.

[0096] In a possible implementation, the information of the vehicle in online operation is recorded based on the topic message of the ROS system; then, the input information is given in the form of data playback in the software simulation platform, and the automatic driving planning module and the control module output the results; then, the evaluation method of the present invention is used to evaluate the operation results, and finally the evaluation score is output.

[0097] The evaluation method for the autonomous driving planning module provided by the present invention mainly includes the following contents:

[0098] 1) When constructing the evaluation scenario of the trajectory, the present invention classifies it from three dimensions: road type, traffic participants, and vehicle status, and further subdivides each category according to the topological shape or motion state, thereby ensuring the universality of the evaluation scenario as much as possible and comprehensively testing the applicability of the planned trajectory.

[0099] 2) When constructing the evaluation index of the trajectory, the present invention proposes five subcategories: proximity index, centrality index, smoothness index, safety index and stability index, and finally weights them to obtain the final evaluation value. Compared with the existing technology, the present invention is innovative in the design of some indicators. For example, in the design of the proximity index, the simple Euclidean distance is not used to judge the difference between the planned trajectory and the actual trajectory, but the vector cross product is used to decouple the two into lateral differences and speed differences, and the details of the trajectory performance are described in more detail; for example, in the stability index, the present invention starts from the perspective of the entire system and uses the output signal of the control module to evaluate the timing stability of the planned trajectory, thereby verifying the adaptability of the planning module to the downstream control module. In addition, the present invention innovatively uses a fuzzy table to establish a connection between the weight coefficient and the scene type, which can realize the focused inspection of certain key performances of the planning module in each scene.

[0100] 3) When constructing the simulation evaluator, the present invention uses the same programming language as the actual vehicle to implement the simulator, and records the actual vehicle online data and inputs it into the planning module in a playback manner, so as to ensure the consistency between the offline evaluation and the online performance as much as possible. Finally, the planning trajectory and control signal output are output using a visualization platform, so that the performance of the planning results can be observed intuitively and conveniently.

[0101] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An evaluation method for an autonomous driving planning module, characterized in that: include: Obtain evaluation data for multiple different scenarios; Each of the evaluation data has multiple labels; Evaluate the evaluation data according to preset indicators to obtain the indicator scores of each preset indicator; Obtaining weight coefficients corresponding to each preset indicator according to the scenario; The evaluation score of the evaluation data is obtained according to the weight coefficients and evaluation scores corresponding to the preset indicators.

2. The evaluation method according to claim 1, characterized in that: The obtaining of evaluation data for multiple different scenarios includes: Get the real car dataset; Screening the real vehicle data set according to the scenario to obtain evaluation data for multiple different scenarios; The label of the evaluation data is set according to the road type, traffic participant type, and vehicle status type in the evaluation data.

3. The evaluation method according to claim 1, characterized in that: The preset indicators include: a proximity indicator that characterizes the difference between the planned evaluation trajectory and the actual trajectory, a centrality indicator that characterizes the degree of deviation between the planned trajectory and the center line of the lane, a smoothness indicator that characterizes the smoothness of the trajectory, a safety indicator that characterizes the collision risk, and a stability indicator that characterizes the inter-frame stability of the planning result.

4. The evaluation method according to claim 3, characterized in that: The step of evaluating the evaluation data according to the preset indicators to obtain the indicator scores of the preset indicators includes: The proximity index speed f is calculated according to the following formula g (t): f g (t)=θ v |v(t)-v gt (t)|+θ d |d(t)| In the formula, θ v Represents the evaluation weight of the speed approach, θ d Represent the evaluation weights of horizontal differences, v gt (t) represents the velocity value corresponding to the true trajectory at time t, v(t) represents the velocity value corresponding to the evaluated trajectory at time t, and d(t) represents the lateral difference between the true trajectory and the evaluated trajectory at time t.

5. The evaluation method according to claim 4, characterized in that: Also includes: The centering index f is calculated according to the following formula l (t): f l (t)=|l(t)| Where l(t) represents the lateral deviation of the vehicle relative to the center line of the lane at time t.

6. The evaluation method according to claim 5, characterized in that: Also includes: The smoothness index f is calculated according to the following formula s (t): f s (t)=θ s |j(t) s |+θ l |j(t) l | In the formula, θ s represents the evaluation weight of longitudinal smoothness, θ l Represents the evaluation weight of lateral smoothness, j(t) s represents the longitudinal acceleration at time t, j(t) l represents the lateral acceleration at time t.

7. The evaluation method according to claim 6, characterized in that: Also includes: The safety index f is calculated according to the following formula c (e(t), o(t)): Where e(t) represents the position of the vehicle at time t, o(i,t) represents the position of the i-th obstacle at time t, and α is an auxiliary coefficient less than 0.

8. The evaluation method according to claim 7, characterized in that: Also includes: The stability index f is calculated according to the following formula a : f a =θ a |c a |+θ t |c t |+θ b |c b | In the formula, θ w Represents the influence weight of the steering wheel angle on the stability index, θ a represents the weight of the accelerator pedal on the stability index, θ b represents the influence weight of the brake pedal on the stability index, c w represents the absolute average change in steering wheel angle, c a represents the absolute average change in the accelerator pedal, c b Represents the absolute average change in brake pedal pressure.

9. The evaluation method according to claim 1, characterized in that: Before obtaining the weight coefficients corresponding to the preset indicators according to the scenario, the method further includes: A fuzzy table is used to establish a corresponding relationship between the scene and the weight coefficient of the preset indicator; The weight coefficient w of the preset indicator is determined according to the following formula: f，g : In the formula, σ i represents the correction gain of the i-th label for the preset index, w g represents the initial value of the preset indicator, and n represents the number of tags.

10. The evaluation method according to claim 1, characterized in that: Also includes: The evaluation scores are output and the evaluation scores are visualized.

Images