An automatic parking motion planning method based on MCTS algorithm
By adopting MCTS algorithm and policy neural network in the automatic parking system, combining imitation learning and reinforcement learning, unified the storage and adjustment stages of parallel parking, the problems of low efficiency and computational intensiveness in the existing technology are solved, and high efficiency is achieved , fast automatic parking motion planning, suitable for small and large warehouse locations, and eliminate longitudinal and heading angle errors.
Patent Information
- Application Number
- CN202111118963.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-09-24
AI Technical Summary
The existing automatic parking system is inefficient and has intensive computing resources in parallel parking scenarios, making it difficult to achieve efficient multi-objective optimization, especially the predefined longitudinal action speed limits parking accuracy and efficiency.
Using the automatic parking motion planning method based on MCTS algorithm, combined with imitation learning and reinforcement learning, a continuous optimal trajectory data set is generated through a strategic neural network, unified the entry and in-store adjustment stages, and eliminated longitudinal and Heading angle error realizes the selection of vehicle speed movement and steering angle correction.
It improves the efficiency and speed of automatic parallel parking, reduces the calculation amount, adapts to small and large warehouse locations, avoids in-situ steering, shortens training time, and improves parking accuracy and scope of application.
Smart Images

Figure CN114906128B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving, and in particular to an automatic parking motion planning method based on MCTS algorithm. Background Art
[0002] With the continuous development of automobile intelligence and the increasing demand for traffic safety, autonomous driving technology is showing a booming trend. The automatic parking system is generally considered to be the most promising application for the commercialization of autonomous driving in the near future. In the analysis of the causes of road traffic accidents, about 23% of accidents are related to parking scenarios, and 75% of parking accidents are related to the reversing process. The automatic parking system relies on computer systems and sensors to liberate human drivers from the process of controlling parking. Therefore, the automatic parking system can improve safety, reduce the occurrence of traffic accidents, and save time and energy, while parallel parking is the most difficult situation in automatic parking.
[0003] The automatic parking system generally consists of two modules: storage location detection and motion planning. The starting location detection module outputs the available storage location information, including the storage location length, storage location depth and the state of the vehicle in the storage location coordinate system. The motion planning module, which guides the vehicle into the detected storage location space, is a key module that determines the intelligence level of the intelligent system.
[0004] Motion planning methods can be divided into classical branches and artificial intelligence-based branches;
[0005] Classic branching methods generally include geometry-based methods, heuristic search-based methods, and optimization-based methods. The geometry-based method is the most commonly used method, which uses a fixed-shape curve as the parking path. Common curves include RS curves, clothoid curves, and β-spline curves. The representative of the heuristic search method is the Hybrid-A* algorithm, which only discretely samples the control space to ensure that the scale of the search is not too large, but it may also fail. The definition of the heuristic function has a great influence on the final result. RRT is also a path planning method that uses heuristic information. The generated path needs to be smoothed by polynomial curve fitting or β-spline. The optimization-based method transforms the planning problem into a numerical optimization problem through precise mathematical modeling, and considers safety, comfort and other goals in the objective function to solve the parking problem, but due to the need for a large amount of computing resources, it hinders its widespread application in actual vehicles.
[0006] Recently, intelligent methods have become increasingly popular, and the basic tool of this branch is artificial neural network (ANN). ANN can directly map the state of the vehicle to the action, but using ANN alone cannot guarantee parking performance. The quality of training data is its bottleneck. In addition, ANN needs to be designed to be very deep and large. Since the advent of AlphaGo and AlphaGo Zeros, Reinforcement Learning (RL) has attracted more and more attention from researchers in the field of autonomous driving, especially in the field of motion planning. "Zhang P, Xiong L, Yu Z, et al. Reinforcement learning-based end-to-end parking for automatic parking system[J]. Sensors, 2019, 19(18): 3996." proposed a model-free reinforcement learning method based on deep deterministic policy gradient (DDPG) to achieve end-to-end automatic perpendicular parking. Compared with traditional path planning, it can achieve higher final parking accuracy, but perpendicular parking is much easier than parallel parking. The learning process of this method is cumbersome, and the vehicle speed in the method is constant. "Du Z, Miao Q, Zong C. Trajectory Planning for Automated Parking Systems Using Deep Reinforcement Learning [J]. International Journal of Automotive Technology, 2020, 21 (4): 881-887." proposed an automatic parking system based on DQN. Although manual demonstrations improved the training efficiency, the convergence time was still long."Zhang J,Chen H,Song S,et al.ReinforcementLearning-Based Motion Planning for Automatic Parking System[J].IEEE Access, 2020, 8:154485-154501." A parallel parking method combining Monte Carlo Tree Search (MCTS) and neural network is proposed. This method can ensure multi-objective optimization, including safety, comfort, parking efficiency and final parking performance. However, this method only considers lateral action, that is, the incremental steering angle within Δt time, while the longitudinal action (speed) is a predefined vt curve. This method first performs the vehicle reversing process based on MCTS and neural network, and then adjusts the vehicle state through a rule-based adjustment procedure to finally complete the parking. Although the efficiency of this model-based reinforcement learning method is much better than the model-free method mentioned above, the method divides the parallel parking process into the reversing stage and the adjustment stage within the parking space. The agent is only responsible for the reversing stage. In the adjustment stage, the vehicle moves with the maximum steering wheel angle to adjust the heading angle. The adjustment time is unknown, the final parking state is not determined, and the model training time is too long. Summary of the invention
[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an automatic parking motion planning method based on the MCTS algorithm with high efficiency and high speed.
[0008] The purpose of the present invention can be achieved by the following technical solutions:
[0009] An automatic parking motion planning method based on MCTS algorithm, comprising:
[0010] The initial state and target state of the vehicle are obtained, and the target state is obtained according to the position of the parking space. goal , the optimal trajectory data set is obtained through the MCTS algorithm, and the optimal trajectory data set expression is:
[0011] {(s i ,α i )|i=0,1...n-1}
[0012] When i is 0, s0 is the initial state, and when i is not 0, s i is the i-th state of the vehicle, α i For the state s i The current optimal action of the vehicle under α i The state of the vehicle after is given by s i becomes s i+1, n is the optimal number of actions, execute α i+1 The vehicle state is then determined by s i becomes s i+1 , execute α n-1 After vehicle status n-1 From the final state s n ;
[0013] Taking the initial state as the starting point, the vehicle is controlled to run to the optimal final state s according to the optimal action set n , the final state s n With the target state s goal The error conditions are met and automatic parking is completed.
[0014] Furthermore, a plane coordinate system is constructed on the plane where the parking space is located, and the orientation of the parking space is parallel to the X-axis of the plane coordinate system;
[0015] The status i The expression is:
[0016]
[0017] Among them, x i and i In state s i The horizontal and vertical coordinates of the midpoint of the rear axle of the lower vehicle, θ i , and v i In state s i The moving direction, steering wheel angle and moving speed of the lower vehicle;
[0018] The expression of the action α is:
[0019]
[0020] in, and Δv are the changes in the steering angle and speed of the vehicle within the planning period Δt, respectively.
[0021] Furthermore, the steering angle change correction value is solved through the sliding film control step, and when the vehicle is controlled to park, the steering angle change in the current optimal action is replaced by the steering angle correction value;
[0022] The steering angle change correction value The calculation formula is:
[0023]
[0024]
[0025]
[0026] slaw=c1x2+c2[A+B·u]
[0027]
[0028]
[0029] u=tan(ψ)
[0030] Among them, ψ′ is the correction value of the front wheel angle of the vehicle, β is the transmission ratio, ψ is the current steering wheel angle of the vehicle, and ψ r is the reference vehicle front wheel turning angle, θ is the current vehicle heading angle, and θ r is the reference vehicle heading angle, slaw is the approach rate, x1 is the error of the y-axis coordinate, x2 is the error of the tangent value of the heading angle, c1 and c2 are the set weights, and L is the vehicle wheelbase.
[0031] Furthermore, the MCTS algorithm includes:
[0032] Build a vehicle model and set m candidate actions;
[0033] Construct a policy neural network π;
[0034] Taking the initial state as the starting point, combined with the policy neural network π, iteratively executes the action optimization step, gradually selects the current optimal action from the m candidate actions, and updates the current trajectory data set;
[0035] When the iteration interruption condition is met, the policy neural network π0 is reconstructed and the action optimization step is iteratively executed;
[0036] If the final state s obtained through the current trajectory data set n,now With the target state s goal When the error condition is met, the iteration ends, and the current trajectory dataset is taken as the optimal trajectory dataset.
[0037] Furthermore, the i-th iteration process of the action optimization step includes:
[0038] For the current state s i-1 The vehicle model performs m candidate actions α′ respectively i-1 , corresponding to m candidate states s′ i ;
[0039] According to the candidate state s′ i and the target state s goal , using the policy neural network π i-1 Generate an imagined trajectory data set; the current trajectory data set, (s i-1 ,α′ i-1) and the imagined trajectory dataset constitute a candidate trajectory dataset, and a total of m candidate trajectory datasets are obtained;
[0040] Use the reward evaluation function to calculate the reward value R of each candidate trajectory dataset and obtain the maximum reward value R i,max ;
[0041] Judge R i,max Is it greater than the maximum reward value R obtained in the previous iteration? i-1,max ,
[0042] If so, it will be judged to have the maximum reward value R i,max Candidate actions α′ in the candidate trajectory dataset i-1 Denoted as α′ i-1,best , α′ i-1,best The corresponding candidate state s′ i Denoted as s′ i,best , will (s i-1 ,α′ i-1,best ) is added to the current trajectory dataset and the current state of the vehicle model is updated to s′ i,best , to have the maximum reward value R i,max The candidate trajectory dataset is updated to the current best trajectory dataset,
[0043] Otherwise, update the current state of the vehicle model to s according to the current best trajectory dataset i,best , and s i,best Add to the current trajectory dataset.
[0044] Furthermore, the generation process of the imagined trajectory dataset includes:
[0045] The candidate state s′ i Input the current policy neural network to obtain the probability distribution of different candidate actions. According to the probability distribution, select a candidate action α′ through the online rolling exploration algorithm. i ,
[0046] According to s′ i and α′ i Get s′ i+1 , repeat the above steps until the current final state s′ n With the target state s goal The error condition is met.
[0047] Furthermore, the online rolling exploration algorithm includes:
[0048] The cumulative probability values of each candidate action are calculated in turn, and m cumulative probability values are obtained in turn. The calculation formula is:
[0049]
[0050] Among them, P cum (a′ j,i ) is the cumulative probability value of the first j candidate actions;
[0051] Create a random number num∈[0, 1] and select the first candidate action corresponding to the cumulative probability value greater than num.
[0052] Furthermore, the iteration interruption condition includes:
[0053] Meet any of the following conditions:
[0054] The iteration time is longer than the set time;
[0055] Imagine that the vehicle model collides during the generation of the trajectory dataset.
[0056] Furthermore, the calculation formula of the reward evaluation function is:
[0057] R=r pose +r time +r collision +r success
[0058] Among them, R is the reward value, r pose is the evaluation value of the final posture of the vehicle, and the calculation formula is:
[0059] r pose =r y +r θ
[0060] r y =20000*(e -abs(y_final-y_target) / 0.08 )
[0061] r θ =20000*(e -abs(θ_final-θ_target) / 0.05 )
[0062] Among them, y_final and y_target are the final ordinate and target ordinate of the midpoint of the rear axle of the vehicle after parking is completed, respectively; θ_final and θ_target are the final steering angle and target steering angle of the vehicle after parking is completed, respectively;
[0063] r time is the evaluation value of the vehicle parking time, and the calculation formula is:
[0064] r time =1000-40*t final
[0065] Among them, t final Time spent parking the vehicle;
[0066] r collision is the collision risk assessment value of vehicle parking, and the calculation formula is:
[0067]
[0068] r success The parking completion evaluation value is calculated as follows:
[0069]
[0070] Furthermore, the vehicle model expression is:
[0071]
[0072] Where L is the vehicle wheelbase, ψ is the vehicle front wheel angle, ψ i β=θ i , β is the transmission ratio, and Δt is the planning period.
[0073] Compared with the prior art, the present invention has the following beneficial effects:
[0074] (1) The present invention combines imitation learning and reinforcement learning. The reinforcement learning process, that is, the demonstration data from nonlinear programming is used to obtain the initialized policy neural network. The MCTS algorithm is used to improve the policy neural network, realize automatic parallel parking, and generate a continuous optimal trajectory data set. The optimal final state of the vehicle relative to the parking space can be determined according to the optimal trajectory data set, unifying the parking entry stage and the adjustment stage within the parking space. At the same time, the MCTS algorithm only constructs a two-layer policy tree, which reduces the amount of calculation during the search. The policy neural network greatly accelerates the reinforcement learning process, with high efficiency and high speed;
[0075] (2) In order to enable the neural network to have the function of changing the direction of vehicle movement, the output of the neural network is added with the vehicle speed action, and the horizontal and vertical actions are discretized to form a unified action type. The trajectory of parallel parking motion planning in a small parking space includes both the parking entry segment and the in-parking adjustment segment. Due to the option of the longitudinal action, the in-parking adjustment planning can be completed directly during planning. There is no on-the-spot turning phenomenon caused by separate planning at the switching point between the two stages. In addition, the system can adapt to both small and large parking spaces and has a wide range of applications.
[0076] (3) In order to eliminate the influence of the model error between the vehicle model used in training and the actual vehicle on the actual parking accuracy, the present invention adds a sliding mode control step to eliminate the errors in the y direction and the heading angle during the parking process. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] Figure 1 Schematic diagram of 20 initial states in the parallel parking scenario;
[0078] Figure 2 is a flow chart of the method of the present invention;
[0079] Figure 3 It is a schematic diagram of the structure of the strategy tree;
[0080] Figure 4 It is a structural diagram of the data pool;
[0081] Figure 5 It is a schematic diagram of the structure of the initial strategy neural network;
[0082] Figure 6 is the parking trajectory obtained by the initial strategy neural network;
[0083] Figure 7 The parking trajectory is obtained by the automatic parking motion planning method proposed in this embodiment;
[0084] Figure 8 A graph showing the average reward value versus the number of iterations of the action optimization step;
[0085] Fig. 9 This is a parking simulation diagram of the Carsim model;
[0086] Fig.10 This is a schematic diagram of the structure of the surround vision monitoring system of the Roewe E50 pure electric vehicle;
[0087] Fig.11 This is the parking trajectory obtained in Experiment 1;
[0088] Fig.12 is the parking trajectory obtained when the initial orientation of the initial posture is –4° in Experiment 2;
[0089] Fig.13 This is the parking trajectory obtained when the initial orientation of the initial posture in Experiment 2 is 4°. DETAILED DESCRIPTION
[0090] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0091] An automatic parking motion planning method based on MCTS algorithm, such as Figure 2 As shown, including:
[0092] 1) Obtain the initial state and target state of the vehicle. The target state is obtained based on the location of the parking space;
[0093] 2) According to the initial state and target state s goal, obtain the optimal trajectory data set through the MCTS algorithm;
[0094] The optimal trajectory dataset expression is:
[0095] {(s i ,α i )|i=0,1...n-1} (1)
[0096] When i is 0, s0 is the initial state, and when i is not 0, s i is the i-th state of the vehicle, α i For the state s i The current optimal action of the vehicle under α i The state of the vehicle after is given by s i becomes s i+1 , n is the optimal number of actions, execute α i+1 The vehicle state is then determined by s i becomes s i+1 , execute α n-1 After vehicle status n-1 From the final state s n ;
[0097] 3) Taking the initial state as the starting point, control the vehicle to run to the optimal final state s according to the optimal action set n , final state s n With the target state s goal The error conditions are met and automatic parking is completed.
[0098] like Figure 1 ,This embodiment uses parallel parking as a research scenario, constructs a rectangular coordinate system on the plane where the parking space is located, and the parking space orientation is parallel to the X-axis of the plane coordinate system;
[0099] Status i The expression is:
[0100]
[0101] Among them, x i and i In state s i The horizontal and vertical coordinates of the midpoint of the rear axle of the lower vehicle, θ i , and v i In state s i The vehicle's direction of movement, steering wheel angle and speed, p i ={x i ,y i ,θ i} represents the vehicle pose.
[0102] Final state n With the target state s goal The error conditions are met, where the error conditions include:
[0103] Final state n With the target state s goal The horizontal and vertical coordinates of the midpoint of the vehicle's rear axle and the vehicle's movement direction (i.e., vehicle posture) are within the set error range.
[0104] The expression of action α is:
[0105]
[0106] in, and Δv are the steering angle change and speed change of the vehicle within the planning period Δt, respectively.
[0107] The MCTS algorithm includes:
[0108] Construct a vehicle model. The vehicle model expression is:
[0109]
[0110] Where L is the vehicle wheelbase, ψ is the vehicle front wheel angle, β is the transmission ratio, Δt is the planning period;
[0111] Set m candidate actions;
[0112] Considering that the maximum steering angle rate of the actual vehicle is 400° / s and the planning period Δt is 50ms, we set The maximum acceleration is set to 0.3m / s 2 , set Δv∈[-0.015m / s,0,0.015m / s], so and Δv can be combined into 15 candidate actions, i.e., m = 15;
[0113] The automatic parking motion planning method based on nonlinear programming (NLP) proposed in "Li B, Zhang Y, Shao Z. Spatio-temporal decomposition: a knowledge-based initialization strategy for parallel parking motion optimization [J]. Knowledge-Based Systems, 2016, 107: 179-196." is used to directly generate several groups of demonstration data to form a training set. Each group of demonstration data is represented in the form of {s, α} according to the timestamp through linear interpolation. The artificial neural network (ANN) is trained with the training set to obtain the initial policy neural network (PNN), which is recorded as π0. The input of the policy neural network is s, and the output is the probability distribution of 15 candidate actions. The demonstration data is processed by linear interpolation according to the timestamp, so the demonstration data is not enough to cover the state space during parking. The initial policy neural network is not enough to perform well in the given parallel parking task.
[0114] Taking the initial state as the starting point, combined with the initial policy neural network π0, iteratively executes the action optimization step, gradually selects the current optimal action from the m candidate actions, and updates the current trajectory data set;
[0115] When the iteration interruption condition is met, the initial policy neural network π0 is reconstructed and the action optimization step is iteratively executed;
[0116] If the final state s obtained through the current trajectory data set n,now With the target state s goal When the error condition is met, the iteration ends, and the current trajectory dataset is taken as the optimal trajectory dataset.
[0117] The i-th iteration process of the action optimization step includes:
[0118] For the current state s i-1 The vehicle model performs m candidate actions α′ respectively i-1 , corresponding to m candidate states s′ i ;
[0119] According to the candidate state s′ i and the target state s goal , using the policy neural network π i-1 Generate an imagined trajectory dataset; current trajectory dataset, (s i-1 ,α′ i-1 ) and the imagined trajectory dataset constitute a candidate trajectory dataset, and a total of m candidate trajectory datasets are obtained;
[0120] Use the reward evaluation function to calculate the reward value R of each candidate trajectory dataset and obtain the maximum reward value R i,max ;
[0121] Judge R i,max Is it greater than the maximum reward value R obtained in the previous iteration? i-1,max ,
[0122] If so, it will be judged to have the maximum reward value R i,max Candidate actions α′ in the candidate trajectory dataset i-1 Denoted as α′ i-1,best , α′ i-1,best The corresponding candidate state s′ i Denoted as s′ i,best , will (s i-1 ,α′ i-1,best ) is added to the current trajectory dataset and the current state of the vehicle model is updated to s′ i,best , to have the maximum reward value R i,max The candidate trajectory dataset is updated to the current best trajectory dataset,
[0123] Otherwise, update the current state of the vehicle model to s according to the current best trajectory dataset i,best , and s i,best Add to the current trajectory dataset.
[0124] When s′ i,best and the target state s goal When the error condition is met, the current trajectory data set is stored in the data pool, and the current strategy neural network π is trained using the current trajectory data set i-1 , get the policy neural network π i ,like Figure 4 , when the initial poses of 20 vehicles are used for automatic parking motion planning in turn, the policy neural network is iterated and trained 20 times in total.
[0125] s′ i,best and the target state s goal The error conditions are met, where the error conditions include:
[0126] s′ i,best and the target state s goal The errors of the horizontal and vertical coordinates of the midpoint of the vehicle's rear axle, the vehicle's movement direction, steering wheel angle and movement speed (i.e., vehicle posture) are within the set error range.
[0127] The generation process of the imagined trajectory dataset includes:
[0128] The candidate state s′ iInput the current policy neural network to obtain the probability distribution of different candidate actions. According to the probability distribution, select a candidate action α′ through the online rollout algorithm i ,
[0129] According to s′ i and α′ i Get s′ i+1 , repeat the above steps until the current final state s′ n With the target state s goal The error condition is met.
[0130] The online rollout algorithm includes:
[0131] The cumulative probability values of each candidate action are calculated in turn, and m cumulative probability values are obtained in turn. The calculation formula is:
[0132]
[0133] Among them, P cum (a′ j,i ) is the cumulative probability value of the first j candidate actions;
[0134] Create a random number num∈[0,1] and select the first candidate action corresponding to the cumulative probability value greater than num.
[0135] The online rollout algorithm uses a roulette wheel rule to select candidate actions, which helps to explore actions with relatively low probability while continuing to exploit actions with high probability.
[0136] Iteration interruption conditions include:
[0137] Meet any of the following conditions:
[0138] The iteration time is longer than the set time;
[0139] Imagine that the vehicle model collides during the generation of the trajectory dataset.
[0140] The calculation formula of the reward evaluation function is:
[0141] R=r pose +r time +r collision +r success (6)
[0142] Among them, R is the reward value, r pose is the evaluation value of the final posture of the vehicle, and the calculation formula is:
[0143] r pose =ry +r θ (7)
[0144] r y =20000*(e -abs(y_final-y_target) / 0.08) (8)
[0145] r θ =20000*(e -abs(θ_final-θ_target) / 0.05 ) (9)
[0146] Among them, y_final and y_target are the final ordinate and target ordinate of the midpoint of the rear axle of the vehicle after parking is completed, respectively; θ_final and θ_target are the final steering angle and target steering angle of the vehicle after parking is completed, respectively;
[0147] r time is the evaluation value of the vehicle parking time, and the calculation formula is:
[0148] r time =1000-40*t final (10)
[0149] Among them, t final Time spent parking the vehicle;
[0150] r collision is the collision risk assessment value of vehicle parking, and the calculation formula is:
[0151]
[0152] r success The parking completion evaluation value is calculated as follows:
[0153]
[0154] like Figure 3 , MCTS algorithm is a series of algorithms that use Monte Carlo samples to find the best action, based on the current state s t As the root node, a two-layer strategy tree is constructed. Each node in the strategy tree represents a state of the vehicle, and each path connecting the nodes represents an action. The nodes in the second layer represent the transfer of s from the current state by executing the corresponding action. i To the next state s i+1 , transfer function s i =f(s i ,α i ) is the kinematic model of the vehicle, that is, formula (4). The nodes in the second layer of the strategy tree are all one step away from the root node, so it is called one-step MCTS. The MCTS algorithm program proposed in this embodiment is as follows:
[0155]
[0156] Each time an action is explored, a false trajectory data set and a reward are obtained, so the vehicle state s at timestamp t t , 15 false trajectory datasets and 15 rewards are obtained, among which the one with the largest reward r is selected. t-best The trajectory is taken as the best trajectory dataset (T t-best ), and compare the best trajectory with the one before timestamp t with the maximum reward r best The best trajectory dataset (T best ) for comparison, if T t-best T best OK, then update T best , that is, let T best =T t-best , if T t-best T best The state of the vehicle at the next time is s t+1 Follow T best .
[0157] During the training of the policy neural network, the vehicle model is an ideal kinematic model. The ideal kinematic model assumes that the actuator has no delay and the state transition is deterministic. However, the real vehicle is equipped with actuators with random delays. The posture of the output trajectory is connected by a series of open-loop nominal commands, which implicitly consider the constraints of maximum steering angle, maximum steering speed and maximum speed, and the trajectory considers the obstacle avoidance requirements and is a kinematically feasible path with relevant nominal open-loop commands. The above effects are eliminated by the non-time reference sliding mode controller, specifically:
[0158] The steering angle change correction value is solved by the sliding film control step, and the steering angle change value in the current optimal action is replaced by the steering angle correction value when the vehicle is controlled to park;
[0159] During the reversing process, the y coordinate error y of the desired trajectory e And the steering angle error θ e At a certain x(t), the variable -x(t) is monotonically increasing and is selected as the reference variable. The x-coordinate error x e ,y e and θ e The expression is:
[0160]
[0161] Among them, y r (x) is the reference y-axis coordinate, y(x) is the actual y-axis coordinate, θ r(x) is the reference heading angle, θ(x) is the actual heading angle;
[0162] By taking the derivative with respect to -x, we get:
[0163]
[0164] Among them, ψ r is the reference front wheel turning angle, L is the wheelbase of the vehicle;
[0165] Then, set the state variables x1 and x2, and the expressions of x1 and x2 are:
[0166]
[0167] Among them, y r is the reference y-axis coordinate;
[0168] According to formulas (14) and (15), we can obtain:
[0169]
[0170]
[0171] Among them, the expressions of A(x) and B(x) are:
[0172]
[0173] The sliding surface is set as:
[0174]
[0175] Among them, slaw is the approach rate. In order to reduce the output vibration, the expression of the approach rate is:
[0176]
[0177] Among them, delta, ε, and k are constants greater than 0.
[0178] Steering angle change correction value The calculation formula is:
[0179]
[0180]
[0181] Among them, ψ′ is the correction value of the front wheel angle of the vehicle, β is the transmission ratio, ψ is the current steering wheel angle of the vehicle, and ψ r is the reference vehicle front wheel turning angle, θ is the current vehicle heading angle, and θ ris the reference vehicle heading angle, slaw is the approach rate, x1 is the error of the y-axis coordinate, x2 is the error of the tangent value of the heading angle, c1 and c2 are the set weights, and L is the vehicle wheelbase.
[0182] When the vehicle moves forward, the reference variable is set to x(t), and equation (14) is transformed into the following equation:
[0183]
[0184] like Figure 5 As shown in Figure 2, a fully connected neural network with 2 hidden layers is trained based on the MATLAB machine learning toolbox as the initial policy neural network π0. Each hidden layer includes 15 neurons. Based on the initial policy neural network π0, the parallel parking process is simulated in MATLAB through one-step MCTS. The results are shown in Figure 2. Figure 6 As shown, it shows that the initial policy neural network π0 from imitation learning is not enough to successfully park the car under 20 initial poses.
[0185] Reinforcement learning is then performed to improve the capabilities of the policy neural network, such as Figure 8 After 10 iterations, the average reward of 20 episodes converges. The parking results based on the improved policy neural network and one-step MCTS are as follows: Figure 7 As shown, the effectiveness of combining imitation learning and reinforcement learning is demonstrated.
[0186] On the one hand, imitation learning accelerates the reinforcement learning process, and on the other hand, reinforcement learning enhances imitation learning. For the application of parallel parking, this embodiment can obtain a near-optimal PNN through 10 iterations in a short time (about one hour). In the previous parallel parking method combining Monte Carlo Tree Search (MCTS) and neural network, there was no imitation learning, and the training time was about 20 hours and 800 iterations. The automatic parking motion planning method proposed in this embodiment greatly reduces the training time.
[0187] Before deploying the automatic parking motion planning method proposed in this embodiment to the real vehicle platform, a simulation experiment is performed using the vehicle simulator CarSim to verify the proposed method on a high-fidelity vehicle model. The configuration of the CarSim model is the same as that of the real vehicle. The outline of the vehicle is represented by an octagon to make full use of the space. The result path obtained by the simulation experiment is as follows: Fig. 9 As shown in the figure, the final posture of the vehicle after completing parking is (-3.8198m, -0.9067m, -0.0001°). It can be seen that although the trajectory planner is based on the kinematic model, when this method is deployed on the CarSim model, it can complete automatic parallel parking in a small parking space without collision.
[0188] The automatic parking motion planning method proposed in this embodiment is deployed on a real vehicle platform to conduct real vehicle experiments. Fig.10 As shown in the figure, the actual vehicle platform is the Roewe E50 pure electric vehicle produced by SAIC Group. The vehicle is equipped with an AVM (surround vision monitoring) system, 3 lidars on the top, 8 body radars, a single front-view camera, an RTK-GPS system and a dSpace MicroAutobox-II (MAB-II) with a 900MHz processor. The entire algorithm is executed in the MAB. The vehicle's actuators include an electronic power system (EPS) and two longitudinal drive motors (Motors) on the front wheels. In the experiment, it is assumed that the parking space has been detected by AVM or lidar. The vehicle position is obtained by RTK-GPS during parking, and the maximum error is within 2cm.
[0189] Table 1: Configuration of the actual vehicle platform
[0190] length 3.57m Wheelbase 2.305m width 1.775m Maximum turning speed 400° / s Maximum steering angle 530° Maximum speed 1m / s
[0191] The real vehicle experiment is divided into Experiment 1 and Experiment 2;
[0192] Experiment 1 includes: Figure 1 First, set the initial poses of 20 parking vehicles. The initial direction of the vehicles is 0°. The length of the real vehicle platform is 3.57m, the length of the parking space is 4.57m, the target steering angle is set to 0°, and the target parking depth of the parking space is set to 0.9m. According to ISO-16787, the initial clear side distance of the vehicle is 0.9-1.5m. In the experiment, the half width of the real vehicle platform is (1.551 / 2=0.775m), the range of x is 2:0.5:3.5m, and the range of y is 1.25:0.25:2.25m. Verify the adaptability of the automatic parking motion planning method proposed in this embodiment to the actual vehicle dynamics. The experimental test results are shown in Table 2:
[0193] Table 2 Test results of 20 parking experiments
[0194]
[0195] Among them, Δy and Δθ are the y coordinate error and steering angle error of the vehicle after parking, respectively. Time is the parking time of the vehicle. After parking, the maximum value of Δy is within 5 cm, the maximum value of Δθ is 0.23, and the longest parking time is 29.5 seconds. The parallel parking paths of 20 initial postures are as follows: Fig.11 As shown, it can be seen that the parallel parking motion planning method proposed in this embodiment can complete automatic parking from 20 initial postures with high accuracy and efficiency.
[0196] Experiment 2 includes:
[0197] The initial orientations of the initial parking postures of the 20 vehicles were changed from 0° to –4° and 4° to test the generalization ability to unseen initial parking postures. The rest was the same as in Experiment 1. The experimental results are shown in Fig.12 and 13 As shown, the final parking posture has a small error with the target y=-0.9m and the target θ=0°.
[0198] This embodiment proposes an automatic parking motion planning method based on the MCTS algorithm, which combines imitation learning and reinforcement learning to realize automatic parallel parking and unify the parking entry stage and the adjustment stage within the parking space. Imitation learning greatly accelerates the reinforcement learning process. Reinforcement learning obtains the initial strategy from the imitation learning and strengthens the strategy. The trained policy neural network (PNN) and MCTS are packaged into a planner to perform motion planning for parallel parking. Since the PNN can output the probability of the steering angle and speed increment value in each state, the planner can uniformly plan the entire parking trajectory, including the adjustment stage within the parking space. In order to be deployed on a real vehicle, a simple sliding mode controller (SMC) based on a non-time reference is used to track the trajectory.
[0199] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. An automatic parking motion planning method based on MCTS algorithm, characterized in that: include: The initial state and target state of the vehicle are obtained, wherein the target state is obtained according to the position of the parking space. goal , the optimal trajectory data set is obtained through the MCTS algorithm, and the optimal trajectory data set expression is: {(s i ,a i )|i=0,1…n-1} When i is 0, s0 is the initial state, and when i is not 0, s i is the i-th state of the vehicle, α i For the state s i The current optimal action of the vehicle under α i The state of the vehicle after is given by s i becomes s i+1 , n is the optimal number of actions, execute α i+1 The vehicle state is then determined by s i becomes s i+1 , execute α n-1 After vehicle status n-1 From the final state s n ; Taking the initial state as the starting point, the vehicle is controlled to run to the optimal final state s according to the optimal action set n , the final state s n With the target state s goal The error condition is met and automatic parking is completed; Construct a plane coordinate system on the plane where the parking space is located, and the direction of the parking space is parallel to the X-axis of the plane coordinate system; The status i The expression is: Among them, x i and i In state s i The horizontal and vertical coordinates of the midpoint of the rear axle of the lower vehicle, θ i , and v i In state s i The moving direction, steering wheel angle and moving speed of the lower vehicle; The expression of the action α is: in, and Δv are the steering angle change and speed change of the vehicle within the planning period Δt, respectively; The steering angle change correction value is solved by the sliding film control step, and the steering angle change value in the current optimal action is replaced by the steering angle correction value when the vehicle is controlled to park; The steering angle change correction value The calculation formula is: slaw=c1x2+c2[A(x)+B(x)u] u=tan(ψ) Among them, ψ′ is the correction value of the front wheel angle of the vehicle, β is the transmission ratio, ψ is the current steering wheel angle of the vehicle, and ψ r is the reference vehicle front wheel turning angle, θ is the current vehicle heading angle, and θ r is the reference vehicle heading angle, slaw is the approach rate, x1 is the error of the y-axis coordinate, x2 is the error of the tangent value of the heading angle, c1 and c2 are the set weights, and L is the vehicle wheelbase; The MCTS algorithm includes: Build a vehicle model and set m candidate actions; Construct a policy neural network π; Taking the initial state as the starting point, combined with the policy neural network π, iteratively executes the action optimization step, gradually selects the current optimal action from the m candidate actions, and updates the current trajectory data set; When the iteration interruption condition is met, the policy neural network π is rebuilt and the action optimization step is iteratively executed; If the final state s obtained through the current trajectory data set n,now With the target state s goal When the error condition is met, the iteration ends, and the current trajectory data set is used as the optimal trajectory data set; The i-th iteration process of the action optimization step includes: For the current state s i-1 The vehicle model performs m candidate actions α′ respectively i-1 , corresponding to m candidate states s′ i ; According to the candidate status s i ′ and the target state s goal , using the policy neural network π i-1 Generate an imagined trajectory data set; the current trajectory data set, (s i-1 ,α′ i-1 ) and the imagined trajectory dataset constitute a candidate trajectory dataset, and a total of m candidate trajectory datasets are obtained; Use the reward evaluation function to calculate the reward value R of each candidate trajectory dataset and obtain the maximum reward value R i,max ; Judge R i,max Is it greater than the maximum reward value R obtained in the previous iteration? i-1,max , If so, it will be judged to have the maximum reward value R i,max Candidate actions α′ in the candidate trajectory dataset i-1 Denoted as α′ i-1,best , will (s i-1 ,α′ i-1,best ) is added to the current trajectory dataset, and the current state of the vehicle model is updated to s according to the current trajectory dataset i ' ,beat , to have the maximum reward value R i,max The candidate trajectory dataset is updated to the current best trajectory dataset, Otherwise, update the current state of the vehicle model to s according to the current best trajectory dataset i ' ,best , and s i ' ,best Add to the current trajectory dataset.
2. The automatic parking motion planning method based on the MCTS algorithm according to claim 1 is characterized in that: The generation process of the imagined trajectory dataset includes: The candidate state s i ′ Input the current policy neural network to obtain the probability distribution of different candidate actions. According to the probability distribution, select a candidate action α′ through the online rolling exploration algorithm i , According to s′ i and α′ i Get s′ i+1 , repeat the above steps until the current final state s′ n With the target state s goal The error condition is met.
3. The automatic parking motion planning method based on the MCTS algorithm according to claim 2 is characterized in that: The online rolling exploration algorithm includes: Calculate the cumulative value of the probability distribution of different candidate actions. The calculation formula is: Among them, P cum (a′ j,i ) is the cumulative probability value of the first j candidate actions, p(α′ i ) is the candidate action α′ i The probability distribution of Create a random number num∈[0, 1] and select the first candidate action corresponding to the cumulative probability value greater than num.
4. The automatic parking motion planning method based on the MCTS algorithm according to claim 2 is characterized in that: The iteration interruption conditions include: Meet any of the following conditions: The iteration time is longer than the set time; Imagine that the vehicle model collides during the generation of the trajectory dataset.
5. The automatic parking motion planning method based on the MCTS algorithm according to claim 1 is characterized in that: The calculation formula of the reward evaluation function is: R=r pose +r time +r collision +r success Among them, R is the reward value, r pose is the evaluation value of the final posture of the vehicle, and the calculation formula is: r pose =r y +r θ r y =20000*(e -abs(y_final-y_target) / 0.08 ) r θ =20000*e -abs(θ_final-θ_target) / 0.05 ) Among them, y_final and y_target are the final ordinate and target ordinate of the midpoint of the rear axle of the vehicle after parking is completed, respectively; θ_final and θ_target are the final steering angle and target steering angle of the vehicle after parking is completed, respectively; r time is the evaluation value of the vehicle parking time, and the calculation formula is: r time =1000-40*t final Among them, t final Time spent parking the vehicle; r collision is the collision risk assessment value of vehicle parking, and the calculation formula is: r success The parking completion evaluation value is calculated as follows:
6. The automatic parking motion planning method based on MCTS algorithm according to claim 1 is characterized in that: The vehicle model expression is: Where L is the vehicle wheelbase, ψ is the vehicle front wheel angle, ψ i β=θ i , β is the transmission ratio, and Δt is the planning period.
Citation Information
Patent Citations
Automatic parking method and system based on optimal path
CN110696818A
Intelligent parking method based on model reinforcement learning
CN112356830A
Automatic parking method and apparatus, electronic device, and storage medium
US20210221361A1
