A cross-scenario behavior decision generalization method for intelligent vehicles
By constructing and generalizing the basic unit library of behavior decisions for intelligent vehicles in simulation scenarios, and using reinforcement learning algorithms, the adaptability of behavior decisions for intelligent vehicles in multiple application scenarios is solved, and efficient and accurate behavior decisions are achieved. Strong scenario migration ability.
Patent Information
- Application Number
- CN202410227870.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-29
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2044-02-29
AI Technical Summary
The existing learning-based intelligent vehicle behavior decision-making method is difficult to adapt to multiple application scenarios when the scene changes, and has high requirements for data quality and quantity, large calculation volume and poor interpretability.
By simulating the driving process in a simulation scenario, data is collected to build a basic unit library for behavior decision making and generalizing it to form a generalized basic unit library for behavior decision making. Then, using reinforcement learning algorithms, especially SMDP+Q-Learning algorithms, learn the best basic unit combination of behavior decisions based on the target scenario and tasks.
It realizes generalization of behavioral decisions of smart vehicles in various driving scenarios, avoids the tedious work of repeated construction of scenarios and data collection, improves the accuracy and reliability of the decision database, and has strong scenario migration and adaptability.
Smart Images
Figure CN118859922B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned driving technology, and in particular to a cross-scenario behavior decision generalization method for intelligent vehicles. Background Art
[0002] Due to the complex environment, strong uncertainty, and multiple dynamic targets in the real urban environment, autonomous behavior decision-making of intelligent vehicles has always been the focus and difficulty of research in the field of intelligent vehicles and robots. At present, intelligent vehicle decision-making methods mainly include rule-based methods and learning-based methods.
[0003] The rule-based method makes behavioral decisions for intelligent vehicles based on pre-set rules and logic. It has the advantages of strong interpretability, simple implementation, and fast response. However, due to the high complexity of real scenarios, it is difficult for artificially set rules and logic to cover all situations. The rule-based method has the disadvantages of insufficient intelligence, flexibility, and adaptability. The learning-based method uses artificial intelligence and machine learning technology to learn autonomous behavior decision-making models from historical data or virtual scenarios, and estimates the optimal behavior in the current optimal scenario based on the current state. Since this method can discover and utilize complex nonlinear relationships between features, its advantage is that it can better handle complex and uncertain situations and simulate human-like strategies. However, the learning-based method has high requirements on data quality and quantity in actual engineering applications, and is sensitive to changes in test data. At the same time, the decision-making model is often complex, with a large amount of computation and poor interpretability.
[0004] How to improve the algorithm's autonomous behavior decision-making capabilities and its migration capabilities when the scene changes are urgent issues to be solved in the current learning-based intelligent vehicle behavior decision-making methods. Summary of the invention
[0005] In view of the above analysis, an embodiment of the present invention aims to provide a cross-scenario behavior decision generalization method for intelligent vehicles, so as to solve the problem that the existing learning-based behavior decision algorithm has a single application scenario and is difficult to adapt to multiple application scenarios.
[0006] An embodiment of the present invention provides a cross-scenario behavior decision generalization method for an intelligent vehicle, comprising the following steps:
[0007] Simulate the driving process in the simulation scene and collect vehicle data during the driving process;
[0008] Based on the data, a behavior decision basic unit set for each driving behavior is obtained to form a behavior decision basic unit library; any of the behavior decision basic unit sets includes a plurality of behavior decision basic units corresponding to the driving behavior, and each behavior decision basic unit includes a control sequence P, a trajectory r and a speed v;
[0009] Generalize each behavior decision basic unit in each behavior decision basic unit set to obtain a generalized behavior decision basic unit library;
[0010] For any target task in any target scenario, based on the generalized behavior decision basic unit library, a reinforcement learning algorithm is used to learn the best combination of behavior decision basic units to complete the target task.
[0011] Specifically, the control sequence P is represented by {Th(t), Br(t), Str(t)}; wherein Th(t) is the sequential sequence of the accelerator pedal control amount, Br(t) is the sequential sequence of the brake pedal control amount, and Str(t) is the sequential sequence of the steering wheel angle control amount;
[0012] The generalization of each behavior decision basic unit in each behavior decision basic unit set includes:
[0013] The sequential sequence Th(t) of the accelerator pedal control amount and the sequential sequence Str(t) of the steering wheel angle control amount corresponding to the behavior decision basic unit are multiplied by the coefficients γ and η respectively to obtain the control sequence P′:
[0014] The above control sequence P′ is input into the simulation platform, and the trajectory r′ of the vehicle under this control sequence is calculated;
[0015] Set the initial speed of the vehicle, input it and the above control sequence P′ into the simulation platform, and calculate the speed v′ of the vehicle under this control sequence;
[0016] The new behavior decision basic unit includes: control sequence P′, trajectory r′ and speed v′.
[0017] Specifically, multiple behavior decision basic units in the behavior decision basic unit set of any driving behavior D are obtained in the following way:
[0018] According to the driving behavior, the vehicle data collected during the driving process is split to obtain data subsets corresponding to multiple driving behaviors;
[0019] Select D corresponding to driving behavior D from all data subsets n a subset of data;
[0020] Extract D n The trajectory r, velocity v and control sequence P in each of the data subsets are obtained, that is, D n behavioral decision basic units; each of which can complete driving behavior D.
[0021] Furthermore, the step of obtaining a plurality of behavior decision basic units in a behavior decision basic unit set for any driving behavior D further includes:
[0022] Right D n The trajectory r, velocity v and control sequence P in the data subset are smoothed using the local weighted regression scatter point smoothing method.
[0023] Specifically, the reinforcement learning algorithm is an SMDP+Q-Learning algorithm, wherein SMDP is used for modeling, the Q-Learning algorithm is used to obtain a Q matrix, and the optimal behavior decision basic unit combination is read from the Q matrix.
[0024] Specifically, the modeling process includes:
[0025] Step S401: Obtain the vehicle's environmental state S at each moment t , t=0,1,2,…,T;
[0026] Step S402: Based on the generalized behavior decision basic unit library, obtain the vehicle action set O, including N o The basic unit of behavioral decision-making;
[0027] Step S403, setting the reward function R generated by the action, which is related to the state of the vehicle after the action occurs, that is, the state of the vehicle at the next moment;
[0028] Step S404: Set the state-action value function Q(S,O) to be T×N o A matrix where any element Q tj Indicates that the vehicle is in state S t Next select Action O j The expected value of the cumulative reward that can be obtained after the operation; and initialize all elements in Q(S,O) to 0.
[0029] Specifically, the operation of obtaining the Q matrix includes:
[0030] Step S405: Set the number of cycles to C, and record the Q matrix obtained in the ith cycle as Q i ; For the first loop, set index i = 1; The loop process includes:
[0031] Step SS0, set time index t=0;
[0032] Step SS1: For the current vehicle environment state S t , select a behavior decision O from the action space O j Take action and observe the selected action O j The reward R generated tj and the vehicle's next environmental state St+1 ;
[0033] Step SS2: Use the simplified Bellman equation to calculate Q i Elements in The value of is updated, and the updated element for:
[0034] Q i (S t ,O j )←R tj +γ′max o′ Q i (S t+1 ,O′);
[0035] Where γ′ is the discount factor in the Bellman equation; max o′ Q i (S t+1 , O′) indicates that the vehicle reaches the environmental state S t+1 When all possible actions are i The maximum value max o′ Q i (S t+1 ,O′);
[0036] Step SS3: Determine the current vehicle environment state S t Is it the target state S? T If not, set t=t+1, repeat steps SS1-SS3 until the vehicle's environmental state reaches the target state, and then execute step SS4; if yes, directly execute step SS4;
[0037] Step SS4: Observed Q i Check whether the matrix converges. If so, use it as the final Q matrix. If not, set i=i+1 and repeat steps SS0-SS4 until the final Q matrix is obtained.
[0038] Furthermore, the operation of reading the optimal behavior decision basic unit combination from the Q matrix is as follows:
[0039] The elements of the final Q matrix are read row by row, and the actions corresponding to the columns with the maximum values of the elements in each row are combined to form the optimal basic unit combination of behavioral decisions to complete the target task in the target scenario.
[0040] Furthermore, the action decision basic unit O is selected from the action space O. j Take action and execute it using the ε-greedy algorithm, where the probability of each action being selected is set to
[0041] Specifically, the environmental state S of the vehicle at any moment t Including the real-time position and speed of the main vehicle, as well as the real-time position and speed of the surrounding interactive vehicles:
[0042] S t ={x host ,y host ,θ host ;v host ;x other ,y other ,θ other ;v other} t
[0043] Among them, x host ,y host ,θ host Indicates the position information of the main vehicle, x other ,y other ,θ other represents the position information of the interactive vehicles around the main vehicle, v host and v other Represent the velocity matrices of the host vehicle and surrounding vehicles respectively;
[0044] Each behavior decision basic unit O in the action space O j They all belong to the generalized basic unit library of behavioral decision-making;
[0045] The reward function R is set to:
[0046] R=R goal +R safety +R efficiency +R vel
[0047] R goal =100
[0048] R xafety =-200
[0049] R eggiciency =-0.5t
[0050]
[0051] Among them, R goal Represents the reward obtained when the vehicle reaches the target point; R safety It is an indicator used to measure the safety of vehicles when performing tasks; R efficiency Used to evaluate the time taken by the vehicle to complete the driving task; R vel Used to measure changes in vehicle speed.
[0052] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0053] 1. The present invention generalizes the constructed behavior decision library to obtain richer behavior decisions that are suitable for various driving scenarios. When the vehicle scenario is different from the constructed simulation scenario, a behavior decision combination suitable for the current scenario can also be selected from the generalized behavior decision library; compared with the learning-based behavior decision method suitable for a single scenario, the method of the present invention can avoid the tedious work of repeatedly building scenarios, collecting data, training models, etc.
[0054] 2. When acquiring each behavior decision, the original collected data is smoothed to reduce errors such as trajectory drift caused by the delay between vehicle operation information and vehicle model communication, thereby improving the accuracy and reliability of the constructed behavior decision library;
[0055] 3. Based on the generalized behavior decision library, the reinforcement learning algorithm is adopted, and the reward function is set according to actual needs. It can efficiently obtain the optimal behavior decision basic unit combination to complete specific tasks in the target scenario, and has strong scene migration and adaptability.
[0056] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;
[0058] Figure 1 A flowchart of the cross-scenario behavior decision generalization method for intelligent vehicles according to the present invention;
[0059] Figure 2 The urban road simulation scene constructed according to the present invention;
[0060] Figure 3 This is a schematic diagram of the composition of the behavior decision library of the present invention;
[0061] Figure 4 (a) and (b) are the vehicle trajectory curves before and after smoothing, respectively;
[0062] Figure 5 (a), (b) and (c) are the control sequence, trajectory and speed curve diagrams in any behavior decision respectively;
[0063] Figure 6 This is a diagram showing the migration effect of the trajectory in the behavioral decision after linear generalization;
[0064] Figure 7 Flowchart for obtaining the Q matrix for the Q-Learning algorithm. DETAILED DESCRIPTION
[0065] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0066] A specific embodiment of the present invention discloses a cross-scenario behavior decision generalization method for intelligent vehicles, such as Figure 1 As shown, including:
[0067] Step S1, simulating a driving process in a simulation scene and collecting vehicle data during the driving process;
[0068] Specifically, Figure 2 As shown, a simulation scene of vehicle driving is first built, including setting the road environment and surrounding obstacles, etc. Then, a simulated driver is used in the simulation scene to simulate the driving process and collect vehicle data during the driving process, including the vehicle's trajectory, speed, and control sequence. The driving process includes a variety of driving behaviors, such as acceleration, deceleration, steering, and lane changing, etc. Each driving behavior is repeated multiple times;
[0069] Step S2, based on the data, obtaining a behavior decision basic unit set under each driving behavior to form a behavior decision basic unit library; the behavior decision basic unit set under any driving behavior includes multiple behavior decision basic units corresponding to the driving behavior, and each behavior decision basic unit includes a control sequence P, a trajectory r and a speed v;
[0070] The control sequence P includes a series of underlying control sequences {Th(t), Br(t), Str(t)} of the vehicle corresponding to the corresponding driving behavior, which are the sequential sequence Th(t) of the accelerator pedal control amount, the sequential sequence Br(t) of the brake pedal control amount, and the sequential sequence Str(t) of the steering wheel angle control amount;
[0071] Specifically, Figure 3 As shown, the behavior decision basic unit library includes a number of behavior decision basic unit sets for driving behaviors, each behavior decision basic unit set includes a plurality of behavior decision basic units corresponding to the driving behavior, and each behavior decision basic unit includes a control sequence P and a corresponding vehicle trajectory r and speed v for each occurrence of the driving behavior; Figure 5As shown, it is the content of any behavior decision basic unit in the behavior decision basic unit set of a certain driving behavior. Figure 5 Where (a) is the control sequence P = {Th(t), Br(t), Str(t)}, Figure 5 (b) is the trajectory r = (x, y) of the vehicle under this control sequence. Figure 5 (c) is the speed v of the vehicle under this control sequence;
[0072] Specifically, the behavior decision unit for obtaining each driving behavior can manually split the trajectory r, speed v and control sequence P of the vehicle during the driving process collected in step S1 according to the driving behavior, so that each section of the control sequence P after processing can control the vehicle to complete the corresponding behavior within the trajectory length range r and at its proper speed v;
[0073] Furthermore, considering that there is often a delay in the communication between the vehicle control information and the vehicle model in the asynchronous mode of the simulation platform, the vehicle trajectory fed back by the sensor often has trajectory drift, such as Figure 4 As shown in (a), therefore, before obtaining the content of each behavioral decision, the collected raw data needs to be smoothed first; exemplarily, the local weighted regression scatter point smoothing method can be used to achieve this, and the specific operations include: 1) selecting a certain proportion of local data from the raw data as the first subset, and using a polynomial fitting algorithm to fit the regression curve of the subset to obtain the law and trend of this local data; 2) selecting a certain proportion of local data from the remaining raw data as the second subset again, and repeating the polynomial fitting operation in step 1) until a continuous and smooth curve is obtained; exemplarily, Figure 4 (b) is for Figure 4 (a) The result after smoothing the original trajectory of the vehicle;
[0074] Step S3, generalizing each behavior decision basic unit in each behavior decision basic unit set to obtain a generalized behavior decision basic unit library;
[0075] Considering that the target scenarios for actually generating behavioral decisions are different from the constructed simulation scenarios, it is necessary to generalize each behavioral decision basic unit to obtain a richer behavioral decision basic unit library to adapt to the rich and diverse target scenarios; illustratively, the present invention generalizes the behavioral decision basic unit using a linear generalization method, specifically, by linearly transforming the control sequence of the behavioral decision basic unit to enrich the behavioral decision basic unit set, and for any behavioral decision basic unit in any behavioral decision basic unit set, the specific operation of generalizing it is:
[0076] 1) Multiply the sequential sequence Th(t) of the accelerator pedal control amount and the sequential sequence Str(t) of the steering wheel angle control amount in the control sequence P of the behavior decision basic unit by the coefficients γ and η respectively, and the obtained control sequence is recorded as P′:
[0077] P′={γ×Th(t),Br(t),η×Str(t)}
[0078] Among them, the throttle control coefficient γ can be set to the range of [0,10], and the steering control coefficient η is set to ±1;
[0079] 2) Input the control sequence P′ into the simulation platform, and calculate the trajectory r′ of the vehicle under the control sequence; illustratively, Simulink simulation software is selected as the simulation platform to establish a simulation model of the vehicle;
[0080] 3) Input the control sequence P′ and the initial speed of the vehicle into the simulation platform, and calculate the speed v′ of the vehicle under the control sequence; for example, the initial speed can be set to a fixed value, such as 5 m / s;
[0081] Through the above operation, a new behavior decision can be obtained, and the content of the behavior decision includes: the new control sequence P′, the trajectory r′ and the speed v′ of the vehicle under the control sequence; the above generalization operation is performed on each behavior decision of all behavior decision units in the behavior decision library, and a generalized behavior decision library can be obtained; Figure 6 It is the migration effect diagram of the behavior decision unit after linear generalization. It can be seen from the figure that the vehicle trajectory r′ obtained after linear generalization of the control sequence is equivalent to adjusting the length, curvature or mirroring the original vehicle trajectory;
[0082] Step S4: for any target task in any target scenario, based on the generalized behavior decision library, a reinforcement learning algorithm is used to learn the best behavior decision basic unit combination to complete the target task;
[0083] Specifically, for any driving task in any target scene, the present invention adopts SMDP+Q-Learning reinforcement learning algorithm to obtain the best behavior decision basic unit combination, wherein SMDP is used for modeling, and Q-Learning algorithm is used to obtain Q matrix, and the best behavior decision basic unit combination is read from the Q matrix;
[0084] Specifically, the modeling process includes:
[0085] Step S401: Obtain the vehicle's environmental state S at each moment t , t=0,1,2,…,T;
[0086] The environmental state S at any momentt Including the real-time position and speed of the main vehicle, as well as the real-time position and speed of the surrounding interactive vehicles:
[0087] S t ={x host ,y host ,θ host ;v host ;x other ,y other ,θ other ;v other} t
[0088] Among them, x host ,y host ,θ host Indicates the position information of the main vehicle, x other ,y other ,θ other represents the position information of the interactive vehicles around the main vehicle, v host and v other Represent the velocity matrices of the host vehicle and surrounding vehicles respectively;
[0089] Step S402: Based on the generalized behavior decision library, obtain the vehicle action set O, including N o Behavioral decision making;
[0090] Specifically, the behavior decision library of the present invention includes behavior decision units for vehicle acceleration, deceleration, steering, lane change, etc., so any behavior decision O in the action space O j for:
[0091] O j ∈{O M}∪{O S}∪{O A}
[0092] Among them, O M ,O S and O A They represent the basic units of behavioral decision-making under the generalized acceleration, deceleration, steering and lane-changing behaviors respectively;
[0093] Step S403, setting a reward function R generated by the action, which is related to the state of the vehicle after the action occurs, that is, the state of the vehicle at the next moment;
[0094] The present invention comprehensively considers the safety and traffic efficiency of intelligent vehicle behavior decision-making and sets the reward function as follows:
[0095] R=R goal +R safety +R efficiency +R vel
[0096] R goal =100
[0097] R safety =-200
[0098] R efficiency =-0.5t
[0099]
[0100] Among them, R goal Represents the reward obtained when the vehicle reaches the target point, that is, the reward is obtained when the vehicle reaches the target point, and the target point information is obtained from the vehicle's driving task; R safety It is an indicator used to measure the safety of the vehicle when performing tasks. For example, when the vehicle chooses an unsafe decision such as collision or driving off the road, it will receive a large negative reward. efficiency It is used to evaluate the time taken by the vehicle to complete the driving task. The shorter the time taken by the vehicle to complete the task, the greater the reward it receives. This setting can avoid long-term parking of the vehicle; R vel Used to measure the speed change of the vehicle. For example, the present invention sets the expected speed of the vehicle to 5m / s. When the speed of the vehicle is too high or too low, the reward value becomes smaller.
[0101] Step S404: Set the state-action value function Q(S,O) to be T×N o A matrix where any element Q tj Indicates that the vehicle is in state S t Next select Action O j The expected value of the cumulative reward that can be obtained after , is used to evaluate the pros and cons of the vehicle taking a specific action in a specific state, so as to guide the vehicle to make the best decision; and the matrix Q i All elements of are initialized to 0;
[0102] The above is the modeling process using SMDP. The process of obtaining the Q matrix using the Q-Learning algorithm is as follows:
[0103] Step S405: Set the number of cycles to C, and record the Q matrix obtained in the ith cycle as Q i ; For the first loop, set index i = 1; the loop process is as follows Figure 7 As shown, including:
[0104] Step SS0, set time index t=0;
[0105] Step SS1: For the current vehicle environment state S t , select a behavior decision O from the action space O jTake action and observe the selected action O j The reward R generated tj and the vehicle's next environmental state S t+1 ;
[0106] Step SS2: Use the simplified Bellman equation to calculate Q i Elements in The value of is updated, and the updated element for:
[0107] Q i (S t ,O j )←R tj +γ′max o′ Q i (S t+1 ,O′);
[0108] Wherein, γ′ is the discount factor in the Bellman equation, and illustratively, the present invention takes γ′=0.99; max o′ Q i (S t+1 , O′) indicates that the vehicle reaches the environmental state S t+1 When all possible actions are i The maximum value max o′ Q i (S t+1 ,O′);
[0109] Step SS3: Determine the current vehicle environment state S t Is it the target state S? T If not, set t=t+1, repeat steps SS1-SS3 until the vehicle's environmental state reaches the target state, and then execute step SS4; if yes, directly execute step SS4;
[0110] Step SS4: Observed Q i Check whether the matrix converges. If so, use it as the final Q matrix. If not, set i=i+1 and repeat steps SS0-SS4 until the final Q matrix is obtained.
[0111] After obtaining the final Q matrix, the best behavior decision basic unit combination can be read from it. The operation is as follows:
[0112] By reading the elements of the final Q matrix row by row and combining the actions corresponding to the column with the maximum value of each row element, we can get the optimal combination of basic behavioral decision units to complete the target task in the target scenario.
[0113] Preferably, the present invention adopts the ε-greedy algorithm to select the action O for the vehicle j ,in In the early stage of system trial, a larger ε value is set to enhance the system's exploration ability; after multiple attempts, the ε value is lowered to enhance the system's utilization ability and select the optimal strategy from the explored information as much as possible.
[0114] In general, compared with the existing learning-based behavior decision-making methods, the method of the present invention has the following beneficial effects:
[0115] 1. The present invention generalizes the constructed behavior decision library to obtain richer behavior decisions that are suitable for various driving scenarios. When the vehicle scenario is different from the constructed simulation scenario, a behavior decision combination suitable for the current scenario can also be selected from the generalized behavior decision library; compared with the learning-based behavior decision method suitable for a single scenario, the method of the present invention can avoid the tedious work of repeatedly building scenarios, collecting data, training models, etc.
[0116] 2. When acquiring each behavior decision, the original collected data is smoothed to reduce errors such as trajectory drift caused by the delay between vehicle operation information and vehicle model communication, thereby improving the accuracy and reliability of the constructed behavior decision library;
[0117] 3. Based on the generalized behavior decision library, the reinforcement learning algorithm is adopted, and the reward function is set according to actual needs. It can efficiently obtain the best behavior decision basic unit combination to complete specific tasks in actual application scenarios, and has strong scene migration and adaptability.
[0118] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0119] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A cross-scenario behavior decision generalization method for intelligent vehicles, characterized in that: The following steps are involved: Simulate the driving process in the simulation scene and collect vehicle data during the driving process; Based on the data, a behavior decision basic unit set for each driving behavior is obtained to form a behavior decision basic unit library; any of the behavior decision basic unit sets includes a plurality of behavior decision basic units corresponding to the driving behavior, and each behavior decision basic unit includes a control sequence P, a trajectory r and a speed v; Generalize each behavior decision basic unit in each behavior decision basic unit set to obtain a generalized behavior decision basic unit library; For any target task in any target scene, based on the generalized behavior decision basic unit library, a reinforcement learning algorithm is used to learn the best behavior decision basic unit combination to complete the target task; wherein, the reinforcement learning algorithm is SMDP+Q-Learning algorithm; wherein, The SMDP is used for modeling, and the process includes: step S401, obtaining the environmental state S of the vehicle at each moment t , t=0,1,2,…,T; Step S402, based on the generalized behavior decision basic unit library, obtain the vehicle action set O, including N o Step S403: Set the reward function R generated by the action, which is related to the state of the vehicle after the action occurs, that is, the state of the vehicle at the next moment; Step S404: Set the state-action value function Q(S,O), which is T×N o A matrix where any element Q tj Indicates that the vehicle is in state S t Next select Action O j The expected value of the cumulative reward that can be obtained after ; and initialize all elements in Q(S,Q) to 0; The Q-Learning algorithm is used to obtain the Q matrix, and the steps include: step S405, setting the number of cycles to C, and the Q matrix obtained in the i-th cycle is recorded as Q i ; In the first cycle, set index i = 1; The cycle process includes: step SS0, set time index t = 0; step SS1, for the current vehicle environment state S t , select a behavior decision O from the action space O j Take action and observe the selected action O j The reward R generated tj and the vehicle's next environmental state S t+1 ; Step SS2, using the simplified Bellman equation, Q i Elements in The value of is updated, and the updated element for: Q i (S t ,O j )←R tj +γ′max o′ Q i (S t+1 ,O′); Where γ′ is the discount factor in the Bellman equation; max o′ Q i (S t+1 , O′) indicates that the vehicle reaches the environmental state S t+1 When all possible actions are i The maximum value max o′ Q i (S t+1 , O′); Step SS3, determine the current vehicle environment state S t Is it the target state S? T If not, set t = t + 1, repeat steps SS1 to SS3 until the vehicle's environmental state reaches the target state, and then execute step SS4; if yes, directly execute step SS4; step SS4, observe the Q i Whether the matrix converges, if so, use it as the final Q matrix, if not, set i=i+1, and repeat steps SS0-SS4 until the final Q matrix is obtained; The best action decision basic unit combination is read from the Q matrix.
2. The cross-scenario behavior decision generalization method of intelligent vehicles according to claim 1 is characterized in that: The control sequence P is represented by {Th(t), Br(t), Str(t)}; wherein Th(t) is the sequential sequence of the accelerator pedal control amount, Br(t) is the sequential sequence of the brake pedal control amount, and Str(t) is the sequential sequence of the steering wheel angle control amount; The generalization of each behavior decision basic unit in each behavior decision basic unit set includes: The sequential sequence Th(t) of the accelerator pedal control amount and the sequential sequence Str(t) of the steering wheel angle control amount corresponding to the behavior decision basic unit are multiplied by the coefficients γ and η respectively to obtain the control sequence P′: The above control sequence P′ is input into the simulation platform, and the trajectory r′ of the vehicle under this control sequence is calculated; Set the initial speed of the vehicle, input it and the above control sequence P′ into the simulation platform, and calculate the speed v′ of the vehicle under this control sequence; The new behavior decision basic unit includes: control sequence P′, trajectory r′ and speed v′.
3. The cross-scenario behavior decision generalization method of intelligent vehicles according to claim 2 is characterized in that: Multiple behavior decision basic units in the behavior decision basic unit set of any driving behavior D are obtained in the following way: According to the driving behavior, the vehicle data collected during the driving process is split to obtain data subsets corresponding to multiple driving behaviors; Select D corresponding to driving behavior D from all data subsets n a subset of data; Extract D n The trajectory r, velocity v and control sequence P in each subset of the data are obtained, that is, D n The basic unit of behavioral decision-making; Each behavior decision basic unit can complete driving behavior D.
4. The cross-scenario behavior decision generalization method of intelligent vehicles according to claim 3 is characterized in that: Acquiring multiple behavior decision basic units in a behavior decision basic unit set of any driving behavior D, further comprising: Right D n The trajectory r, velocity v and control sequence P in the data subset are smoothed using the local weighted regression scatter point smoothing method.
5. The cross-scenario behavior decision generalization method of intelligent vehicles according to claim 1 is characterized in that: The operation of reading the optimal behavior decision basic unit combination from the Q matrix is as follows: The elements of the final Q matrix are read row by row, and the actions corresponding to the columns with the maximum values of the elements in each row are combined to form the optimal basic unit combination of behavioral decisions to complete the target task in the target scenario.
6. The cross-scenario behavior decision generalization method of intelligent vehicles according to claim 1 is characterized in that: The method selects a behavior decision basic unit O from the action space O. j Take action and execute it using the ε-greedy algorithm, where the probability of each action being selected is set to 7. The cross-scenario behavior decision generalization method of intelligent vehicles according to claim 1 is characterized in that: The vehicle's environmental state S at any moment t Including the real-time position and speed of the main vehicle, as well as the real-time position and speed of the surrounding interactive vehicles: S t ={x host ,y host ,θ host ;v host ;x other ,y other ,θ other ;v other } t Among them, x host ,y host ,θ host Indicates the position information of the main vehicle, x other ,y other ,θ other represents the position information of the interactive vehicles around the main vehicle, v host and v other Represent the velocity matrices of the host vehicle and surrounding vehicles respectively; Each behavior decision basic unit O in the action space O j They all belong to the generalized basic unit library of behavioral decision-making; The reward function R is set to: R=R goal +R safety +R efficiency +R vel R goal ＝100 R safety ＝-200 R efficiency =-0.5t Among them, R goal Represents the reward obtained when the vehicle reaches the target point; R safety It is an indicator used to measure the safety of vehicles when performing tasks; R efficiency Used to evaluate the time taken by the vehicle to complete the driving task; R vel Used to measure changes in vehicle speed.