Driving decision method and device

Abstract

The present application relates to the field of autonomous driving, and discloses a driving decision method and device for improving the robustness of driving decision strategies and ensuring the optimal output decision results. The method comprises: constructing a Monte Carlo tree based on the current driving environment state, the Monte Carlo tree comprising a root node and N-1 non-root nodes, each node representing a driving environment state, wherein the root node represents the current driving environment state, the driving environment state represented by any non-root node, and the driving action prediction of the non-root node based on the driving environment state represented by the parent node of the non-root node and the parent node of the non-root node through the driving environment random model; according to the number of visits and / or value function of each node in the Monte Carlo tree, determining a node sequence starting from the root node and ending at the leaf node in the Monte Carlo tree; determining a driving action sequence according to the driving actions corresponding to each node in the node sequence.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of autonomous driving, and in particular to a driving decision-making method and device. Background Art

[0002] With the advent of the era of autonomous driving, intelligent vehicles with autonomous driving functions have become the focus of research by major manufacturers. At present, autonomous driving includes assisted driving and fully autonomous driving, and the key technologies for its implementation include: environmental perception, driving decision-making, and control execution. Among them, driving decisions are based on the perceived information of traffic participants and give driving actions for the vehicle to execute.

[0003] At present, driving decisions are usually made based on the reinforcement learning (deep Q network, DQN) algorithm. The DQN model built by the DQN algorithm learns the driving environment status of a large number of vehicles at a certain moment (such as the speed of the own vehicle, the speed of adjacent vehicles, etc.), as well as the driving actions output by the vehicle driver based on the driving environment status at that moment (such as changing lanes to the left, changing lanes to the right, etc.). When making driving decisions, the driving actions that the autonomous driving vehicle needs to perform can be obtained based on the current driving environment status of the vehicle through the DQN model.

[0004] However, when outputting driving actions, driving decisions based on the DQN model do not take into account the impact of the output driving actions on the subsequent driving of the vehicle, making it difficult to ensure that the output decision results are optimal. Summary of the invention

[0005] The embodiments of the present application provide a driving decision-making method and device to improve the robustness of the driving decision-making strategy and ensure that the output decision result is optimal.

[0006] In a first aspect, an embodiment of the present application provides a driving decision-making method, the method comprising: constructing a Monte Carlo tree based on a current driving environment state, wherein the Monte Carlo tree comprises N nodes, each node represents a driving environment state, the N nodes comprise a root node and N-1 non-root nodes, wherein the root node represents the current driving environment state, the driving environment state represented by the first node is obtained by predicting the driving environment state represented by the parent node of the first node through a driving environment random model, and a first driving action, wherein the first driving action is a driving action determined by the parent node of the first node in the process of expanding the first node (that is, the first driving action is a driving action of the parent node of the first node extending the first node), and the first node is the N-1 Any node in the non-root node, wherein N is a positive integer greater than or equal to 2; according to the number of visits and / or the value function of each node in the Monte Carlo tree, a node sequence starting from the root node and ending at the leaf node is determined in the Monte Carlo tree; according to the driving actions corresponding to each node contained in the node sequence, a driving action sequence is determined, and the driving action sequence is used for driving decision-making; wherein the number of visits to each node is determined according to the number of visits to the child nodes of the node and the initial number of visits to the node, and the value function of each node is determined according to the value function of the child nodes of the node and the initial value function of the node, wherein the initial number of visits to each node is 1, and the initial value function is determined according to the value function matching the driving environment state represented by the node.

[0007] By adopting the above method, through the random model of the driving environment, it is possible to predict various driving environment states that the autonomous driving vehicle may face in multiple driving actions in the future, construct a Monte Carlo tree, and determine the driving action sequence that is most conducive to vehicle driving based on the various driving environment states that the autonomous driving vehicle may face in multiple driving actions in the future, thereby improving the robustness of the driving decision-making strategy and ensuring that the output decision result is optimal.

[0008] In a possible design, the driving environment state represented by the first node is predicted based on the driving environment state represented by the parent node of the first node and the first driving action through the driving environment random model, including: using Dropout forward propagation through the driving environment random model to predict the probability distribution of the driving environment state after executing the first driving action based on the driving environment state represented by the parent node of the first node; and sampling the driving environment state represented by the first node from the probability distribution.

[0009] In the above design, the Dropout forward propagation method is adopted through the driving environment random model to predict the driving environment state represented by the parent node of the first node and the probability distribution of the driving environment state after executing the first driving action; and the driving environment state represented by the first node is sampled from the probability distribution. When expanding the nodes in the Monte Carlo tree, the uncertainty of the driving environment state is fully considered, the diversity of node expansion is increased, and the driving decision-making strategy is more robust.

[0010] In one possible design, the initial value function of the node is determined based on the value function that matches the driving environment state represented by the node, including: selecting a first number of target driving environment states that have the highest matching degree with the driving environment state represented by the node from an episode memory repository; and determining the initial value function of the node based on the value functions corresponding to the first number of target driving environment states.

[0011] In the above design, by introducing an episode memory repository for the Monte Carlo tree, the initial value function of the node can be accurately and quickly estimated based on historical experience data, avoiding estimating the initial value function of the node through inefficient expansion, reducing the computational overhead brought by estimating the initial value function of the node, and helping to improve the efficiency of Monte Carlo tree search.

[0012] In a possible design, the method also includes: after executing the first driving action in the driving action sequence, obtaining the actual driving environment state after executing the first driving action; and updating the driving environment random model according to the current driving environment state, the first driving action, and the actual driving environment state after executing the first driving action.

[0013] In the above design, the random model of the driving environment can be continuously trained and updated, which is beneficial to improving the accuracy of the random model of the driving environment.

[0014] In one possible design, determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree includes: determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the principle of maximum number of visits according to the number of visits of each node in the Monte Carlo tree; or, determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the principle of maximum value function according to the value function of each node in the Monte Carlo tree; or, determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the principle of maximum number of visits first and maximum value function second according to the number of visits and the value function of each node in the Monte Carlo tree.

[0015] In the above design, multiple node sequence determination methods are provided to help meet the needs of different driving decisions.

[0016] In a possible design, the method also includes: after the driving scenario ends, determining the cumulative reward return value corresponding to the real driving environment state after each driving action in the driving scenario is executed according to the driving results; using the cumulative reward return value corresponding to the real driving environment state after each driving action is executed as its corresponding value function, and updating the scenario memory storage repository.

[0017] In the above design, updating the episodic memory repository according to the cumulative reward return value is conducive to accurately determining the initial value function of the node in the Monte Carlo tree, thereby ensuring the reliability of the output decision results.

[0018] In a second aspect, an embodiment of the present application provides a driving decision-making device, comprising a unit for executing each step in the above-mentioned first aspect or any possible design of the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a driving decision-making device, comprising a processor and a memory, wherein the memory is used to store computer programs or instructions, and the processor is used to call the computer programs or instructions stored in the memory to execute the method described in the first aspect or any possible design of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium has a computer program or instructions for executing the method described in the above-mentioned first aspect or any possible design of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed, can implement the method described in the above-mentioned first aspect or any possible design of the first aspect.

[0022] In a sixth aspect, the present application also provides a chip, wherein the chip is used to implement the method described in the above-mentioned first aspect or any possible design of the first aspect.

[0023] The technical effects that can be achieved in the second to sixth aspects mentioned above can refer to the technical effects that can be achieved in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of an autonomous driving system provided in an embodiment of the present application;

[0025] Figure 2A schematic diagram of a tree structure provided for an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the Dropout processing effect provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the driving decision process provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the environment in which the autonomous driving vehicle is located provided in an embodiment of the present application;

[0029] Figure 6 One of the Monte Carlo tree structure diagrams provided in the embodiment of the present application;

[0030] Figure 7 The second schematic diagram of the Monte Carlo tree structure provided in the embodiment of the present application;

[0031] Figure 8 One of the schematic diagrams of the driving decision-making device provided in the embodiment of the present application;

[0032] Fig. 9 This is a second schematic diagram of the driving decision-making device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] Figure 1 This is a schematic diagram of an autonomous driving system that can be applied in an embodiment of the present application, including an environment perception module, a behavior decision module based on an efficient Monte Carlo tree search (MCTS), a path planning module, and a control execution module, wherein the behavior decision module based on the efficient MCTS is the core module for driving decision-making in an embodiment of the present application.

[0034] Environmental perception module: This module is used to perceive the state of the surrounding environment and obtain the current state of the driving environment of the vehicle. Specifically, it integrates the information transmitted by various sensors to judge the state of the vehicle, including the position, speed, heading angle, etc. of the vehicle; judges the state of the surrounding environment, including the position of static targets (lane lines and road boundaries, green belts, etc.), the position and speed of dynamic targets (social vehicles and pedestrians, etc.), road conditions, weather conditions (temperature, humidity, light), etc., and transmits this information to the path planning module and the behavior decision control module based on the efficient MCTS.

[0035] Behavior decision module based on efficient MCTS: Based on the current driving environment state of the vehicle, it outputs the future multi-step driving action sequence.

[0036] Path planning module: According to the status of the self-vehicle and other vehicles and the determined driving action sequence, a path trajectory with position and speed information is planned to complete the driving action sequence.

[0037] Control execution module: This module outputs reasonable control quantities by controlling the vehicle's steering wheel, accelerator and brake pedals, so that the vehicle's driving trajectory can track the planned path trajectory.

[0038] The driving decision-making solution provided in the embodiment of the present application can be applied to applications such as Figure 1 In the subsequent description of the embodiments of the present application, the vehicle-mounted equipment of the autonomous driving system, the autonomous driving vehicle, etc. are used as shown. Figure 1 The autonomous driving system shown is illustrated by taking the autonomous driving vehicle as an example.

[0039] Before introducing the embodiments of the present application, some terms in the present application are first explained to facilitate understanding by those skilled in the art.

[0040] 1) Tree structure (tree) is a data structure, which is a hierarchical set of n nodes, n is usually a positive integer not less than 1. It is called a "tree" because it looks like an upside-down tree, that is, its roots are facing upwards and its leaves are facing downwards. The nodes without child nodes in the tree are leaf nodes, and the top node of the tree is the root node of the tree. For example: Figure 2 In the tree structure shown, node 11 is located at the first level of the tree; nodes 21, 22 and 23 are located at the second level of the tree; nodes 31, 32, 33 and 34 are located at the third level of the tree, node 11 is the root node of the tree, and nodes 31, 32, 33, 34 and 23 are leaf nodes of the tree. The Monte Carlo tree involved in the embodiments of the present application is also a tree structure.

[0041] 2) Monte Carlo Tree Search (MCTS) is a heuristic search algorithm used for certain decision-making processes. It is usually used in games. Its main purpose is to select the next step with the highest winning rate under a given game state. The main concept of Monte Carlo Tree Search is search, which is a set of traversal processes along the Monte Carlo tree (also known as the game tree). The path of a single traversal will extend from the root node (that is, the current game state) to the node that is not fully expanded. The node that is not fully expanded means that at least one of its child nodes has not been visited (or has not been expanded). When encountering an incompletely expanded node, one of its unvisited child nodes will be expanded. The expanded child node will calculate its corresponding initial value function according to a certain strategy, where the initial value function can also be called the initial value or simulation result. For example, for a chess game, based on the chess game represented by the expanded child node, according to the fast rollout policy, etc., until the end of the game, a simulation result (such as loss or win) is obtained, that is, the initial value function of the child node is obtained (such as 0 / 1 for loss and 1 / 1 for win). After obtaining the initial value function of the child node, it will be back-propagated back to the root node of the current Monte Carlo tree, and the initial value function of the child node (simulation result) will be added to all the ancestor nodes of the child node. For example, if the initial value function of the child node is 0 / 1 (indicating that the game is lost), then add 0 / 1 to the value function of all the ancestor nodes of this node. Once the search time or the maximum search step length of the Monte Carlo tree (i.e., the maximum number of nodes expanded from the root node) is reached, the search is terminated, and a decision is made based on the value function of each node in the Monte Carlo tree. It should be noted that in the Monte Carlo tree, each node represents a state (also called a situation, such as a chess game), and each node records the statistical data of the node, such as the value function of the node, the number of visits, etc. The value function recorded for the node can also be called the value of the node, which can be a specific value.

[0042] 3) Dropout refers to discarding some neurons in the model with probability P, and the output of the discarded neurons is set to zero. Figure 3 As shown in Figure 1, it is a schematic diagram of a model after Dropout processing. Figure 3 (A) is a schematic diagram of the structure of the model before Dropout processing. Figure 3 (B) is a schematic diagram of the structure of the model after Dropout processing. It can be seen that some neurons in the model after Dropout processing are temporarily discarded and the output is zero.

[0043] In addition, it should be understood that in the embodiments of the present application, at least one can also be described as one or more, and multiple can be two, three, four or more, and the present application does not limit it. In the embodiments of the present application, " / " can indicate that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" can be used to describe that there are three relationships between associated objects, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations, and any embodiment or design described as "exemplary" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or design. The use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way for easy understanding.

[0044] The embodiment of the present application aims to predict various driving environment states that the autonomous driving vehicle may face in multiple driving actions in the future through a random model of the driving environment, construct a Monte Carlo tree, and determine the driving action sequence that is most conducive to the vehicle's driving based on various driving environment states that the autonomous driving vehicle may face in multiple driving actions in the future, so as to improve the robustness of the driving decision-making strategy and ensure the optimal output decision result. At the same time, the initial value function of the node in the Monte Carlo tree can also be estimated through the episode memory repository, thereby reducing the computational overhead caused by estimating the node initial value function and improving the efficiency of the Monte Carlo tree search.

[0045] The following takes the lane change decision scenario as an example and describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0046] Figure 4 A schematic diagram of a driving decision process provided in an embodiment of the present application, the process includes:

[0047] S401: The autonomous driving vehicle constructs a Monte Carlo tree based on the current driving environment state.

[0048] The Monte Carlo tree includes N nodes, each node represents a driving environment state, and the N nodes include a root node and N-1 non-root nodes, wherein the root node represents the current driving environment state, the driving environment state represented by the first node is predicted based on the driving environment state represented by the parent node of the first node through a driving environment random model, and a first driving action, wherein the first driving action is a driving action determined by the parent node of the first node in the process of expanding the first node, the first node is any one of the N-1 non-root nodes, and N is a positive integer greater than or equal to 2.

[0049] In the lane change decision scenario, the environmental state of the autonomous driving vehicle may include the speed of the autonomous driving vehicle, the relative speed and relative distance of the social vehicle in front of the autonomous driving vehicle (in the direction of the autonomous driving vehicle), the arrival time of the social vehicle behind the autonomous driving vehicle (in the opposite direction of the autonomous driving vehicle) (i.e., the time required to catch up with the autonomous driving vehicle), etc. The driving environment state of the autonomous driving vehicle may be composed of the environmental states of T historical moments including the current moment, where T is a positive integer greater than or equal to 1. For example: Taking the current moment as 10:00:00 as an example, the current driving environment state of the autonomous driving vehicle may be composed of the environmental states of the autonomous driving vehicle at 8 historical moments: 9:56:30, 9:57:00, 9:57:30, 9:58:00, 9:58:30, 9:59:00, 9:59:30, and 10:00:00.

[0050] The environmental status of the autonomous vehicle can be obtained through the on-board sensors of the autonomous vehicle. For example: the on-board sensors of the autonomous vehicle include speed sensors, acceleration sensors, distance sensors (such as radar distance sensors), etc. The speed sensor can measure the speed of the autonomous vehicle, the acceleration sensor can measure the acceleration of the autonomous vehicle, and the distance sensor can measure the relative distance between the autonomous vehicle and the social vehicle. The autonomous vehicle can determine the relative speed of the social vehicle relative to the autonomous vehicle based on the change in the relative distance between the autonomous vehicle and the social vehicle; and can determine the arrival time of the social vehicle behind the autonomous vehicle based on the relative distance between the social vehicle and the autonomous vehicle and the relative speed of the social vehicle relative to the autonomous vehicle.

[0051] In a possible implementation, the autonomous driving vehicle can also learn the relative speed or relative distance of the social vehicle relative to the autonomous vehicle by communicating with the social vehicle. For example, the autonomous driving vehicle can receive the speed and position information sent by the social vehicle, and determine the relative speed and relative distance of the social vehicle relative to the autonomous vehicle based on the speed and position information of the autonomous vehicle.

[0052] Take the environment of the autonomous driving vehicle at a certain moment as Figure 5 As shown in the figure, for example, the autonomous driving vehicle can obtain the relative distance (△D) and relative speed (△V) of the vehicles in the three lanes in front of the vehicle, the arrival time (TTA) of the vehicles in the three lanes behind the vehicle, and the speed (V) of the vehicle. Figure 5L, M, and R represent the social vehicles in the left, middle, and right lanes of the autonomous driving vehicle, respectively. In the embodiment of the present application, the environmental state of T historical moments corresponding to the current moment can be used as the current driving environment state. When making driving decisions, the autonomous driving vehicle can use the current driving environment state of the autonomous driving vehicle as the root node of the Monte Carlo tree to construct a Monte Carlo tree. Assume Figure 5 is the current environment of the autonomous driving vehicle, then the current driving environment state of the autonomous driving vehicle (S t ) can be expressed as S t =(△D i，t-T：t , △V i，t-T：t , V t-T：t , TTA j，t-T：t ), where i∈(L, M, R), j∈(L, R), t represents the current moment, and tT:t represents the T historical moments corresponding to the current moment.

[0053] In the lane change decision scenario, the optional driving actions of the autonomous driving vehicle include one or more of changing lanes to the left, keeping straight, and changing lanes to the right. The training of the random model of the driving environment can be achieved by pre-collecting the driving environment states of a large number of vehicles before and after performing a certain driving action. Specifically, the driving environment states of a large number of vehicles before and after performing a certain driving action and the driving actions performed can be collected in advance as sample pairs to construct a training set for training the random model of the driving environment. Each sample pair in the training set can be expressed as (S t +A t , S t+1 ), where A t Indicates a driving action performed by the vehicle, which can be one of changing lanes to the left, keeping straight, or changing lanes to the right. t Indicates that the vehicle executes A t The current driving environment state, S t+1 Indicates that the vehicle executes A t The driving environment status after that.

[0054] It should be understood that in the embodiments of the present application, the driving environment state before the vehicle performs the driving action generally refers to the driving environment state of the vehicle when the vehicle starts to perform the driving action, and the driving environment state after the vehicle performs the driving action generally refers to the driving environment state of the vehicle when the vehicle finishes performing the driving action. For example: the vehicle performs a driving action of changing lanes to the left at 10:10:00-10:10:10, and the driving environment state of the vehicle at 10:10:00 can be used as the driving environment state of the vehicle before performing the driving action of changing lanes to the left, and the driving environment state of the vehicle at 10:10:10 can be used as the driving environment state of the vehicle after performing the driving action of changing lanes to the left.

[0055] In the random model of driving environment (f θ (s,a,z)) for training, the S in the sample pair can be t +A t Input into the driving environment random model, and obtain the output of the driving environment random model based on S t Execute A t The driving environment state after t+1 '). According to the output of the driving environment model S t+1 ' and the true S in the sample t+1 , the loss function can be used to calculate the loss of the random model of the driving environment. The higher the loss, the lower the output S of the random model of the driving environment. t+1 'With the real S t+1 The greater the difference, the driving environment random model adjusts the parameters in the driving environment random model according to the loss. For example, the random gradient descent method is used to update the parameters of the neurons in the driving environment random model. Then the training process of the driving environment random model becomes a process of minimizing the loss as much as possible. The driving environment random model is continuously trained through the samples in the training set. When the loss is reduced to the preset range, the trained driving environment random model can be obtained. The hidden variable z of the driving environment random model can be used to characterize the uncertainty of the model. The driving environment random model can be selected as a deep neural network, a Bayesian neural network, etc.

[0056] Based on the trained random model of the driving environment, the autonomous driving vehicle can predict various driving environment states that the autonomous driving vehicle may face in multiple driving actions in the future based on the current driving environment state, and build a Monte Carlo tree. Specifically, the autonomous driving vehicle can select nodes layer by layer starting from the root node of the Monte Carlo tree. When the selected target node has one or more optional driving actions that are not predicted, a target driving action is selected, and the driving environment state after the autonomous driving vehicle performs the target driving action based on the target node is predicted by the random model of the driving environment, and the driving environment state represented by a child node of the target node is used to expand the Monte Carlo tree.

[0057] In one possible implementation, in order to fully consider the uncertainty of other social vehicles in the driving environment, when predicting the driving environment state of the autonomous driving vehicle after performing the target driving action based on the target node through the driving environment random model, the driving environment random model is used to obtain the probability distribution of the driving environment state after the autonomous driving vehicle performs the target driving action based on the target node by using the Dropout forward propagation method, and the driving environment state is sampled from the probability distribution as the driving environment state after the autonomous driving vehicle performs the target driving action based on the target node.

[0058] Example: An extended Monte Carlo tree for self-driving cars Figure 6 As shown, the autonomous driving vehicle starts from the root node 11 of the Monte Carlo tree and selects nodes layer by layer (such as layer by layer traversal selection, etc.), and selects node 21. The unpredicted optional driving actions (A) of node 21 include changing lanes to the left, keeping straight, and changing lanes to the right. The autonomous driving vehicle can randomly select a target driving action from changing lanes to the left, keeping straight, and changing lanes to the right. For example: randomly selecting the target driving action changing lanes to the left (A) from changing lanes to the left, keeping straight, and changing lanes to the right t After selecting the target driving action, the autonomous driving vehicle predicts the execution of the target driving action (A) based on node 21 by using the Dropout forward propagation method multiple times based on the random model of the driving environment. t ) after the multiple possible driving environment states, and according to the multiple possible driving environment states, calculate and execute the A t The mean (μ) and variance (δ) of the predicted driving environment state after 2 ). The A is obtained by Gaussian sampling based on the mean and variance. t The probability distribution of the driving environment state value after s′=N(μ,δ 2 ), and sample from the probability distribution (e.g., randomly draw one) to execute the A t The subsequent driving environment state expands a child node (node ​​31) of node 21.

[0059] After expanding a child node of the target node, the autonomous driving vehicle initializes the statistical data access count (N) and value function (Q) of the child node. That is, it is necessary to determine the initial access count and initial value function of the child node. For the initial access count of the expanded child node, the autonomous driving vehicle sets the initial access count of the child node to 1, and traces back from the child node to the root node, and updates the access count of each node on the node path (search path) corresponding to the child node. For example: add 1 to the access count of each node on the node path corresponding to the expanded child node, that is, N=N'+1, where N is the updated access count, and N' is the access count before the update, that is, the access count of each node in the Monte Carlo tree is the sum of the access count of all child nodes of the node and the initial access count of the node. It should be understood that in the present application, the node path (search path) corresponding to a node refers to a node path composed of all ancestor nodes of the node. For example: After the autonomous driving vehicle expands the child node (node ​​31) of node 21, the initial visit count of node 31 is updated to 1, and the visit counts of nodes 21 and 11 on the node path (search path) of node 31 are increased by 1 respectively, thus completing the update of the visit count of each node on the node path corresponding to node 31.

[0060] For the initial value function of the expanded child node, the autonomous driving vehicle can determine the initial value function of the child node according to the episode memory repository (EM). If the driving environment state represented by the child node is recorded in the episode memory repository, the value function corresponding to the driving environment state stored in the episode memory repository is directly output as the initial value function of the child node; otherwise, the first number of target driving environment states with the highest degree of match with the driving environment state represented by the child node can be selected from the episode memory repository; the initial value function of the child node is determined according to the value functions corresponding to the first number of target driving environment states. The first number (K) can be 3, 5, etc. For example: the autonomous driving vehicle can select the K target driving environment states with the highest degree of match with the driving environment state represented by the expanded child node from the episode memory repository, and use the average of the value functions corresponding to the K target driving environment states as the initial value function of the child node.

[0061] After determining the initial value function of the expanded child node, the autonomous driving vehicle traces back from the child node to the root node and updates the value function of each node on the node path corresponding to the child node. L Update the value function of each node on the node path corresponding to the child node, where Q is the value function of the node after update, Q' is the value function of the node before update, and Q L is the initial value function of the child node, that is, the value function of each node in the Monte Carlo tree is the sum of the value functions of all child nodes of the node and the initial value function of the node. In another possible implementation, Q=Q'+(Q L -Q') / N updates the value function of each node on the node path corresponding to the child node, where Q is the value function of the node after update, Q' is the value function of the node before update, and Q L is the initial value function of the child node, and N is the number of visits after the node is updated.

[0062] In an embodiment of the present application, the maximum number of steps for expanding the Monte Carlo tree can be limited, that is, the maximum number of nodes that can be expanded starting from the root node of the Monte Carlo tree is limited. For example, the maximum number of steps for expanding the Monte Carlo tree is limited to 20 steps, and the expansion of the Monte Carlo tree is stopped when the maximum number of steps is reached. In addition, in an embodiment of the present application, each time a leaf node of the Monte Carlo tree is expanded (that is, a child node of a node in the Monte Carlo tree is expanded), if the number of steps for expanding the Monte Carlo tree does not meet the maximum number of steps for expanding the Monte Carlo tree, return to the step of selecting nodes layer by layer starting from the root node of the Monte Carlo tree, and continue to expand the leaf nodes of the Monte Carlo tree. Among them, the nodes are selected layer by layer starting from the root node of the Monte Carlo tree, and the selection method can be based on the selection strategy of the upper confidence bound apply to tree (UCT) algorithm, and the nodes corresponding to the driving action (st+1) are selected layer by layer. The selected driving action (A t ) is compared with other optional driving actions, satisfying the maximization of the corresponding value function (Q) of the driving action and the exploration weighted term The sum of the selected driving actions makes the selected driving actions balanced between the nodes maximizing the value function and the nodes with low visit times, thus ensuring the optimality of driving action selection. Where Q is the value function of the selection node, in the exploration weighted term, C is the weight coefficient of the exploration term, N'' is the number of visits to the selection node, and N is the number of visits to the leaf node of the optional action (A) corresponding to the selection node.

[0063] S402: The autonomous driving vehicle determines a node sequence starting from a root node and ending at a leaf node in the Monte Carlo tree based on the number of visits and / or the value function of each node in the Monte Carlo tree.

[0064] In an embodiment of the present application, a node sequence starting from the root node and ending at the leaf node can be determined in the Monte Carlo tree according to one of the principles of maximum number of visits, maximum value function, or maximum number of visits first and maximum value function second.

[0065] For example, we take the example of determining a node sequence from the root node to the leaf node in the Monte Carlo tree according to the principle of maximum number of visits. Figure 7 As shown, starting from node 11 (root node), the child node with the maximum number of visits to node 11 is node 21, the child node with the maximum number of visits to node 21 is node 31, and the child node with the maximum number of visits to node 31 is node 41, and the node sequence is determined to be node 11-node 21-node 31-node 41.

[0066] S403: Determine a driving action sequence according to the driving actions corresponding to the nodes included in the node sequence, wherein the driving action sequence is used for driving decision-making.

[0067] In the embodiment of the present application, the driving action corresponding to a node in the Monte Carlo tree is the driving action of the node extended by the parent node of the node, wherein the root node has no corresponding driving action. Figure 7 As shown, the driving action of node 22 expanded by parent node 11 of node 22 is to keep going straight, and the driving action corresponding to node 22 is to keep going straight. Taking the determined node sequence of node 11-node 21-node 31-node 41 as an example, the driving action sequence is change lane to the left-change lane to the left-keep going straight.

[0068] In order to ensure the reliability of the random model of the driving environment, in a possible implementation, after the autonomous driving vehicle performs the first driving action in the driving action sequence, the driving environment state before the autonomous driving vehicle performs the first driving action + the first driving action and the driving environment state after the first driving action can be used as a new sample pair (S t +A t , S t+1 ), added to the training set, and the random model of the driving environment is updated.

[0069] Example: The stochastic model of the driving environment can be updated in the following way. Minimum batch samples are sampled from new samples added to the training set Where M is the number of samples in the minimum batch sample set, (s, a, s') represents a sample pair, such as a (S t +A t , S t+1 ). Calculate the loss function of the random model of the driving environment, and update the random model of the driving environment according to the stochastic gradient descent method so that the error between the predicted value of the random model of the driving environment and the actual perception result is minimized. The loss function can be It consists of two items. The first item The mean square error is used to represent the predicted value f of the driving environment state after the autonomous driving vehicle performs the driving action by the random model of the driving environment θ (s i , a i , z i ) and the true observation value s′ i The second term λD KL [q φ (z;s i , s′ i )||p(z)] is a regularization term introduced for the latent variable z, constraining the estimated distribution q of the latent variable z φ (z; si, s′ i ) and the KL divergence between the prior hypothesis distribution p(z) to prevent overfitting, and λ is the proportional coefficient for adjusting the regularization strength.

[0070] When the driving scenario is over, such as when the autonomous vehicle reaches the destination or an accident occurs and the vehicle stops driving, the autonomous vehicle can get a reward value based on whether it has reached the destination, etc., and can determine the scenario trajectory sequence (in chronological order) based on the actual driving environment state after each driving action in the driving scenario. For each step in the reverse scenario trajectory sequence (the scenario trajectory sequence is from back to front), the autonomous vehicle calculates each driving environment state (S t ) corresponding to the driving environment state action pair (S t , A t ) of the discounted cumulative return, where A t For autonomous vehicles in S t The driving action is performed based on the driving environment state action pair (S t , A t ) is the reward value (i.e., the cumulative reward reward value), R' is the reward value of the next driving environment state action pair, γ is the discount factor, and r is the reward value of the next driving environment state action pair. t ) to perform driving action A t The reward function obtained after the driving environment state action pair (S t , A t ) already exists in the episodic memory repository (EM), the value function in EM is updated to the larger of R and the stored value; otherwise, the new sample pair (S t , A t ) and R are written into EM, R is (S t , A t ) in the driving environment state ((S t ) corresponds to the value function.

[0071] The above mainly introduces the solution provided by the present application from the perspective of the method flow. It is understandable that in order to achieve the above functions, the device may include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0072] In the case of an integrated unit, Figure 8A possible exemplary block diagram of a driving decision-making device involved in an embodiment of the present application is shown. The driving decision-making device 800 may exist in the form of software. The driving decision-making device 800 may include: a construction unit 801, a determination unit 802, and an update unit 803.

[0073] Specifically, in one embodiment, the construction unit 801 is used to construct a Monte Carlo tree based on the current driving environment state, wherein the Monte Carlo tree includes N nodes, each node represents a driving environment state, and the N nodes include a root node and N-1 non-root nodes, wherein the root node represents the current driving environment state, the driving environment state represented by the first node is obtained by predicting the driving environment state represented by the parent node of the first node through a driving environment random model, and a first driving action, wherein the first driving action is a driving action determined by the parent node of the first node in the process of expanding the first node, the first node is any one of the N-1 non-root nodes, and N is a positive integer greater than or equal to 2;

[0074] A determination unit 802 is used to determine a node sequence starting from the root node and ending at a leaf node in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree; and determine a driving action sequence according to the driving actions corresponding to each node included in the node sequence, wherein the driving action sequence is used for driving decision-making;

[0075] The number of visits to each node is determined based on the number of visits to the node's child nodes and the initial number of visits to the node, and the value function of each node is determined based on the value function of the node's child nodes and the initial value function of the node, wherein the initial number of visits to each node is 1 and the initial value function is determined based on the value function that matches the driving environment state represented by the node.

[0076] In a possible design, when the construction unit 801 predicts the driving environment state represented by the first node based on the driving environment state represented by the parent node of the first node and the first driving action through the driving environment random model, it is specifically used to: predict the probability distribution of the driving environment state after executing the first driving action based on the driving environment state represented by the parent node of the first node through the driving environment random model using Dropout forward propagation; and obtain the driving environment state represented by the first node by sampling from the probability distribution.

[0077] In one possible design, when the construction unit 801 determines the initial value function of the node based on the value function that matches the driving environment state represented by the node, it is specifically used to: select a first number of target driving environment states that have the highest matching degree with the driving environment state represented by the node from the scenario memory storage repository; and determine the initial value function of the node based on the value functions corresponding to the first number of target driving environment states.

[0078] In a possible design, the updating unit 803 is used to obtain the actual driving environment state after executing the first driving action in the driving action sequence; and update the driving environment random model according to the current driving environment state, the first driving action and the actual driving environment state after executing the first driving action.

[0079] In a possible design, when the determination module determines a node sequence starting from the root node and ending at the leaf node in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree, the determination module is specifically used to: determine a node sequence starting from the root node and ending at the leaf node in the Monte Carlo tree according to the principle of maximum number of visits based on the number of visits of each node in the Monte Carlo tree; or, determine a node sequence starting from the root node and ending at the leaf node in the Monte Carlo tree according to the principle of maximum value function based on the value function of each node in the Monte Carlo tree; or, determine a node sequence starting from the root node and ending at the leaf node in the Monte Carlo tree according to the principle of maximum number of visits first and maximum value function second based on the number of visits and the value function of each node in the Monte Carlo tree.

[0080] In a possible design, the updating unit 803 is further used to determine, after the driving scenario ends, based on the driving results, a cumulative reward return value corresponding to the real driving environment state after each driving action in the driving scenario is executed; and to update the scenario memory storage repository using the cumulative reward return value corresponding to the real driving environment state after each driving action is executed as its corresponding value function.

[0081] Based on the above driving decision method, the embodiment of the present application also provides a driving decision device, such as Fig. 9As shown, the driving decision device 900 includes a memory 901 and a processor 902, and the memory 901 and the processor 902 are interconnected. Optionally, the memory 901 and the processor 902 can be interconnected through a bus; the bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0082] When implementing the driving decision-making method, the driving decision-making device 900:

[0083] The memory stores computer programs or instructions;

[0084] The processor is used to call the computer program or instruction stored in the memory to execute the following method: construct a Monte Carlo tree based on the current driving environment state, wherein the Monte Carlo tree includes N nodes, each node represents a driving environment state, and the N nodes include a root node and N-1 non-root nodes, wherein the root node represents the current driving environment state, the driving environment state represented by the first node is predicted by a driving environment random model based on the driving environment state represented by the parent node of the first node, and a first driving action, wherein the first driving action is a driving action determined by the parent node of the first node in the process of expanding the first node, the first node is any one of the N-1 non-root nodes, and N is a positive integer greater than or equal to 2; according to the number of visits and / or value function of each node in the Monte Carlo tree, determine a node sequence starting from the root node and ending at a leaf node in the Monte Carlo tree; according to the driving actions corresponding to each node included in the node sequence, determine a driving action sequence, and the driving action sequence is used for driving decision-making;

[0085] The number of visits to each node is determined based on the number of visits to the node's child nodes and the initial number of visits to the node, and the value function of each node is determined based on the value function of the node's child nodes and the initial value function of the node, wherein the initial number of visits to each node is 1 and the initial value function is determined based on the value function that matches the driving environment state represented by the node.

[0086] In a possible design, the driving environment state represented by the first node is predicted based on the driving environment state represented by the parent node of the first node and the first driving action through the driving environment random model, including: using Dropout forward propagation through the driving environment random model to predict the probability distribution of the driving environment state after executing the first driving action based on the driving environment state represented by the parent node of the first node; and sampling the driving environment state represented by the first node from the probability distribution.

[0087] In one possible design, the initial value function of the node is determined based on the value function that matches the driving environment state represented by the node, including: selecting a first number of target driving environment states that have the highest matching degree with the driving environment state represented by the node from an episode memory repository; and determining the initial value function of the node based on the value functions corresponding to the first number of target driving environment states.

[0088] In a possible design, the method also includes: after executing the first driving action in the driving action sequence, obtaining the actual driving environment state after executing the first driving action; and updating the driving environment random model according to the current driving environment state, the first driving action, and the actual driving environment state after executing the first driving action.

[0089] In a possible design, determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree includes: determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the principle of maximum number of visits based on the number of visits of each node in the Monte Carlo tree; or, determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the principle of maximum value function based on the value function of each node in the Monte Carlo tree; or, determining a node sequence from the root node to the leaf nodes in the Monte Carlo tree according to the principle of maximum number of visits first and maximum value function second based on the number of visits and the value function of each node in the Monte Carlo tree.

[0090] In a possible design, the method further includes: after the driving scenario ends, determining, based on the driving results, a cumulative reward return value corresponding to the real driving environment state after each driving action in the driving scenario; and using the cumulative reward return value corresponding to the real driving environment state after each driving action is executed as its corresponding value function to update the scenario memory storage repository.

[0091] As another form of this embodiment, a computer-readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed, the driving decision-making method in the above method embodiment can be executed.

[0092] As another form of this embodiment, a computer program product including instructions is provided, which, when executed, can execute the driving decision-making method in the above method embodiment.

[0093] As another form of this embodiment, a chip is provided, which can implement the driving decision-making method in the above method embodiment.

[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0099] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A driving decision method, characterized in that: include: Constructing a Monte Carlo tree based on a current driving environment state, wherein the Monte Carlo tree includes N nodes, each node represents a driving environment state, and the N nodes include a root node and N-1 non-root nodes, wherein the root node represents the current driving environment state, the driving environment state represented by the first node is predicted by a driving environment random model based on the driving environment state represented by the parent node of the first node, and a first driving action, wherein the first driving action is a driving action determined by the parent node of the first node in the process of expanding the first node, the first node is any one of the N-1 non-root nodes, and N is a positive integer greater than or equal to 2; Determine a node sequence starting from the root node and ending at a leaf node in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree; Determining a driving action sequence according to driving actions corresponding to each node included in the node sequence, wherein the driving action sequence is used for driving decision-making; The value function of each node is determined according to the value function of the child nodes of the node and the initial value function of the node.

2. The method according to claim 1, characterized in that The method predicts the driving environment state represented by the first node based on the driving environment state represented by the parent node of the first node and the first driving action by using the driving environment stochastic model, including: Predicting, by using the driving environment stochastic model and using Dropout forward propagation, a probability distribution of the driving environment state after executing the first driving action based on the driving environment state represented by the parent node of the first node; The driving environment state represented by the first node is obtained by sampling the probability distribution.

3. The method according to claim 1, characterized in that The method further comprises: Selecting, from the episodic memory repository, a first number of target driving environment states having the highest matching degree with the driving environment state represented by the node; The initial value function of the node is determined according to the value functions respectively corresponding to the first number of target driving environment states.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: After executing a first driving action in the driving action sequence, obtaining a real driving environment state after executing the first driving action; The driving environment stochastic model is updated according to the current driving environment state, the first driving action, and the actual driving environment state after executing the first driving action.

5. The method according to claim 1, characterized in that Determining a node sequence starting from the root node and ending at a leaf node in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree comprises: According to the number of visits to each node in the Monte Carlo tree, a node sequence starting from the root node and ending at the leaf node is determined in the Monte Carlo tree according to the principle of maximum number of visits; or, According to the value function of each node in the Monte Carlo tree and the principle of maximum value function, a node sequence starting from the root node and ending at the leaf node is determined in the Monte Carlo tree; or, According to the number of visits and the value function of each node in the Monte Carlo tree, a node sequence starting from the root node and ending at the leaf node is determined in the Monte Carlo tree according to the principle of giving priority to the largest number of visits and secondarily giving priority to the largest value function.

6. The method according to claim 3, characterized in that The method further comprises: After the driving scenario is finished, the accumulated reward return value corresponding to the real driving environment state after each driving action in the driving scenario is determined according to the driving result; The accumulated reward return value corresponding to the real driving environment state after each driving action is executed is used as its corresponding value function to update the episode memory storage library.

7. A driving decision-making device, characterized in that: include: A construction unit, configured to construct a Monte Carlo tree based on a current driving environment state, wherein the Monte Carlo tree comprises N nodes, each node representing a driving environment state, the N nodes comprising a root node and N-1 non-root nodes, wherein the root node represents the current driving environment state, the driving environment state represented by a first node is predicted by a driving environment random model based on the driving environment state represented by a parent node of the first node, and a first driving action, wherein the first driving action is a driving action determined by the parent node of the first node in a process of expanding the first node, the first node is any one of the N-1 non-root nodes, and N is a positive integer greater than or equal to 2; a determining unit, configured to determine a node sequence starting from the root node and ending at a leaf node in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree; and to determine a driving action sequence according to the driving actions corresponding to each node included in the node sequence, wherein the driving action sequence is used for driving decision-making; The value function of each node is determined according to the value function of the child nodes of the node and the initial value function of the node.

8. The device according to claim 7, characterized in that When the construction unit predicts the driving environment state represented by the first node based on the driving environment state represented by the parent node of the first node and the first driving action through the driving environment stochastic model, it is specifically used to: The driving environment stochastic model is used to adopt Dropout forward propagation to predict the probability distribution of the driving environment state after executing the first driving action based on the driving environment state represented by the parent node of the first node; and the driving environment state represented by the first node is obtained by sampling from the probability distribution.

9. The device according to claim 7, characterized in that The building blocks are specifically used for: A first number of target driving environment states having the highest matching degree with the driving environment state represented by the node are selected from the scenario memory storage repository; and an initial value function of the node is determined according to the value functions respectively corresponding to the first number of target driving environment states.

10. The device according to any one of claims 7 to 9, characterized in that The device also includes: An updating unit is used to obtain a real driving environment state after executing the first driving action in the driving action sequence; and update the driving environment random model according to the current driving environment state, the first driving action and the real driving environment state after executing the first driving action.

11. The device according to claim 7, characterized in that When the determining unit determines a node sequence starting from the root node and ending at a leaf node in the Monte Carlo tree according to the number of visits and / or the value function of each node in the Monte Carlo tree, the determining unit is specifically used to: According to the number of visits to each node in the Monte Carlo tree, and in accordance with the principle of maximum number of visits, determining a node sequence starting from the root node and ending at the leaf node in the Monte Carlo tree; Or, according to the value function of each node in the Monte Carlo tree and in accordance with the maximum value function principle, a node sequence starting from the root node and ending at the leaf node is determined in the Monte Carlo tree; Or, according to the number of visits and the value function of each node in the Monte Carlo tree, a node sequence starting from the root node and ending at the leaf node is determined in the Monte Carlo tree in accordance with the principle of maximum number of visits first and maximum value function second.

12. The device according to claim 9, characterized in that The device also includes: The updating unit is used to determine, after the driving scenario ends, the cumulative reward return value corresponding to the real driving environment state after each driving action in the driving scenario is executed according to the driving result; and update the scenario memory storage repository by taking the cumulative reward return value corresponding to the real driving environment state after each driving action is executed as its corresponding value function.

13. A driving decision-making device, characterized in that: include: Memory, storing computer programs or instructions; A processor, configured to call a computer program or instruction stored in the memory to execute the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed by the driving decision-making device, the method as described in any one of claims 1-6 is implemented.

Images