Driving decision-making method, driving decision-making device and chip

Abstract

This application discloses a driving decision-making method, a driving decision-making device, and a chip for improving the robustness of driving decision-making policies and ensuring optimal decision-making results are output. The method includes: constructing a Monte Carlo tree based on a current driving environment state, the Monte Carlo tree including a root node and [N-1] non-root nodes, each node representing a driving environment state, the root node representing the current driving environment state, and a driving environment state represented by any non-root node being predicted based on the driving environment state represented by the parent node of the non-root node and, by extension, based on driving behaviors determined by the parent node of the non-root node in the process of obtaining non-root nodes; determining a node sequence in the Monte Carlo tree starting from the root node and ending with a leaf node based on the access count and / or value function of each node in the Monte Carlo tree; and determining a driving behavior sequence based on the driving behaviors corresponding to each node in the node sequence.

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202010584738.2, entitled "DRIVING DECISION-MAKING METHOD AND APPARATUS," filed with the State Intellectual Property Office of the People's Republic of China on June 23, 2020, which is incorporated herein by reference in its entirety.

[0002] The embodiments of the present application relate to the field of autonomous driving, and in particular to a driving decision-making method and a driving decision-making device and chip. [Background technology]

[0003] With the beginning of the autonomous driving era, smart vehicles with autonomous driving capabilities have become the focus of research by various manufacturers. Currently, autonomous driving includes both driver-assisted and fully autonomous driving. The key technologies that realize autonomous driving include environment perception, driving decision-making, and executive control. Driving decision-making provides driving actions that the vehicle should perform based on sensory information about traffic participants.

[0004] Currently, driving decision-making is usually implemented based on a reinforcement learning (deep Q network, DQN) algorithm. The DQN model built using the DQN algorithm learns the driving environment state (e.g., the speed of the current vehicle and the speed of nearby vehicles) of a large number of vehicles at a certain time and the driving behavior (e.g., changing to the left lane or changing to the right lane) that the vehicle driver will take based on the driving environment state at that time. In driving decision-making, the driving behavior that the autonomous vehicle needs to perform can be obtained using the DQN model and based on the current driving environment state of the vehicle.

[0005] However, in driving decision-making based on the DQN model, the impact of a driving action on the vehicle's next driving is not taken into account, which makes it difficult to guarantee optimal decision-making results. Summary of the Invention

[0006] The embodiments of the present application provide a driving decision-making method and a driving decision-making apparatus and chip to improve the robustness of driving decision-making policy and ensure the optimal decision-making result is output.

[0007] According to a first aspect, an embodiment of the present application provides a driving decision-making method, the method including the steps of: constructing a Monte Carlo tree based on a current driving environment state, the Monte Carlo tree including N nodes, each node representing a driving environment state, the N nodes including a root node and [N-1] non-root nodes, the root node representing a current driving environment state, a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving behavior using a probabilistic model of the driving environment, the first driving behavior being a driving behavior determined by the parent node of the first node in the process of obtaining the first node by extension (in other words, the first driving behavior is predicted based on the ...). The method includes constructing a Monte Carlo tree, the Monte Carlo tree including a first node and a parent node of the first node serving as a driving behavior from which the first node is obtained by extension (wherein the first node is any one of [N-1] non-root nodes, where N is a positive integer greater than or equal to 2); determining a node sequence starting from the root node and ending with a leaf node in the Monte Carlo tree based on an access count and / or a value function of each node in the Monte Carlo tree; and determining a driving behavior sequence based on a driving behavior corresponding to each node included in the node sequence, the driving behavior sequence being used for driving decision-making. The access count of each node is determined based on the access counts of the subnodes of the node and an initial access count of the node, and the value function of each node is determined based on the value function of the subnodes of the node and an initial value function of the node, where the initial access count of each node is 1, and the initial value function is determined based on a value function that matches the driving environment state represented by the node.

[0008] By using the above-mentioned method, the probabilistic model of the driving environment may be used to predict various driving environment states that the multiple driving behaviors of the autonomous vehicle may encounter in a future period to construct a Monte Carlo tree, and the most favorable driving behavior sequence for the vehicle operation is determined based on the various driving environment states that the multiple driving behaviors of the autonomous vehicle may encounter in a future period, which helps to improve the robustness of the driving decision-making policy and ensures that optimal decision-making results are output.

[0009] In a feasible design example, predicting a driving environment state represented by a first node based on a driving environment state represented by a parent node of the first node and based on a first driving action using a probabilistic model of the driving environment includes predicting a probability distribution of the driving environment state after the first driving action is executed based on the driving environment state represented by the parent node of the first node by dropout-based forward propagation using the probabilistic model of the driving environment, and obtaining the driving environment state represented by the first node by sampling from the probability distribution.

[0010] In the above design example, the probability distribution of the driving environment state after the first driving action is executed based on the driving environment state represented by the parent node of the first node is predicted by a dropout-based forward propagation using a probabilistic model of the driving environment, and the driving environment state represented by the first node is obtained by sampling from the probability distribution. In this way, the uncertainty of the driving environment state is fully taken into account when expanding the node in the Monte Carlo tree. This increases the diversity of node expansion and makes the driving decision-making policy more robust.

[0011] In a feasible design example, determining an initial value function for a node based on a value function that matches a driving environment state represented by the node includes selecting from the episodic memory a first quantity of target driving environment states that most closely match the driving environment state represented by the node, and determining an initial value function for the node based on a value function corresponding to each first quantity of the target driving environment state.

[0012] In the above design example, episodic memory is introduced into the Monte Carlo tree. In this way, the initial value function of a node can be accurately and quickly estimated based on past empirical data, instead of being estimated by an inefficient expansion method. This reduces the computational overhead brought by the estimation of the initial value function of a node, and helps to improve the efficiency of the Monte Carlo tree search.

[0013] In a feasible design example, the method further includes, after a first driving action in the driving action sequence is executed, acquiring an actual driving environment state after the first driving action is executed, and updating a probabilistic model of the driving environment based on the current driving environment state, the first driving action, and the actual driving environment state after the first driving action is executed.

[0014] In the above design example, the probabilistic model of the driving environment can be continuously trained and updated, which helps to improve the accuracy of the probabilistic model of the driving environment.

[0015] In a possible design example, determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the access counts and / or value function of each node in the Monte Carlo tree includes determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the access counts of each node in the Monte Carlo tree according to a maximum access count rule, or determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the value function of each node in the Monte Carlo tree according to a maximum value function rule, or determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the access counts and value function of each node in the Monte Carlo tree according to a "maximum access count first, then maximum value function" rule.

[0016] In the above design example, multiple methods are provided to determine the node sequence, which helps to meet different driving decision-making requirements.

[0017] In a possible design example, the method further includes, when the driving episode ends, determining a cumulative reward return value corresponding to the actual driving environment state after each driving action in the driving episode is performed, and updating the episodic memory by using the cumulative reward return value corresponding to the actual driving environment state after each driving action is performed as a value function corresponding to the actual driving environment state.

[0018] In the above design example, the episodic memory is updated based on the cumulative reward return value, which leads to accurately determining the initial value function of the node in the Monte Carlo tree, thereby ensuring the reliability of the output decision-making results.

[0019] According to a second aspect, an embodiment of the present application provides an operational decision-making device including a unit configured to perform each step in the first aspect or any one of the possible design examples of the first aspect.

[0020] According to a third aspect, an embodiment of the present application provides a driving decision-making device including a processor and a memory, the memory being configured to store a computer program or instruction, the processor being configured to call the computer program or instruction stored in the memory to perform the method of the first aspect or any one of the possible design examples of the first aspect.

[0021] According to a fourth aspect, an embodiment of the present application provides a computer readable storage medium having a computer program or instructions for performing the method of the first aspect or any one of the possible design examples of the first aspect.

[0022] According to a fifth aspect, an embodiment of the present application provides a computer program product including a computer program or instructions that, when executed, can implement the method of the first aspect or any one of the possible design examples of the first aspect.

[0023] According to a sixth aspect, an embodiment of the present application provides a chip, the chip being configured to implement the method of the first aspect or any one of the possible design examples of the first aspect.

[0024] For technical effects that can be achieved in the second to sixth aspects, please refer to the technical effects that can be achieved in the first aspect, and details will not be described again here. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram of an automated driving system according to an embodiment of the present specification.

[0026] [Diagram 2] FIG. 2 is a schematic diagram of a tree-like structure according to an embodiment of the present disclosure;

[0027] [Diagram 3] FIG. 1 is a schematic diagram of a dropout processing effect according to an embodiment of the present disclosure.

[0028] [Figure 4] FIG. 2 is a schematic diagram of a driving decision-making process according to an embodiment of the present disclosure.

[0029] [Diagram 5] FIG. 1 is a schematic diagram of an environment in which an autonomous vehicle is located, according to an embodiment of the present disclosure.

[0030] [Figure 6] FIG. 2 is a first schematic diagram of a Monte Carlo tree structure according to an embodiment of the present specification.

[0031] [Figure 7] FIG. 2 is a second schematic diagram of a Monte Carlo tree structure according to an embodiment of the present disclosure.

[0032] [Figure 8] 1 is a first schematic diagram of a driving decision-making device according to an embodiment of the present specification.

[0033] [Figure 9] FIG. 2 is a second schematic diagram of a driving decision-making device according to an embodiment of the present specification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] 1 is a schematic diagram of an autonomous driving system according to an embodiment of the present specification. The autonomous driving system includes an environment perception module, a behavior decision-making module based on Monte Carlo tree search (MCTS), a path planning module, and an execution control module. The efficient MCTS-based behavior decision-making module is the core module of driving decision-making in this embodiment of the present application.

[0035] The environment perception module is useful for perceiving the state of the surrounding environment and obtaining the current driving environment state of the current vehicle. Specifically, the environment perception module considers the information sent from each sensor to determine the state of the current vehicle, including the position, speed, and course angle of the current vehicle, and determines the state of the surrounding environment, including information such as the positions of static objects (such as lane boundaries, road boundaries, and green areas), the positions and speeds of dynamic objects (such as general vehicles and pedestrians), road conditions, and weather conditions (such as temperature, humidity, brightness, etc.), and sends this information to the path planning module and the efficient MCTS-based behavior decision-making module.

[0036] An efficient MCTS-based behavioral decision-making module outputs a multi-stage future driving behavior sequence based on the current driving environment state of the current vehicle.

[0037] The route planning module plans a route trajectory having position and speed information and used to complete the driving behavior sequence based on the current vehicle and general vehicle conditions and the determined driving behavior sequence.

[0038] The executive control module outputs reasonable control values ​​by controlling the steering wheel, accelerator, and brake pedal of the current vehicle, so that the driving trajectory of the current vehicle can follow the planned route trajectory.

[0039] The driving decision-making solution provided in the embodiment of the present application may be applied to an in-vehicle device and an autonomous vehicle equipped with the autonomous driving system shown in Fig. 1. In the following description of the embodiment of the present application, the autonomous vehicle equipped with the autonomous driving system shown in Fig. 1 is used as an example for description.

[0040] Before describing the embodiments of the present application, some terms used in the present application will be explained and some characteristics will be described in order to facilitate understanding by those skilled in the art.

[0041] (1) A tree-like structure (tree) is a data structure and a hierarchical group that includes n nodes, where n is usually a positive integer equal to or greater than 1. It is called a "tree" because it looks like an upside-down tree, that is, the root of the tree faces upwards and the leaves of the tree face downwards. A node in a tree that has no subnodes is a leaf node of the tree, and the node at the top of the tree is the root node. For example, in the tree-like structure shown in FIG. 2, node 11 is located at the first layer of the tree, node 21, node 22, and node 23 are located at the second layer of the tree, and node 31, node 32, node 33, and node 34 are located at the third layer of the tree. Node 11 is the root node of the tree, and node 31, node 32, node 33, node 34, and node 23 are leaf nodes of the tree. The Monte Carlo tree in the embodiment of the present application is also a tree-like structure.

[0042] (2) Monte Carlo Tree Search (MCTS) is a heuristic search algorithm used in several decision-making processes, usually in games, and mainly serves the purpose of selecting the move with the highest success rate in a given game state. The main concept of Monte Carlo Tree Search is a search, which is a series of downward traversal processes along a Monte Carlo tree (also known as a game tree). A traversal path extends from the root node (i.e., the current game state) to a node that is not fully expanded. A node that is not fully expanded is a node that has at least one unvisited (or unexpanded) subnode. When a node that is not fully expanded is encountered, the unvisited subnode of the node is obtained by expansion. The corresponding initial value function of the subnode obtained by expansion is calculated based on a certain policy. The initial value function is also known as the initial value or the simulation result. Taking a board game as an example, based on the configuration represented by the subnode obtained by the expansion, a game may be played based on a rollout policy until the game ends, and a simulation result (e.g., loss or win) may be obtained, which is the initial value function of the subnode (e.g., 0 / 1 if the game is lost, or 1 / 1 if the game is won). After obtaining the initial value function of the subnode, the initial value function is backpropagated back to the root node of the current Monte Carlo tree, and the initial value function (simulation result) of the subnode is added to all ancestor nodes of the subnode. For example, the initial value function of the subnode is 0 / 1 (which represents the game is lost), and 0 / 1 is added to the value functions of all ancestor nodes of this node. When the search time or maximum search steps of the Monte Carlo tree (i.e., the maximum number of nodes obtained by the expansion starting from the root node) is reached, the search stops and a decision is made based on the value function of each node in the Monte Carlo tree. Note that in a Monte Carlo tree, each node represents a state (a situation, e.g., sometimes called a configuration in a board game), and statistical data is recorded for each node (e.g., the node's value function or access count).The value function recorded at a node may also be referred to as the value of the node, which may be a specific numerical value.

[0043] (3) Dropout means excluding some neurons in the model with a probability P. The output of the excluded neurons is set to 0. Figure 3 is a schematic diagram of the model that has undergone the dropout process. Figure 3(A) is a schematic diagram of the model before the dropout process, and Figure 3(B) is a schematic diagram of the model that has undergone the dropout process. We can see that after the dropout process, some neurons in the model are temporarily excluded and output 0.

[0044] Furthermore, it should be noted that in the embodiments of the present application, "at least one" may be described as one or more, and "multiple" may be two, three, four, or even more. This is not limited in the embodiments of the present application. In the embodiments of the present application, " / " may represent an "or" relationship between related objects. For example, A / B may represent A or B. "And / or" may be used for three relationships that may exist between related objects. For example, "A and / or B" may represent three cases: only A exists, both A and B exist, and only B exists. A and B may be singular or plural. In the embodiments of the present application, the term "by way of example" or "for example" is used to represent an example, an illustration, or an explanation. Any embodiment or design solution described as "by way of example" or "for example" should not be construed as preferred or advantageous over another embodiment or design solution. The use of the term "by way of example" or "for example" is intended to present the related concept in a concrete manner to facilitate understanding.

[0045] In the embodiment of the present application, the probabilistic model of the driving environment may be used to predict various driving environment conditions that the driving behaviors of the autonomous vehicle may face in a future period to construct a Monte Carlo tree, and the most favorable driving behavior sequence for the vehicle driving is determined based on the various driving environment conditions that the driving behaviors of the autonomous vehicle may face in a future period. This helps to improve the robustness of the driving decision-making policy and ensures that optimal decision-making results are output. In addition, the episodic memory may be further used to estimate the initial value function of the nodes in the Monte Carlo tree. This reduces the computational overhead caused by the estimation of the initial value function of the nodes and improves the efficiency of the Monte Carlo tree search.

[0046] In the following, an embodiment of the present application will be described with reference to the accompanying drawings, taking a lane change decision-making scenario as an example.

[0047] 4 is a schematic diagram of a driving decision-making process according to one embodiment of the present disclosure. The process includes the following steps:

[0048] S401: An autonomous vehicle builds a Monte Carlo tree based on the current driving environment state.

[0049] The Monte Carlo tree includes N nodes, each node representing a driving environment state, the N nodes include a root node and [N-1] non-root nodes, the root node represents a current driving environment state, a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving action using a probabilistic model of the driving environment, the first driving action being a driving action determined by the parent node of the first node in the process of obtaining the first node by expansion, the first node being any node among the [N-1] non-root nodes, and N is a positive integer greater than or equal to 2.

[0050] In a lane change decision-making scenario, the environmental state of the autonomous vehicle may include the speed of the autonomous vehicle, the relative speed and distance of a general vehicle in front of the autonomous vehicle (in the direction of travel of the autonomous vehicle) to the autonomous vehicle, and the arrival time (time required to catch up with the autonomous vehicle) of a general vehicle behind the autonomous vehicle (in the direction opposite to the direction of travel of the autonomous vehicle). The driving environment state of the autonomous vehicle may include environmental states for T past time points including the current time point, where T is a positive integer greater than or equal to 1. For example, the current time point is 10:00:00. The current driving environment state of the autonomous vehicle may include environmental states for a total of eight past time points of the autonomous vehicle: 9:56:30, 9:57:00, 9:57:30, 9:58:00, 9:58:30, 9:59:00, 9:59:30, and 10:00:00.

[0051] The environmental state of the autonomous vehicle may be acquired using an on-board sensor of the autonomous vehicle. For example, the on-board sensor of the autonomous vehicle may include a vehicle speed sensor, an acceleration sensor, and a distance sensor (e.g., a radar distance sensor). The vehicle speed sensor may measure the speed of the autonomous vehicle, the acceleration sensor may measure the acceleration of the autonomous vehicle, and the distance sensor may measure the relative distance between the autonomous vehicle and a general vehicle. The autonomous vehicle may determine the relative speed of the general vehicle with respect to the current vehicle based on the relative distance variation between the current vehicle and the general vehicle, and may determine the arrival time of a general vehicle behind the current vehicle based on the relative distance of the general vehicle with respect to the current vehicle and based on the relative speed of the general vehicle with respect to the current vehicle.

[0052] In a possible implementation, the autonomous vehicle may further communicate with the general vehicle to obtain the relative speed or distance of the general vehicle to the current vehicle. For example, the autonomous vehicle may receive speed and position information sent from the general vehicle, and determine the relative speed or distance of the general vehicle to the current vehicle based on the speed and position information of the current vehicle.

[0053] For example, an autonomous vehicle is located in the environment shown in FIG. 5 at a certain time point. The autonomous vehicle can obtain the relative distance (ΔD) and relative speed (ΔV) of general vehicles in the three lanes in front of the current vehicle with respect to the current vehicle, and can obtain the time to arrival (TTA) of general vehicles in the three lanes behind the current vehicle and the speed of the current vehicle. In FIG. 5, L, M, and R represent general vehicles in the left lane, center lane, and right lane of the autonomous vehicle, respectively. In this embodiment of the present application, the current driving environment state may be used as the environmental state at T past time points corresponding to the current time point. When making driving decision, the autonomous vehicle may construct a Monte Carlo tree by using the current driving environment state of the autonomous vehicle as the root node of the Monte Carlo tree. Assume that the autonomous vehicle is currently located in the environment shown in FIG. 5. The current driving environment state (S t ) is S t =(ΔD i,t-T:t , ΔV i,t-T:t , V t-T:t , T.T.A. j,t-T:t ), where i ∈ (L, M, R), j ∈ (L, R), t represents the current time, and tT:t represents the T past time points that correspond to the current time.

[0054] In a lane change decision-making scenario, the selectable driving actions of the autonomous vehicle include one or more of changing to the left lane, staying in the current lane, and changing to the right lane. Training of a probabilistic model of the driving environment may be performed by collecting in advance driving environment states of a large number of vehicles before and after performing driving actions. Specifically, the previously collected driving environment states of a large number of vehicles before and after performing driving actions and the performed driving actions may be used as sample pairs to construct a training set for training the probabilistic model of the driving environment. Each sample pair in the training set is denoted by (S t +A t ,S t+1 ), and A t represents a driving action performed by the vehicle, which may be one of changing to the left lane, staying in the current lane, or changing to the right lane, and S t A trepresents the driving environment state before executing S t+1 A t This represents the driving environment state after the execution of the above.

[0055] In the embodiment of the present application, it should be understood that the driving environment state of the vehicle before executing the driving behavior is usually the driving environment state of the vehicle when the vehicle starts to execute the driving behavior, and the driving environment state of the vehicle after executing the driving behavior is usually the driving environment state of the vehicle when the vehicle completes the execution of the driving behavior. For example, a vehicle executes a driving behavior of changing to the left lane from 10:10:00 to 10:10:10. The driving environment state of the vehicle at 10:10:00 may be used as the driving environment state of the vehicle before executing the driving behavior of changing to the left lane, and the driving environment state of the vehicle at 10:10:10 may be used as the driving environment state of the vehicle after executing the driving behavior of changing to the left lane.

[0056] Stochastic model of the driving environment

number

[0057] Based on the trained probabilistic model of the driving environment and the current driving environment state, the autonomous vehicle can predict various driving environment states that the autonomous vehicle's multiple driving behaviors may encounter in the future to build a Monte Carlo tree. Specifically, the autonomous vehicle can start selecting a node by navigating through each layer starting from the root node of the Monte Carlo tree. If the selected target node has one or more selectable driving behaviors that are not predicted, one target driving behavior is selected. The driving environment state of the autonomous vehicle after performing the target driving behavior based on the target node is predicted using the probabilistic model of the driving environment and is used as the driving environment state represented by the subnode of the target node to extend the Monte Carlo tree.

[0058] In a possible implementation example, in order to fully take into account the uncertainty of other general vehicles in the driving environment, when the driving environment state of the autonomous vehicle after performing a target driving behavior based on the target node is predicted using a probabilistic model of the driving environment, a probability distribution of the driving environment state of the autonomous vehicle after performing a target driving behavior based on the target node is obtained by dropout-based forward propagation using the probabilistic model of the driving environment, and the driving environment state is obtained by sampling from the probability distribution and used as the driving environment state of the autonomous vehicle after performing the target driving behavior based on the target node.

[0059] For example, FIG. 6 illustrates an augmented Monte Carlo tree for an autonomous vehicle. The autonomous vehicle selects a node (e.g., selects by layer-by-layer traversal) by navigating through each layer starting from the root node 11 of the Monte Carlo tree to node 21. The unpredicted selectable driving actions (A) of node 21 include changing to the left lane, staying in the current lane, or changing to the right lane. The autonomous vehicle may randomly select a target driving action from changing to the left lane, staying in the current lane, or changing to the right lane. For example, a target driving action randomly selected from changing to the left lane, staying in the current lane, or changing to the right lane may be changing to the left lane (A t After selecting the target driving behavior, the autonomous vehicle predicts the target driving behavior (A) based on the node 21 by multiple dropout-based forward propagation based on a probabilistic model of the driving environment. t ) is performed, and a plurality of possible driving environment states are predicted based on the plurality of possible driving environment states. t The mean (μ) and variance (δ 2 ) is calculated. A t Probability distribution of the value of the driving environment state after

number

[0060] After the subnode of the target node is acquired by expansion, the autonomous vehicle initializes the statistical data of the subnode (i.e., access count (N) and value function (Q)). In other words, the initial access count and the initial value function of the subnode also need to be determined. The autonomous vehicle sets the initial access count of the subnode to 1, and updates the access count of each node on the node path (search path) corresponding to the subnode by tracing back from the subnode to the root node. For example, the access count of each node on the node path corresponding to the subnode acquired by expansion is increased by 1 (i.e., N=N′+1). N is the access count after the update, and N′ is the access count before the update. In other words, the access count of each node in the Monte Carlo tree is the sum of the access counts of all subnodes of the node and the initial access count of the node. It should be noted that in this application, the node path (search path) corresponding to a certain node is a node path including all ancestor nodes of the node. For example, after an autonomous vehicle acquires a subnode of node 21 (node ​​31), it updates the initial access count of node 31 to 1, and increments the access counts of both node 21 and node 11 on the node path (search path) of node 31 by 1, thereby completing the update of the access counts of each node on the node path corresponding to node 31.

[0061] For the initial value function of the subnode obtained by the expansion, the autonomous vehicle may determine the initial value function of the subnode based on the episodic memory (EM). If there is a record included in the episodic memory and corresponding to the driving environment state represented by the subnode, the value function stored in the episodic memory and corresponding to the driving environment state is directly output and used as the initial value function of the subnode. Otherwise, a first quantity of target driving environment states that are most consistent with the driving environment state represented by the subnode may be selected from the episodic memory, and the initial value function of the subnode is determined based on the value function respectively corresponding to the first quantity of each target driving environment state. The first quantity (K) may be 3 or 5, etc. For example, the autonomous vehicle may select K target driving environment states that are most consistent with the driving environment state represented by the subnode obtained by the expansion from the episodic memory, and use the average value of the value functions respectively corresponding to the K target driving environment states as the initial value function of the subnode.

[0062] After the initial value function of the subnode obtained by the expansion is determined, the autonomous vehicle updates the value function of each node on the node path corresponding to the subnode, starting from the subnode and tracing back to the root node. For example, the autonomous vehicle updates the value function of each node on the node path corresponding to the subnode as Q=Q′+Q L Q is the updated value function of the node, Q′ is the value function of the node before the update, and Q L is the initial value function of the subnode. In other words, the value function of each node in the Monte Carlo tree is the sum of the value functions of all its subnodes and the initial value function of the node. In another possible implementation, the value function of each node on the node path corresponding to the subnode may alternatively be expressed as Q=Q′+(Q L −Q′) / N, where Q is the updated value function of the node, Q′ is the value function of the node before the update, and Q L is the initial value function of the subnode, and N is the updated access count of the node.

[0063] In this embodiment of the present application, the maximum number of steps of the Monte Carlo tree may be limited. In other words, the maximum number of nodes obtained by the expansion starting from the root node of the Monte Carlo tree is limited. For example, the maximum number of steps of the Monte Carlo tree is limited to 20. When the maximum number of steps is reached, the expansion of the Monte Carlo tree is stopped. Furthermore, in this embodiment of the present application, every time after a leaf node of the Monte Carlo tree is obtained by expansion (i.e., a subnode of a node in the Monte Carlo tree is obtained by expansion), if the number of steps of the Monte Carlo tree does not satisfy the maximum number of steps of the Monte Carlo tree, the step of selecting a node by navigating through each layer starting from the root node of the Monte Carlo tree is repeated to continue expanding the leaf node of the Monte Carlo tree. In the selection method of selecting a node by navigating through each layer starting from the root node of the Monte Carlo tree, the node (st+1) corresponding to the driving action may be selected by navigating each layer based on the selection policy of an upper confidence bound apply to tree (UCT) algorithm applied to the tree. Compared with other selectable driving actions, the selected driving action (A t ) is a value function (Q) corresponding to the driving behavior and a weighted search term

number

[0064] S402: The autonomous vehicle determines a sequence of nodes in a Monte Carlo tree starting from a root node and ending with a leaf node based on an access count and / or a value function of each node in the Monte Carlo tree.

[0065] In this embodiment of the present application, in a Monte Carlo tree, a node sequence starting from a root node and ending with a leaf node may be determined according to one of the following rules: a maximum access count rule, a maximum value function rule, or a "first maximum access count, then maximum value function" rule.

[0066] For example, a node sequence starting from a root node and ending with a leaf node may be determined according to the maximum access count rule in a Monte Carlo tree. As shown in Figure 7, starting from node 11 (the root node), the subnode with the maximum access count of node 11 is node 21, the subnode with the maximum access count of node 21 is node 31, and the subnode with the maximum access count of node 31 is node 41. The determined node sequence is node 11-node 21-node 31-node 41.

[0067] S403: Determine a driving action sequence based on the driving actions corresponding to each node included in the node sequence. The driving action sequence is used for driving decision-making.

[0068] In this embodiment of the present application, a driving action corresponding to a node in a Monte Carlo tree is a driving action in which the parent node of the node acquires the node by expansion. The root node does not have a corresponding driving action. For example, as shown in FIG. 7, a driving action in which the parent node 11 of node 22 acquires node 22 by expansion is maintain current lane. Therefore, the driving action corresponding to node 22 is maintain current lane. Using the determined node sequence node 11-node 21-node 31-node 41 as an example, the driving action sequence is change to left lane-change to left lane-change to left lane-maintain current lane.

[0069] In order to ensure the reliability of the probabilistic model of the driving environment, in a feasible implementation example, when the autonomous vehicle executes the first driving action of the driving action sequence, [the driving environment state of the autonomous vehicle before executing the first driving action + the first driving action] and [the driving environment state after executing the first driving action] are generated as a new sample pair (S t +A t ,S t+1 ) and can be added to the training set to update a probabilistic model of the driving environment.

[0070] For example, the probabilistic model of the driving environment may be updated in the following manner:

number

number

number

number

number

number

number

number

[0071] After a driving episode is completed, for example, after the autonomous vehicle arrives at a destination or after the autonomous vehicle encounters an accident and stops driving, the autonomous vehicle may obtain a return value based on whether the destination has been reached, etc., and may determine an episode trajectory sequence (in time series) based on the actual driving environment state after each driving action is performed in the driving episode. The autonomous vehicle may determine each driving environment state (S t ) corresponding driving environment state and action pair (S t ,A t ) to calculate the discounted cumulative return. t is S t is the driving behavior executed by the autonomous vehicle based on t ,A t ) return value (i.e., cumulative reward return value), R′ is the return value of the next driving environment state and action pair, γ is the discount coefficient, and r is the driving environment state (S t ) Driving behavior A t is executed. t ,A tIf the sample pair (S) already exists in the episodic memory (EM), update the value function in the EM to the larger of R or the stored value. Otherwise, update the new sample pair (S t ,A t ) and R are written directly into EM. R is written into (S t ,A t ) in the operating environment (S t ) is the value function corresponding to

[0072] The above mainly describes the solution provided in the present application in terms of a method process. It can be understood that, to realize the above functions, the apparatus may include corresponding hardware structures and / or software modules that realize each function. Those skilled in the art should easily recognize that the units, algorithms, and steps in each example described with reference to the embodiments disclosed herein can be realized in the form of hardware or a combination of hardware and computer software in the present application. Whether a function is performed in hardware or hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the described functions for each specific application, but such realization should not be considered as going beyond the scope of the present application.

[0073] In the case of using an integrated unit, FIG. 8 is a block diagram of a possible example of the driving decision-making device according to an embodiment of the present application. The driving decision-making device 800 may exist in the form of software. The driving decision-making device 800 may include a construction unit 801, a determination unit 802, and an update unit 803.

[0074] Specifically, in one embodiment, the construction unit 801 is configured to construct a Monte Carlo tree based on a current driving environment state, the Monte Carlo tree includes N nodes, each node representing a driving environment state, the N nodes include a root node and [N-1] non-root nodes, the root node represents the current driving environment state, a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving action using a probabilistic model of the driving environment, the first driving action is a driving action determined by the parent node of the first node in the process of obtaining the first node by expansion, the first node is any node among the [N-1] non-root nodes, and N is a positive integer greater than or equal to 2.

[0075] The determination unit 802 is configured to determine a node sequence starting from a root node and ending with a leaf node in the Monte Carlo tree based on the access count and / or value function of each node in the Monte Carlo tree, and determine a driving action sequence based on driving actions corresponding to each node included in the node sequence, where the driving action sequence is used for driving decision-making.

[0076] The access count of each node is determined based on the access counts of the node's subnodes and the initial access count of the node, the value function of each node is determined based on the value function of the node's subnodes and the initial value function of the node, the initial access count of each node is 1, and the initial value function is determined based on the value function that matches the driving environment state represented by the node.

[0077] In a feasible design example, when a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving action using a probabilistic model of the driving environment, the construction unit 801 is specifically configured to use the probabilistic model of the driving environment to predict, by dropout-based forward propagation, a probability distribution of the driving environment state after the first driving action is executed based on the driving environment state represented by the parent node of the first node, and obtain the driving environment state represented by the first node by sampling from the probability distribution.

[0078] In a feasible design example, when the initial value function of a node is determined based on a value function that matches the driving environment state represented by the node, the construction unit 801 is specifically configured to select a first quantity of a target driving environment state that has the highest degree of match with the driving environment state represented by the node from the episodic memory, and determine the initial value function of the node based on the value functions corresponding to the first quantities of each target driving environment state.

[0079] In a feasible design example, the update unit 803 is configured to obtain, after a first driving action in the driving action sequence is executed, an actual driving environment state after the first driving action is executed, and update the probabilistic model of the driving environment based on the current driving environment state, the first driving action, and the actual driving environment state after the first driving action is executed.

[0080] In a possible design example, when determining a node sequence starting from a root node and ending with a leaf node in a Monte Carlo tree based on the access counts and / or value function of each node in the Monte Carlo tree, the determination module is specifically configured to: determine a node sequence starting from a root node and ending with a leaf node in a Monte Carlo tree based on the access counts of each node in the Monte Carlo tree according to a maximum access count rule; or determine a node sequence starting from a root node and ending with a leaf node in a Monte Carlo tree based on the value function of each node in the Monte Carlo tree according to a maximum value function rule; or determine a node sequence starting from a root node and ending with a leaf node in a Monte Carlo tree based on the access counts and value function of each node in the Monte Carlo tree according to a "maximum access count first, then maximum value function" rule.

[0081] In a possible design example, the update unit 803 is further configured to determine, when the driving episode ends, a cumulative reward return value corresponding to the actual driving environment state after each driving action is performed in the driving episode, and update the episodic memory by using the cumulative reward return value corresponding to the actual driving environment state after each driving action is performed as a value function corresponding to the actual driving environment state.

[0082] Based on the above driving decision-making method, an embodiment of the present application further provides a driving decision-making device. As shown in Fig. 9, the driving decision-making device 900 includes a memory 901 and a processor 902. The memory 901 and the processor 902 are connected to each other. Optionally, the memory 901 and the processor 902 can be connected to each other using a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be classified into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 uses only one thick line to represent a bus, but this does not mean that there is only one bus or only one type of bus.

[0083] When the driving decision-making apparatus 900 implements a driving decision-making method, the memory stores a computer program or instruction, and the processor is configured to call the computer program or instruction stored in the memory to perform the following method, which includes the steps of: constructing a Monte Carlo tree based on a current driving environment state, the Monte Carlo tree includes N nodes, each node representing a driving environment state, the N nodes include a root node and [N-1] non-root nodes, the root node represents the current driving environment state, and a driving environment state represented by a first node is determined based on a driving environment state represented by a parent node of the first node and based on a first driving action using a probabilistic model of the driving environment. the first driving action is determined by a parent node of the first node in a process of obtaining the first node by expansion, the first node being any one of [N-1] non-root nodes, N being a positive integer greater than or equal to 2; determining a node sequence starting from a root node and ending with a leaf node in a Monte Carlo tree based on an access count and / or a value function of each node in the Monte Carlo tree; and determining a driving action sequence based on a driving action corresponding to each node included in the node sequence, the driving action sequence being used for driving decision-making.

[0084] The access count of each node is determined based on the access counts of the node's subnodes and the initial access count of the node, the value function of each node is determined based on the value function of the node's subnodes and the initial value function of the node, the initial access count of each node is 1, and the initial value function is determined based on the value function that matches the driving environment state represented by the node.

[0085] In a feasible design example, predicting a driving environment state represented by a first node based on a driving environment state represented by a parent node of the first node and based on a first driving action using a probabilistic model of the driving environment includes predicting a probability distribution of the driving environment state after the first driving action is executed based on the driving environment state represented by the parent node of the first node by dropout-based forward propagation using the probabilistic model of the driving environment, and obtaining the driving environment state represented by the first node by sampling from the probability distribution.

[0086] In a feasible design example, determining an initial value function for a node based on a value function that matches a driving environment state represented by the node includes selecting from the episodic memory a first quantity of target driving environment states that most closely match the driving environment state represented by the node, and determining an initial value function for the node based on a value function corresponding to each first quantity of the target driving environment state.

[0087] In a feasible design example, the method further includes, after a first driving action in the driving action sequence is executed, acquiring an actual driving environment state after the first driving action is executed, and updating a probabilistic model of the driving environment based on the current driving environment state, the first driving action, and the actual driving environment state after the first driving action is executed.

[0088] In a possible design example, determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the access counts and / or value function of each node in the Monte Carlo tree includes determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the access counts of each node in the Monte Carlo tree according to a maximum access count rule, or determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the value function of each node in the Monte Carlo tree according to a maximum value function rule, or determining a node sequence in the Monte Carlo tree starting from a root node and ending with a leaf node based on the access counts and value function of each node in the Monte Carlo tree according to a "maximum access count first, then maximum value function" rule.

[0089] In a possible design example, the method further includes, when the driving episode ends, determining a cumulative reward return value corresponding to the actual driving environment state after each driving action in the driving episode is performed, and updating the episodic memory by using the cumulative reward return value corresponding to the actual driving environment state after each driving action is performed as a value function corresponding to the actual driving environment state.

[0090] In another aspect of this embodiment, a computer readable storage medium is provided having stored thereon a program or instructions that, when executed, may perform the driving decision making method in the method embodiments described above.

[0091] In another aspect of the present embodiment, there is provided a computer program product including instructions that, when executed, may effect driving decision making in the method embodiments described above.

[0092] As another aspect of this embodiment, a chip is provided, which can implement the driving decision-making method in the above-mentioned method embodiment.

[0093] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a hardware-only embodiment, a software-only embodiment, or an embodiment that combines software and hardware. Furthermore, the present application may take the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0094] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that computer program instructions may be used to realize each process and / or each block in the flowcharts and / or block diagrams, and combinations of processes and / or blocks in the flowcharts and / or block diagrams. The computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or another programmable data processing device to produce a machine, whereby the instructions executed by the processor of the computer or another programmable data processing device produce an apparatus for implementing a particular function in one or more steps of the flowcharts and / or one or more blocks of the block diagrams.

[0095] Computer program instructions, capable of directing a computer or another programmable data processing device to operate in a particular manner, may alternatively be stored in a computer readable memory, such that the instructions stored in the computer readable memory produce an artifact that includes an instruction apparatus that implements a particular function in one or more steps of the flowcharts and / or one or more blocks of the block diagrams.

[0096] The computer program instructions may alternatively be loaded into a computer or other programmable data processing device such that a sequence of operations and steps are performed on the computer or other programmable device to produce a computer-implemented process. Thus, the instructions that execute on the computer or other programmable device provide steps for implementing a particular function in one or more steps of the flowcharts and / or one or more blocks of the block diagrams.

[0097] Although some preferred embodiments of the present application have been described, those skilled in the art can make changes and modifications to these embodiments once they understand the basic inventive concept. Therefore, it is intended that the following claims be interpreted as including the preferred embodiments, as well as all changes and modifications that fall within the scope of the present application.

[0098] It is apparent that a person skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. In this case, as long as these modifications and variations fall within the scope of protection defined by the following claims and their equivalent technologies, the present application is intended to cover these modifications and variations.

Claims

1. constructing a Monte Carlo tree based on a current driving environment state, the Monte Carlo tree including N nodes, each node representing a driving environment state, the N nodes including a root node and [N-1] non-root nodes, the root node representing the current driving environment state, a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving behavior using a probabilistic model of a driving environment, the first driving behavior being a driving behavior determined by the parent node of the first node in a process of obtaining the first node by extension, the first node being any node among the [N-1] non-root nodes, and N being a positive integer greater than or equal to 2; determining a sequence of nodes in the Monte Carlo tree starting from the root node and ending with a leaf node based on a value function for each node in the Monte Carlo tree or based on an access count for each node in the Monte Carlo tree and the value function; determining a driving action sequence based on a driving action corresponding to each node included in the node sequence, the driving action sequence being used for driving decision making; Equipped with A driving decision-making method, wherein the access count of each node is determined based on the access counts of the node's subnodes and an initial access count of the node, the value function of each node is determined based on the value function of the node's subnodes and an initial value function of the node, the initial access count of each node is 1, and the initial value function is determined based on a value function that matches the driving environment state represented by the node.

2. a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving behavior using a probabilistic model of the driving environment; predicting, by a dropout-based forward propagation using the probabilistic model of a driving environment, a probability distribution of a driving environment state after the first driving action is executed based on the driving environment state represented by the parent node of the first node; obtaining the driving environment state represented by the first node by sampling from the probability distribution; The driving decision-making method of claim 1 , comprising:

3. The initial value function of a node is determined based on a value function that matches the driving environment state represented by the node. selecting from an episodic memory a first quantity of target driving environment states that most closely match the driving environment state represented by the node; determining the initial value function for the node based on a value function corresponding to the first quantity for each target driving environment state; The driving decision-making method of claim 1 , comprising:

4. The driving decision-making method further comprises: After the first driving action in the driving action sequence is executed, acquiring an actual driving environment state after the first driving action is executed; updating the probabilistic model of a driving environment based on the current driving environment state, the first driving behavior, and the actual driving environment state after the first driving behavior is performed. The driving decision-making method according to claim 1 , comprising:

5. determining a sequence of nodes in the Monte Carlo tree starting from the root node and ending with a leaf node based on a value function for each node in the Monte Carlo tree or based on an access count and the value function for each node in the Monte Carlo tree; determining a sequence of nodes in the Monte Carlo tree starting from the root node and ending with a leaf node according to a maximum value function rule based on the value function for each node in the Monte Carlo tree; or determining a sequence of nodes in the Monte Carlo tree starting from the root node and ending with a leaf node according to a "maximum access count first, then maximum value function" rule based on the access counts and the value function of each node in the Monte Carlo tree; The driving decision-making method of claim 1 , comprising:

6. The driving decision-making method further comprises: determining, when a driving episode is completed, a cumulative reward return value corresponding to an actual driving environment state after each driving action in the driving episode is performed; updating the episodic memory by using the cumulative reward return value corresponding to the actual driving environment state after each driving action is performed as a value function corresponding to the actual driving environment state; The driving decision-making method of claim 3 comprising:

7. a construction unit configured to construct a Monte Carlo tree based on a current driving environment state, the Monte Carlo tree including N nodes, each node representing a driving environment state, the N nodes including a root node and [N-1] non-root nodes, the root node representing the current driving environment state, a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving behavior using a probabilistic model of a driving environment, the first driving behavior being a driving behavior determined by the parent node of the first node in a process of obtaining the first node by extension, the first node being any node among the [N-1] non-root nodes, and N being a positive integer greater than or equal to 2; a decision unit configured to determine a node sequence starting from the root node and ending with a leaf node in the Monte Carlo tree based on a value function of each node in the Monte Carlo tree or based on an access count of each node in the Monte Carlo tree and the value function, and to determine a driving action sequence based on a driving action corresponding to each node included in the node sequence, the driving action sequence being used for driving decision-making; Equipped with A driving decision-making device, wherein the access count of each node is determined based on the access counts of the node's subnodes and an initial access count of the node, the value function of each node is determined based on the value function of the node's subnodes and an initial value function of the node, the initial access count of each node is 1, and the initial value function is determined based on a value function that matches the driving environment state represented by the node.

8. When a driving environment state represented by a first node is predicted based on a driving environment state represented by a parent node of the first node and based on a first driving behavior using a probabilistic model of a driving environment, the construction unit specifically:

8. The driving decision-making device of claim 7, configured to use the probabilistic model of the driving environment to predict a probability distribution of a driving environment state after the first driving action is executed based on the driving environment state represented by the parent node of the first node by dropout-based forward propagation, and to obtain the driving environment state represented by the first node by sampling from the probability distribution.

9. When the initial value function of a node is determined based on a value function that matches a driving environment state represented by the node, the construction unit specifically:

8. The driving decision-making device of claim 7, configured to select from an episodic memory a first quantity of a target driving environment state that most closely matches the driving environment state represented by the node, and determine the initial value function of the node based on a value function corresponding to each of the first quantities of each target driving environment state.

10. The driving decision-making device further comprises:

10. The driving decision-making device according to claim 7, further comprising an update unit configured to acquire an actual driving environment state after the first driving action in the driving action sequence is executed, and update the probabilistic model of the driving environment based on the current driving environment state, the first driving action, and the actual driving environment state after the first driving action is executed.

11. In the case where the Monte Carlo tree determines a node sequence starting from the root node and ending with a leaf node based on a value function of each node in the Monte Carlo tree, or based on an access count of each node in the Monte Carlo tree and the value function, the determining unit specifically:

8. The driving decision-making device of claim 7, configured to determine a node sequence in the Monte Carlo tree starting from the root node and ending with a leaf node according to a maximum value function rule based on the value function of each node in the Monte Carlo tree, or to determine a node sequence in the Monte Carlo tree starting from the root node and ending with a leaf node according to a "maximum access count first, then maximum value function" rule based on the access counts and the value function of each node in the Monte Carlo tree.

12. The driving decision-making device further comprises:

10. The driving decision-making device of claim 9, further comprising: an updating unit configured to determine, when a driving episode ends, a cumulative reward return value corresponding to an actual driving environment state after each driving action in the driving episode is performed, and update the episode memory by using the cumulative reward return value corresponding to the actual driving environment state after each driving action is performed as a value function corresponding to the actual driving environment state.

13. a memory configured to store computer programs or instructions; A processor configured to call the computer program or instructions stored in the memory to perform the driving decision-making method according to any one of claims 1 to 6. A driving decision-making device comprising:

14. A computer-readable storage medium, the computer-readable storage medium storing a computer program or instructions, the computer program or instructions being executed by a driving decision-making device to realize the driving decision-making method according to any one of claims 1 to 6.

15. A computer program product for causing a computer to carry out the driving decision-making method according to any one of claims 1 to 6.

16. A chip, the operation of which implements a driving decision-making method according to any one of claims 1 to 6.

Images