Interactive decision method, apparatus and system for autonomous valet driving, and medium

Abstract

Disclosed in some embodiments of the present application are an interactive decision method, apparatus and system for autonomous valet driving, and a medium. In the method, a full sampling tree is constructed on the basis of a state transition model and an observation model, an inverse dynamic programming method is used to perform solving starting from confident leaf nodes of the full sampling tree to a confident root node, and finally an optimal interaction strategy is obtained. Using the optimal interaction strategy to instruct the operation of an ego vehicle can improve the safety and reliability of autonomous valet driving. The method comprises: using a partially observable Markov decision process to model an interactive decision problem, in order to obtain a modeling result; on the basis of the modeling result, constructing a full sampling tree; and using an inverse dynamic programming method to perform solving starting from confident leaf nodes of the full sampling tree to a confident root node, in order to determine an optimal interaction strategy, wherein the optimal interaction strategy comprises an acceleration action, a holding action, a deceleration action or a parking action.

Description

Autonomous guest driving car interaction decision method, device, system and medium

[0001] The present application claims priority to the Chinese patent application No. 202410724757.9, filed on June 5, 2024, and entitled "Autonomous guest driving car interaction decision method, device, system and medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of autonomous guest driving car, and in particular to an autonomous guest driving car interaction decision method, device, system and medium. BACKGROUND

[0003] In the scene where the obstacles around the parking lot intersection constitute the self-vehicle perception visual field blind area, how to ensure the safe operation of the self-vehicle and other vehicles when the self-vehicle is in autonomous guest driving car, that is, how the self-vehicle obtains a safe and reliable autonomous guest driving car interaction strategy, is a major challenge faced by the autonomous guest driving car decision planning field.

[0004] In view of this major challenge faced by the autonomous guest driving car decision planning field, some researches use deep neural networks to infer the optimal interaction strategy for autonomous guest driving car facing the blind area of the parking lot intersection, but the learning-based method has poor scene migration performance. Therefore, how to improve the safety and reliability of the autonomous guest driving car interaction strategy has become a technical problem to be solved by the technical personnel in the field.

[0005] SUMMARY

[0006] The present application provides an autonomous guest driving car interaction decision method, device, system and medium. In the method, a full-sample tree is constructed based on a state transition model and an observation model, and a reverse dynamic programming method is used to solve from a confident leaf node of the full-sample tree to a confident root node, and finally the best interaction strategy is obtained. The best interaction strategy is used to indicate the operation of the self-vehicle, which can improve the safety and reliability of the autonomous guest driving car.

[0007] In a first aspect, an embodiment of the present application provides an autonomous guest driving car interaction decision method, comprising:

[0008] modeling an interaction decision problem by using a partially observable Markov decision process to obtain a modeling result;

[0009] constructing a full-sample tree according to the modeling result; wherein the full-sample tree comprises a plurality of paths, and each path comprises a confident root node, a confident internal node and a confident leaf node arranged from top to bottom.

[0010] Solving from the confident leaf node of the full-sampling tree to a confident root node using a backward dynamic programming method to determine an optimal interaction strategy, the optimal interaction strategy including an acceleration action, a keep action, a deceleration action, or a stop action.

[0011] In the present application, a full-sampling tree is constructed based on a state transition model and an observation model, a backward dynamic programming method is used to solve from a confident leaf node of the full-sampling tree to a confident root node, and finally an optimal interaction strategy is obtained. Using the optimal interaction strategy to instruct the ego vehicle to operate can improve the safety and reliability of autonomous guest driving.

[0012] In some embodiments, the modeling result includes a state space, an observation space, an action space, a state transition model, and an observation model; and the step of constructing a full-sampling tree according to the modeling result includes:

[0013] Sampling the state space generated when the ego vehicle actually observes the other vehicle as the confident root node of the full-sampling tree multiple times to obtain multiple paths; wherein the confident leaf node is the deepest confident internal node, and the confident internal node is generated according to the state space, the observation space, the action space, the state transition model, and the observation model.

[0014] In the present application, the state space generated when the ego vehicle actually observes the other vehicle is sampled multiple times as the confident root node of the full-sampling tree to obtain a path, and then an optimal interaction strategy is obtained. Using the optimal interaction strategy to instruct the ego vehicle to operate can improve the safety and reliability of autonomous guest driving.

[0015] In some embodiments, sampling the state space generated when the ego vehicle actually observes the other vehicle as the confident root node of the full-sampling tree once to obtain one path includes:

[0016] Taking the confident root node as a target node, generating a next state space using the state transition model according to the target node and the action space;

[0017] Determining a target observation space using the observation model according to the next state space and the observation space;

[0018] According to the next state space and the target observation space, determining the confident internal node, taking the confident internal node as a target node, and repeating the step of generating a next state space using the state transition model according to the target node and the action space.

[0019] In the present application, the confidence root node is used as the target node, and the state transition model is used together with the action space to generate the next state space. According to the next state space and the observation space, the target observation space is obtained by using the observation model, and finally the confidence internal node is obtained. The step of generating the next state space is repeated to obtain a path in the full sampling number, which helps to obtain the best interaction strategy. Using the best interaction strategy to indicate the operation of the ego vehicle can improve the safety and reliability of autonomous driving for passengers.

[0020] In some embodiments, the modeling result further includes a reward function.

[0021] In some embodiments, the step of solving the best interaction strategy by using the inverse dynamic programming method from the confidence leaf node of the full sampling tree includes:

[0022] Pruning the full sampling tree to obtain a full sampling tree strategy tree, wherein the confidence internal node in the full sampling tree strategy tree only contains one action branch;

[0023] If the depth of the full sampling tree strategy tree is not greater than a preset value, determining the optimal regular weighted discount value of the confidence leaf node based on the full sampling tree and using the reward function;

[0024] Determining the optimal regular weighted discount value of the confidence internal node according to the optimal regular weighted discount value of the confidence leaf node;

[0025] According to the optimal regular weighted discount value of the confidence internal node, the optimal regular weighted discount value of the confidence root node is calculated;

[0026] According to the optimal regular weighted discount value of the confidence root node, the optimal regular weighted discount value of the confidence internal node, and the optimal regular weighted discount value of the confidence leaf node, the best interaction strategy is determined.

[0027] In the present application, the full sampling number is pruned to obtain a full sampling number strategy tree, and the depth thereof is compared with a preset value to finally determine the optimal regular weighted discount value of the confidence internal node, and then the best interaction strategy is obtained. Using the best interaction strategy to indicate the operation of the ego vehicle can improve the safety and reliability of autonomous driving for passengers.

[0028] In some embodiments, the state space includes the longitudinal position and speed information of the ego vehicle, and the longitudinal position and speed information of the other vehicle.

[0029] In the present application, the multi-directional state space can be understood to obtain the best interaction strategy. Using the best interaction strategy to indicate the operation of the ego vehicle can improve the safety and reliability of autonomous driving for passengers.

[0030] In some embodiments, the action space includes driving actions that the ego vehicle can take and driving actions that the other vehicle can take; wherein the driving actions include accelerating actions, maintaining actions, decelerating actions, and stopping actions.

[0031] In a second aspect, the embodiments of the present application further provide an autonomous guest driving vehicle interaction decision device, comprising:

[0032] a modeling unit configured to model the interaction decision problem by using a partially observable Markov decision process, and obtain a modeling result;

[0033] a construction unit configured to construct a full sampling tree according to the modeling result; wherein the full sampling tree includes a plurality of paths, and the paths include confidence root nodes, confidence internal nodes, and confidence leaf nodes arranged from top to bottom;

[0034] a solving unit configured to solve from the confidence leaf nodes of the full sampling tree to the confidence root nodes by using a backward dynamic programming method to determine an optimal interaction strategy, wherein the optimal interaction strategy includes accelerating actions, maintaining actions, decelerating actions, or stopping actions.

[0035] The device in the present application constructs a full sampling tree based on a state transition model and an observation model, solves from the confidence leaf nodes of the full sampling tree to the confidence root nodes by using a backward dynamic programming method, and finally obtains an optimal interaction strategy. The optimal interaction strategy is used to indicate the operation of the ego vehicle, which can improve the safety and reliability of autonomous guest driving vehicles.

[0036] In a third aspect, the embodiments of the present application further provide an autonomous guest driving vehicle interaction decision system, comprising:

[0037] a processor; and

[0038] a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the steps of the autonomous guest driving vehicle interaction decision method.

[0039] The system in the present application constructs a full sampling tree based on a state transition model and an observation model, solves from the confidence leaf nodes of the full sampling tree to the confidence root nodes by using a backward dynamic programming method, and finally obtains an optimal interaction strategy. The optimal interaction strategy is used to indicate the operation of the ego vehicle, which can improve the safety and reliability of autonomous guest driving vehicles.

[0040] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the autonomous guest driving vehicle interaction decision method.

[0041] The readable storage medium in the application constructs a full sampling tree based on a state transition model and an observation model, starts from a confident leaf node of the full sampling tree, and finally obtains an optimal interaction strategy by using a reverse dynamic programming method to solve to a confident root node. The optimal interaction strategy is used to indicate the operation of the ego vehicle, which can improve the safety and reliability of autonomous driving for passengers. BRIEF DESCRIPTION OF DRAWINGS

[0042] FIG. 1 exemplarily shows a scene schematic diagram provided according to some embodiments;

[0043] FIG. 2 exemplarily shows a flowchart of an autonomous driving for passengers interaction decision-making method provided according to some embodiments;

[0044] FIG. 3 exemplarily shows a schematic diagram of a state transition model provided according to some embodiments;

[0045] FIG. 4 exemplarily shows a schematic diagram of an observation model provided according to some embodiments;

[0046] FIG. 5 exemplarily shows a schematic diagram of a standard tree provided according to some embodiments;

[0047] FIG. 6 exemplarily shows a schematic diagram of a full sampling tree provided according to some embodiments;

[0048] FIG. 7 exemplarily shows a path schematic diagram formed by combining any one scene sequence and all action sequences provided according to some embodiments;

[0049] FIG. 8 exemplarily shows a path schematic diagram formed by combining any one action sequence and all scene sequences provided according to some embodiments;

[0050] FIG. 9 exemplarily shows a structural schematic diagram of an autonomous driving for passengers interaction decision-making device provided according to some embodiments. DETAILED DESCRIPTION

[0051] In order to make the purpose and implementation of the present application more clear, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all the embodiments.

[0052] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the subsequently described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0053] The terms "first", "second", "third", and the like in the description and in the claims of the present application and the above figures are used for distinguishing between similar or identical objects or entities without necessarily implying a specific order or sequence thereof, unless otherwise specified. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

[0054] The terms "comprises", "comprising", "includes", "including", "has", "having", and their variants are intended to cover and encompass both the assigned and unassigned instances, for example, a product or device that comprises a list of components is not necessarily limited to the specified components, but can include other components not expressly listed or inherent to such product or device.

[0055] In the scene where the obstacles around the parking lot intersection constitute the self-vehicle perception blind area, how to ensure the safe operation of the self-vehicle and the other vehicle when the self-vehicle is in autonomous guest driving, that is, how the self-vehicle obtains a safe and reliable autonomous guest driving interaction strategy, is a major challenge in the field of autonomous guest driving decision planning.

[0056] FIG. 1 exemplarily shows a schematic diagram of a scene according to some embodiments, in which a plurality of vehicles are parked in the parking spaces of a parking lot. Due to the presence of the parked vehicles, the self-vehicle 11 has a perception blind area 12 when driving, which leads to the fact that the self-vehicle cannot directly observe the other vehicle 13 when the other vehicle enters the self-vehicle's perception blind area, and thus the self-vehicle and the other vehicle may be in danger when the self-vehicle is in autonomous guest driving.

[0057] In view of this major challenge in the field of autonomous guest driving decision planning, some researches use deep neural networks to infer the optimal interaction strategy for autonomous guest driving facing the blind area of the parking lot intersection, but the learning-based method has poor scene transferability. Therefore, how to improve the safety and reliability of the autonomous guest driving interaction strategy has become a technical problem to be solved by the technical personnel in the field.

[0058] To solve the above technical problem, the embodiments of the present application provide an autonomous guest driving interaction decision method, device, system and medium, which constructs a full-sampling tree based on a state transition model and an observation model, starts from a confident leaf node of the full-sampling tree, and uses a backward dynamic programming method to solve to a confident root node, finally obtains the best interaction strategy, and uses the best interaction strategy to indicate the operation of the self-vehicle, which can improve the safety and reliability of autonomous guest driving.

[0059] FIG. 2 exemplarily shows a flowchart of an autonomous guest driving interaction decision method according to some embodiments, which includes steps S100-S300.

[0060] It should be noted that the autonomous customer driving interaction decision method in the embodiments of the present application can be applied to the intersection blind area interaction scene of the parking lot, and can also be applied to the intersection blind area interaction scene of the urban navigation driving without light control, etc.

[0061] In some embodiments, the system architecture of the autonomous customer driving interaction strategy method includes two blocks of problem modeling and problem solving, wherein the problem modeling block includes step S100, and the problem solving block includes steps S200 and S300.

[0062] S100, the interaction decision problem is modeled by using a partially observable Markov decision process, and a modeling result is obtained.

[0063] In the embodiments of the present application, the modeling result includes a state space, an observation space, an action space, a state transition model, an observation model, and a reward function.

[0064] Specifically, the modeling process of the autonomous customer driving interaction decision problem facing the intersection blind area of the parking lot by using the general framework of the partially observable Markov decision process includes:

[0065] The state space is composed of the state quantity χ0=[s0 v0] based on the longitudinal position and speed information of the ego vehicle. T And the state quantity χ1=[s1 v1] based on the longitudinal position and speed information of the other vehicle. T The state space of the autonomous customer driving interaction decision problem facing the intersection blind area of the parking lot is constructed as: χ=[χ0, χ1] T (1)

[0066] In the embodiments of the present application, the autonomous customer driving interaction decision only considers that the ego vehicle should perform acceleration, speed maintenance, deceleration and parking actions in the longitudinal direction, so the state space is generated only according to the longitudinal position and speed information of the ego vehicle and the longitudinal position and speed information of the other vehicle.

[0067] In the embodiments of the present application, before constructing the full sampling tree, the first state space is generated according to the longitudinal position and speed information of the ego vehicle and the longitudinal position and speed information of the other vehicle obtained when the ego vehicle actually observes the other vehicle. The other vehicle can be actually observed by the ego vehicle before the other vehicle enters the non-perception blind area of the ego vehicle. The longitudinal position and speed information of the ego vehicle and the other vehicle are determined when the other vehicle is in the non-perception blind area of the ego vehicle, and the specific obtaining method can be obtained according to the real sensor installed on the ego vehicle.

[0068] Since there are multiple state spaces in the full sampling tree constructed subsequently, the remaining state spaces are determined according to the state transition model, which will be described in detail hereinafter.

[0069] Observation space: The state variable o0 = [s0 v0] is composed of the longitudinal position and velocity information of the vehicle detected by sensors. T The state quantity o1 = [s1 v1] consists of the longitudinal position and velocity information of the other vehicle detected by the virtual sensor. T The observation space for the autonomous valet driving interactive decision-making problem in blind spots at parking lot intersections is constructed as: z=[o0,o1] T (2)

[0070] In this embodiment, the observation space is predicted, and multiple observation spaces may be predicted at once. Each observation space has different parameters, and each observation space has a corresponding probability. In some embodiments, the probability corresponding to the observation space represents the probability of the occurrence of the longitudinal position and speed information of the vehicle itself, as well as the longitudinal position and speed information of other vehicles in the observation space.

[0071] Motion space: Based on the driving actions that the vehicle may take. And the driving actions that other cars might take. The action space for constructing the autonomous valet driving interactive decision-making problem in blind spots at parking lot intersections is: A = [a0, a1] T (3)

[0072] In the formula, and These represent the vehicle's acceleration, speed maintenance, deceleration, and stopping actions, respectively. and These respectively represent the acceleration, speed maintenance, deceleration, and stopping actions of another vehicle.

[0073] In this embodiment, the action space is randomly generated and used to simulate the driving actions that the vehicle and other vehicles may take.

[0074] State transition model: Based on the kinematic models of the self-vehicle and other vehicles, the state transition model for the autonomous valet driving interactive decision-making problem in the blind spot of parking lot intersection is constructed as follows: T(χ′|χ,a)=P(χ′|χ,a)=P(χ′0|χ0,a0)P(χ′1|χ1,a1) (4)

[0075] In the formula, P(χ′0|χ0,a0) and P(χ′1|χ1,a1) are the state transition models of the vehicle and the other vehicle, respectively, and can be expressed as:

[0076] For example, Figure 3 illustrates a schematic diagram of a state transition model provided according to some embodiments. The specific usage of the state transition model will be described later and will not be repeated here.

[0077] Observation model: Assuming that the output of the sensor of the ego vehicle longitudinal position and speed information and the output of the virtual sensor of the other vehicle longitudinal position and speed information are decoupled, the observation model of the autonomous guest driving interaction decision problem facing the parking lot intersection blind area can be expressed as: O(z|χ',a) = P(z|χ',a) = P(o0|χ'0)P(o1|χ'1) (7)

[0078] Assuming that the sensor can accurately detect the ego vehicle longitudinal position and speed information, the observation model of the ego vehicle can be expressed as:

[0079] Assuming that the other vehicle longitudinal position and speed information detected by the virtual sensor has noise, the observation model of the other vehicle can be expressed as: P(o1|χ'1) = N(χ'1,σ) (9)

[0080] In the formula, N(χ'1,σ) represents a Gaussian distribution.

[0081] For example, FIG. 4 exemplarily shows a schematic diagram of an observation model according to some embodiments. The specific use method of the observation model is described hereinafter and will not be described here again.

[0082] Reward function: The reward function considering safety, comfort and efficiency can be described as:

[0083] In the formula, 0≤λ≤1 is a weight coefficient.

[0084] S200, constructing a full-sample tree according to the modeling result; wherein the full-sample tree includes multiple paths, and the path includes a confidence root node, a confidence internal node and a confidence leaf node arranged from top to bottom.

[0085] Specifically, the full-sample tree is a sparse approximation of the standard tree, and contains all action sequences and sampled scene sequences. As can be seen from the comparison between the standard tree of FIG. 5 and the full-sample tree of FIG. 6, the full-sample tree performs a pruning operation on the standard tree.

[0086] In order to ensure the clarity and tidiness of FIG. 6, FIG. 6 only draws a full-sample tree including 4 scene sequences obtained by 4 samplings and all action sequences. Due to the limitation of the length, the full-sample tree in FIG. 6 is set to a certain depth. One scene sequence obtained by 1 sampling and all action sequences form a path. All paths of the full-sample tree in FIG. 6 include a confidence root node (b0), confidence internal nodes (b11, b12, b13 and b14) and confidence leaf nodes (b201, b204, b206, b208, b210, b211, b213, b214 and b215) arranged from top to bottom.

[0087] FIG. 7 illustrates a path diagram of any one scene sequence combined with all action sequences according to some embodiments. As shown in FIG. 7, the path of any one scene sequence combined with all action sequences obtained by sampling splits at different action branches of the full-sampling tree belief node. FIG. 8 illustrates a path diagram of any one action sequence combined with all scene sequences according to some embodiments. As shown in FIG. 8, the path of any one action sequence combined with all scene sequences obtained by sampling does not split, and the number of paths is equal to the number of all scene sequences obtained by sampling, K. Therefore, if the full-sampling tree depth is D, and the number of actions is |A|, the number of full-sampling tree belief leaf nodes is in the order of O(|A| D K).

[0088] In some embodiments, the modeling result includes a state space, an observation space, an action space, a state transition model, and an observation model; and the step of constructing the full-sampling tree according to the modeling result includes:

[0089] sampling the belief root node of the full-sampling tree with the state space generated by using real observations of the ego vehicle to the other vehicle to obtain a plurality of paths; wherein the belief leaf node is the deepest belief internal node, and the belief internal node is generated according to the state space, the observation space, the action space, the state transition model, and the observation model.

[0090] In the embodiments of the present application, the number of samplings is the same as the number of scene sequences.

[0091] In some embodiments, sampling the belief root node of the full-sampling tree with the state space generated by using real observations of the ego vehicle to the other vehicle to obtain a path includes:

[0092] using the state transition model to generate a next state space according to the target node and the action space.

[0093] In the embodiments of the present application, the action space is used to simulate possible driving actions of the ego vehicle and possible driving actions of the other vehicle. First, a next state space is generated based on the state space generated by using real observations of the ego vehicle to the other vehicle and the state transition model, that is, the longitudinal position and speed information of the ego vehicle and the other vehicle are predicted.

[0094] Referring to FIG. 6, the belief root node b0 in FIG. 6 extends different action branches (including a1 branch corresponding to the action space and a2 branch corresponding to the action space a2). It should be noted that a1 in FIG. 6 has nothing to do with a1 in the action space A. a1 in FIG. 6 contains a0 and a1 in the action space A, and a2 in FIG. 6 is the same.

[0095] According to the next state space and the observation space, a target observation space is determined by using the observation model.

[0096] In the embodiments of the present application, the observation model is used to screen observation spaces with a possibility meeting a requirement from the observation spaces as target observation spaces. A random number can be generated by the observation model, which is a probability. The possibility of the observation space is compared with the random number. If the possibility is greater than the random number, the observation space with the greater possibility is determined as the target observation space. It should be noted that the target observation space can be multiple. Referring to FIG. 6, z1 and z2 in FIG. 6 are both target observation spaces.

[0097] According to the next state space and the target observation space, a confidence internal node is determined. The confidence internal node is taken as a target node, and the step of generating the next state space according to the target node and the action space by using the state transition model is repeatedly performed. In this way, the next state space is continuously generated, and a full sampling tree with a certain depth can be formed.

[0098] In some embodiments, the step of generating the next state space according to the target node and the action space by using the state transition model can be automatically stopped from being repeatedly performed, or a preset stopping depth can be set, and the repeatedly performed step is stopped when the preset stopping depth is reached.

[0099] In the embodiments of the present application, the next state space and the target observation space are used to determine a confidence internal node corresponding to the next state space. The confidence internal node is a confidence degree determined by using the next state space and the target observation space. The confidence degree can represent a reliable degree of the occurrence of the next state space.

[0100] For example, the path generation process in FIG. 7 includes the following steps. First, the a1 branch side of the action branch is described. A corresponding next state space is generated by using the state transition model through the confidence root node b0 and the action space a1. The confidence internal node b11 corresponding to the next state space is determined by using the next state space and the target observation space z1. The confidence internal node b11 is the confidence degree of the next state space. Then, the confidence internal node b11 is taken as a target node. A corresponding next state space is generated by using the state transition model according to the target node b11 and the action space a1. The confidence internal node b201 corresponding to the next state space is determined by using the next state space and the target observation space z1. In addition, the confidence internal node b11 is taken as a target node. A corresponding next state space is generated by using the state transition model according to the target node b11 and the action space a2. The confidence internal node b204 corresponding to the next state space is determined by using the next state space and the target observation space z2.

[0101] For the action branch (a2 branch) side, through the belief root node b0 and the action space a2, a corresponding next state space is generated by using the state transition model, and the belief internal node b14 corresponding to the next state space is determined by using the next state space and the target observation space z2, the belief internal node b14 is the confidence of the next state space. Then the belief internal node b14 is taken as the target node, and according to the target node b14 and the action space a1, a corresponding next state space is generated by using the state transition model, and the belief internal node b214 corresponding to the next state space is determined by using the next state space and the target observation space z2. In addition, the belief internal node b14 is taken as the target node, and according to the target node b14 and the action space a2, a corresponding next state space is generated by using the state transition model, and the belief internal node b215 corresponding to the next state space is determined by using the next state space and the target observation space z1.

[0102] In some other embodiments, the confidence represented by the belief internal node can be determined according to the probability of reaching the belief internal node. In the embodiments of the present application, the probability of reaching the belief internal node of the full-sampling tree is approximately represented by the ratio of the number of particles of the belief node of the full-sampling tree to the number of particles of the belief root node of the full-sampling tree.

[0103] For example, the number of particles of the belief root node of the full-sampling tree in FIG. 6 can be the number of sampling times of the full-sampling tree. FIG. 6 only draws the paths corresponding to four scene sequences obtained by four sampling times, so the number of particles of the belief root node of the full-sampling tree in FIG. 6 is four. Only two of the four sampling times pass through the belief node b11 of the full-sampling tree, so the number of particles of the belief node b11 of the full-sampling tree can be considered to be two, and the number of particles of the belief node b12 of the full-sampling tree is also two, the number of particles of the belief node b13 of the full-sampling tree is one, and the number of particles of the belief node b14 of the full-sampling tree is three. At this time, the probability of reaching the belief node b11 of the full-sampling tree is 2 / 4=50%, the probability of reaching the belief node b12 of the full-sampling tree is 2 / 4=50%, the probability of reaching the belief node b13 of the full-sampling tree is 1 / 4=25%, and the probability of reaching the belief node b14 of the full-sampling tree is 3 / 4=75%.

[0104] The sum of probabilities of reaching all the belief child nodes on one action branch of the belief parent node of the full sampling tree is equal to the probability of reaching the belief parent node of the full sampling tree. Here, the belief parent node is the upper level node of the belief child node, and the example belief node b11 of the full sampling tree is the belief parent node of the belief leaf node b201, and the belief root node b0 is the belief parent node of the belief node b11 of the full sampling tree. The sum of probabilities of the belief child nodes b213 and b214 on one action branch (a1 branch) of the belief parent node b14 is equal to the probability of reaching the belief parent node b14. Therefore, in the process of calculating the value of the belief parent node of the full sampling tree, the probability of reaching the belief child node of the full sampling tree is used to represent the degree of influence of the value of the belief child node of the full sampling tree on the value of the belief parent node of the full sampling tree.

[0105] S300, the belief leaf node of the full sampling tree is used to solve the belief root node by using the backward dynamic programming method to determine the optimal interaction strategy, and the optimal interaction strategy includes the acceleration action, the maintenance action, the deceleration action or the parking action.

[0106] In the embodiments of the present application, the optimal interaction strategy is determined by the backward dynamic programming method, so that the ego vehicle drives according to the optimal interaction strategy.

[0107] In the embodiments of the present application, if the pruning operation is performed on the full sampling tree, the full sampling tree strategy tree is obtained after pruning, and the belief internal node of the full sampling tree strategy tree only contains one action branch. For example, as shown in FIG. 6, the full sampling tree strategy tree can include one action branch (a1 branch) of the belief root node b0 and one action branch (a1 branch) of the belief node b11.

[0108] For any strategy tree π of the full sampling tree, the true value V π (b0) of the belief root node b0 under the strategy tree π satisfies the following relationship with the estimated value

[0109] In the formula, |π| is the number of belief nodes of the strategy tree π, and the estimated value is calculated by using the return function.

[0110] The right side of the inequality (11) is maximized, and the approximate optimal strategy of the belief root node b0 in any strategy tree π of the full sampling tree is

[0111] Using π' to represent any strategy of the belief root node b0, the true value V of the belief root node b0 under the approximate optimal strategy tree π' is the true value V​π′ (b0) satisfies the following relationship with at least probability 1-τ

[0112] From inequality (13), it can be seen that the value V π′ (b0) of the confidence root node b0 under the approximate optimal policy tree can be approximated by any policy tree π' with the approximation error From the right side of inequality (13), it can be seen that the less the number of confidence nodes of any policy tree π', the smaller the approximation error, and the influence of the number of confidence nodes of any policy tree π' on the approximation error can be suppressed by increasing the number of scenario sequences of the full sampling tree. Further, taking the full sampling tree policy tree as any policy tree on the right side of inequality (13), and selecting the policy tree that maximizes the following optimization objective from the full sampling tree policy tree set Π D as the optimal policy of the confidence root node b0.

[0113] For the optimization objective described in equation (14), the regular weighted discounted value of the confidence node b under the full sampling tree policy tree π is defined as:

[0114] where |Φ b | is the number of full sampling tree scenario sequences passing through the confidence node b; γ is the discount factor; Δ(b) is the depth of the confidence node b in the full sampling tree policy tree π; π b is the subtree of the full sampling tree policy tree π whose root node is the confidence node b; |Φ b | / K represents the probability of reaching the confidence node b.

[0115] From equation (15), it can be seen that the regular weighted discounted value of the confidence root node b0 under the full sampling tree policy tree π can be expressed as

[0116] where s φ is the starting state of the scenario sequence φ; R(s φ ,π(b0)) is the immediate reward of the confidence root node b0 under the scenario sequence φ and the full sampling tree policy tree π; CH(b0,π(b0)) is the set of confidence child nodes generated by the confidence root node b0 under the scenario sequence φ and the full sampling tree policy tree π.

[0117] From equation (16), it can be seen that if the policy subtree π bWhen the number of included belief nodes is large and the number of scene sequences of the full-sampling tree of the belief sub-node is insufficient, the regular weighted discounted value of the belief sub-node is small, and the influence on the regular weighted discounted value of the belief root node is small. At the same time, it can be known from formula (16) that the regular weighted discounted value of the belief root node b0 under the action of the full-sampling tree strategy tree π can be represented by the belief sub-nodes. If the belief sub-nodes obtain the optimal regular weighted discounted value under the action of the full-sampling tree strategy tree π, the belief root node b0 also obtains the optimal regular weighted discounted value under the action of the full-sampling tree strategy tree π. Therefore, the optimal regular weighted discounted value of the belief root node b0 is obtained by screening the strategy space of the full-sampling tree under the belief root node b0, and then the optimal strategy of the belief root node b0 is obtained.

[0118] Based on the derivation and specific description of formula (11) to formula (16), in some embodiments, the step of obtaining the optimal interaction strategy by using the inverse dynamic programming method from the belief leaf node of the full-sampling tree comprises:

[0119] Pruning the full-sampling tree to obtain a plurality of full-sampling tree strategy trees, wherein the belief internal nodes in the full-sampling tree strategy trees only contain one action branch.

[0120] In the embodiments of the present application, if the depth of the full-sampling tree strategy tree π is greater than R max / λ(1-γ)+1, there is:

[0121] It can be known from the Bellman equation that the greater the depth of the belief sub-node is, the smaller the influence on is, and is bounded. Substituting into inequality (17), it can be obtained that: π (b0)≤-λ (18)

[0122] It can be known from inequalities (17) and (18) that the factor λ|π| increases with the increase of the depth of the full-sampling tree strategy tree π, and when the depth of the full-sampling tree strategy tree π is greater than R max / λ(1-γ)+1, it plays a leading role, so that the regular weighted discounted value of the belief root node b0 is less than the regular weighted discounted value υ π0 (b0)≥0. Therefore, the full-sampling tree strategy tree π cannot contain the belief sub-node with a depth greater than R max / λ(1-γ)+1.

[0123] Therefore, in the embodiments of the present application, if the depth of the full sampling tree strategy tree is greater than the preset value, the full sampling tree strategy tree is pruned, and the depth after pruning is not greater than the preset value.

[0124] If the depth of all the full sampling tree strategy trees is not greater than the preset value, the optimal regular weighted discounted value of the confidence leaf node is determined based on the full sampling tree and using the return function.

[0125] Specifically, according to the deduction of formula (11) to formula (16) above, the dynamic programming method shown in FIG. 6 can be used to recursively solve the optimal strategy of the confidence root node b0 from the confidence leaf node to the confidence root node b0.

[0126] wherein the optimal regular weighted discounted value of the confidence leaf node can be expressed as

[0127] In the formula, π0 is the full sampling tree, wherein is generated according to the return function.

[0128] According to the optimal regular weighted discounted value of the confidence leaf node, the optimal regular weighted discounted value of the confidence internal node is determined.

[0129] In some embodiments, the optimal regular weighted discounted value of the confidence internal node can be expressed as:

[0130] In the formula, τ(b, a, z) represents the confidence child node of the confidence internal node b under the action branch a and the observation branch z, wherein represents the driving action guiding the operation of the ego vehicle in the interaction strategy, and b represents the confidence of the confidence node.

[0131] It should be noted that the parameters not specifically described in the embodiments of the present application are all calibration parameters.

[0132] According to the optimal regular weighted discounted value of the confidence internal node, the optimal regular weighted discounted value of the confidence root node is calculated.

[0133] In some embodiments, according to the optimal regular weighted discounted value of the confidence internal node, the optimal regular weighted discounted value of the confidence root node is also calculated using formula (20), and at this time, the confidence internal node is taken as the confidence leaf node and substituted into the formula, and at this time, the optimal regular weighted discounted value of the confidence root node is calculated.

[0134] Based on the full sampling tree π0, the confidence leaf node b 201 , b 204 , b 206 , b208 b 210 、b 211 、b 213 、b 214 and b 215 the optimal regular weighted discounted value of the belief root node b0 is calculated again using formula (20). 11 12 13 14

[0135] The optimal interaction strategy is determined according to the optimal regular weighted discounted value of the belief root node, the optimal regular weighted discounted value of the belief internal node, and the optimal regular weighted discounted value of the belief leaf node.

[0136] In the embodiment of the application, as shown in FIG. 6, the maximum optimal regular weighted discounted value is selected from the optimal regular discounted values calculated from the two action branches (a1 branch and a2 branch) of the belief root node b0. For example, the optimal regular weighted discounted value of the belief root node calculated from the path corresponding to the action branch (a1 branch) is w, the optimal regular weighted discounted value of the belief root node calculated from the path corresponding to the action branch (a2 branch) is y, and w is greater than y. Then the maximum optimal regular weighted discounted value of the belief internal node is selected from the optimal regular weighted discounted values of the belief internal nodes (b11 and b12) subordinate to the belief root node in the path corresponding to the action branch (a1 branch). For example, the optimal regular weighted discounted value corresponding to b11 is greater than the optimal regular weighted discounted value corresponding to b12. Then the maximum optimal regular weighted discounted value of the belief leaf node is selected from the paths corresponding to the action branches (a1 branch and a2 branch) corresponding to the maximum belief internal node (b11). For example, it is determined that the optimal regular weighted discounted value corresponding to the belief leaf node b201 is greater than the optimal regular weighted discounted value corresponding to the belief leaf node b204. The optimal interaction strategy is determined according to the maximum optimal regular weighted discounted value of each node. Since the belief root node corresponding to the action branch (a1 branch) has the maximum optimal regular weighted discounted value and the belief internal node corresponding to the action branch (a1 branch) has the maximum optimal regular weighted discounted value in the embodiment of the application, it is determined that the optimal interaction strategy is to first drive according to the action space a1, form the next state space, and then continue to drive according to the action space a1.

[0137] ​​​​In the embodiments of the present application, since the optimal interaction strategy is determined according to the action space on the action branch corresponding to the maximum optimal regular weighted discount value, the safety and reliability of autonomous driving for passengers can be improved when the ego vehicle drives according to the optimal interaction strategy.

[0138] In the embodiments, the method, device, system and medium for autonomous driving for passengers are provided. The full-sample tree is constructed based on the state transition model and the observation model. The inverse dynamic programming method is used to solve from the confident leaf node to the confident root node to obtain the optimal interaction strategy. The safety and reliability of autonomous driving for passengers can be improved when the ego vehicle drives according to the optimal interaction strategy. The method comprises: modeling an interaction decision problem by using a partially observable Markov decision process to obtain a modeling result; constructing a full-sample tree according to the modeling result; wherein the full-sample tree comprises a plurality of paths, and the paths comprise a confident root node, a confident internal node and a confident leaf node arranged from top to bottom; and solving from the confident leaf node of the full-sample tree to the confident root node by using an inverse dynamic programming method to determine an optimal interaction strategy, wherein the optimal interaction strategy comprises an acceleration action, a maintaining action, a deceleration action or a parking action.

[0139] Further, as an implementation of the method shown in FIG. 1, the embodiments of the present application provide a device for autonomous driving for passengers. FIG. 9 shows a structural schematic diagram of a device for autonomous driving for passengers according to some embodiments. As shown in FIG. 9, the device comprises a modeling unit 901, a construction unit 902 and a solving unit 903.

[0140] The modeling unit is configured to model an interaction decision problem by using a partially observable Markov decision process to obtain a modeling result.

[0141] The construction unit is configured to construct a full-sample tree according to the modeling result, wherein the full-sample tree comprises a plurality of paths, and the paths comprise a confident root node, a confident internal node and a confident leaf node arranged from top to bottom.

[0142] The solving unit is configured to solve from the confident leaf node of the full-sample tree to the confident root node by using an inverse dynamic programming method to determine an optimal interaction strategy, wherein the optimal interaction strategy comprises an acceleration action, a maintaining action, a deceleration action or a parking action.

[0143] In a specific application scenario, the modeling result comprises a state space, an observation space, an action space, a state transition model and an observation model. The construction unit specifically performs:

[0144] The state space generated by using the real observation of the ego vehicle to the other vehicle is sampled multiple times to obtain multiple paths, the confidence leaf node is the deepest confidence internal node, and the confidence internal node is generated according to the state space, the observation space, the action space, the state transition model and the observation model.

[0145] In a specific application scenario, the construction unit specifically performs:

[0146] The confidence root node is taken as a target node, and a next state space is generated by using the state transition model according to the target node and the action space;

[0147] A target observation space is determined by using the observation model according to the next state space and the observation space;

[0148] The confidence internal node is determined according to the next state space and the target observation space, the confidence internal node is taken as a target node, and the step of generating the next state space by using the state transition model according to the target node and the action space is repeatedly performed.

[0149] In a specific application scenario, the solving unit includes:

[0150] A pruning unit is configured to perform a pruning operation on the full-sample tree to obtain a full-sample tree policy tree, and the confidence internal node in the full-sample tree policy tree only includes one action branch;

[0151] A first determination unit is configured to determine an optimal regular weighted discount value of the confidence leaf node by using the return function based on the full-sample tree if the depth of the full-sample tree policy tree is not greater than a preset value;

[0152] A second determination unit is configured to determine an optimal regular weighted discount value of the confidence internal node according to the optimal regular weighted discount value of the confidence leaf node;

[0153] A third determination unit is configured to calculate an optimal regular weighted discount value of the confidence root node according to the optimal regular weighted discount value of the confidence internal node;

[0154] A fourth determination unit is configured to determine an optimal interaction strategy according to the optimal regular weighted discount value of the confidence root node, the optimal regular weighted discount value of the confidence internal node and the optimal regular weighted discount value of the confidence leaf node.

[0155] According to an embodiment of the present application, a storage medium is provided, and the storage medium stores at least one executable instruction. The computer executable instruction can execute the autonomous proxy driving interaction decision method in any method embodiment.

[0156] Based on such understanding, the technical scheme of the present application can be embodied in the form of a software product, which can be stored in a nonvolatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in various implementation scenarios of the present application.

[0157] The embodiment of the present application also provides an electronic device for autonomous customer driving interaction decision, which can be a computer, a smart phone, a tablet computer, a smart watch, a server, or a network device, etc. The entity device includes a storage medium and a processor; the storage medium is used to store a computer program; and the processor is used to execute the computer program to realize the autonomous customer driving interaction decision method.

[0158] Optionally, the electronic device can also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface can include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The optional user interface can also include a USB interface, a card reader interface, etc. The network interface can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0159] Those skilled in the art can understand that the structure of the electronic device provided by the embodiment does not constitute a limitation on the electronic device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0160] The storage medium can also include an operating device and a network communication module. The operating device is a program for managing and saving hardware and software resources of the electronic device, supporting the running of information processing programs and other software and / or programs. The network communication module is used to realize the communication between the controls inside the storage medium, and the communication with other hardware and software in the entity device.

[0161] The embodiment of the present application also provides an autonomous customer driving interaction decision system, which includes:

[0162] a processor; and

[0163] a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the steps of the autonomous customer driving interaction decision method.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software with a necessary general hardware platform, or by hardware.

[0165] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of a preferred embodiment scenario, and the units or flows in the drawings are not necessarily essential for implementing the present application. Those skilled in the art can understand that the units in the devices in the embodiment scenario can be distributed in the devices in the embodiment scenario according to the description of the embodiment scenario, or can be changed and located in one or more devices different from the embodiment scenario. The units of the above embodiment scenario can be combined into one unit, or can be further split into multiple sub-units.

[0166] The above serial numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiment scenario. The above disclosure is only several specific embodiment scenarios of the present application, but the present application is not limited thereto, and any changes that those skilled in the art can think of should fall within the protection scope of the present application.

Claims

1. An autonomous guest driving interaction decision-making method, wherein, The method comprises the following steps: modeling an interactive decision-making problem by using a partially observable Markov decision process, and obtaining a modeling result; constructing a full-sample tree according to the modeling result, wherein the full-sample tree comprises a plurality of paths, and the paths comprise confidence root nodes, confidence internal nodes and confidence leaf nodes arranged from top to bottom; solving from the confidence leaf nodes of the full-sample tree to the confidence root nodes by using a backward dynamic programming method to determine an optimal interactive strategy, wherein the optimal interactive strategy comprises an acceleration action, a maintaining action, a deceleration action or a parking action.

2. The method of claim 1, wherein, The modeling result comprises a state space, an observation space, an action space, a state transition model and an observation model; and the step of constructing the full-sample tree according to the modeling result comprises: sampling the state space generated by using real observations of the ego vehicle to the other vehicle as the confidence root nodes of the full-sample tree for a plurality of times to obtain a plurality of paths; wherein the confidence leaf nodes are the deepest confidence internal nodes, and the confidence internal nodes are generated according to the state space, the observation space, the action space, the state transition model and the observation model.

3. The method of claim 2, wherein, sampling the state space generated by using real observations of the ego vehicle to the other vehicle as the confidence root nodes of the full-sample tree once to obtain one path, which comprises: taking the confidence root nodes as target nodes, generating a next state space by using the state transition model according to the target nodes and the action space; determining a target observation space by using the observation model according to the next state space and the observation space; determining the confidence internal nodes according to the next state space and the target observation space, taking the confidence internal nodes as target nodes, and repeatedly performing the step of generating a next state space by using the state transition model according to the target nodes and the action space.

4. The method of claim 1, wherein, The modeling result further comprises a reward function.

5. The method of claim 4, wherein, The step of solving from the confidence leaf nodes of the full-sample tree to the confidence root nodes by using the backward dynamic programming method to obtain the optimal interactive strategy comprises: performing a pruning operation on the full-sample tree to obtain a plurality of full-sample tree strategy trees, wherein the confidence internal nodes in the full-sample tree strategy trees only comprise one action branch; if the depths of all the full-sample tree strategy trees are not greater than a preset value, determining optimal regular weighted discount values of the confidence leaf nodes based on the full-sample tree by using the reward function; determining optimal regular weighted discount values of the confidence internal nodes according to the optimal regular weighted discount values of the confidence leaf nodes; calculating optimal regular weighted discount values of the confidence root nodes according to the optimal regular weighted discount values of the confidence internal nodes; determining the optimal interactive strategy according to the optimal regular weighted discount values of the confidence root nodes, the optimal regular weighted discount values of the confidence internal nodes and the optimal regular weighted discount values of the confidence leaf nodes.

6. The method of claim 2, wherein, The state space comprises ego vehicle longitudinal position and speed information and other vehicle longitudinal position and speed information.

7. The method of claim 2, wherein, The action space includes driving actions that the ego vehicle can take and driving actions that the other vehicle can take; wherein the driving actions include accelerating action, maintaining action, decelerating action, and stopping action.

8. An autonomous guest driving interaction decision device, wherein, Comprise: a modeling unit configured to model the interactive decision problem using a partially observable Markov decision process to obtain a modeling result; a constructing unit configured to construct a full-sampling tree according to the modeling result; wherein the full-sampling tree includes a plurality of paths, and each path includes a belief root node, a belief internal node, and a belief leaf node arranged from top to bottom; a solving unit configured to solve from the belief leaf node of the full-sampling tree to the belief root node using a backward dynamic programming method to determine an optimal interaction strategy, wherein the optimal interaction strategy includes accelerating action, maintaining action, decelerating action, or stopping action.

9. An autonomous guest-driving car interactive decision system, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the steps of the autonomous guest-driving car interactive decision method of any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, wherein, The computer program, when executed by a processor, implements the steps of the autonomous guest-driving car interactive decision method of any one of claims 1-7.

Images