Autonomous valet driving interactive decision-making method, device and system and medium
By constructing a full sampling tree and using inverse dynamic programming to determine the optimal interaction strategy, the safety and reliability problem of autonomous valet driving in the blind spot of parking lot intersections was solved, and the safe operation of the vehicle and other vehicles was achieved.
Patent Information
- Application Number
- CN202410724757.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-05
- Publication Date
- 2025-12-05
AI Technical Summary
In scenarios where obstacles around parking lot intersections create blind spots for the vehicle's perception, how can we ensure the safe operation of both the vehicle and other vehicles during autonomous valet driving, and improve the safety and reliability of autonomous valet driving interaction strategies?
A full sampling tree is constructed using a state transition model and an observation model. Starting from the confidence leaf node of the full sampling tree, an inverse dynamic programming method is used to solve the problem up to the confidence root node to determine the optimal interaction strategy, including acceleration, holding, deceleration, or stopping actions.
It improves the safety and reliability of autonomous valet driving, ensuring the safe operation of both the vehicle and other vehicles in complex environments.
Smart Images

Figure CN121069744A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of autonomous guest driving, and in particular to an autonomous guest driving interactive decision method, device, system and medium. BACKGROUND
[0002] In the scene where the obstacles around the parking lot intersection constitute the self-vehicle perception visual field blind area, how to ensure the safe operation of the self-vehicle and other vehicles when the self-vehicle is in autonomous guest driving, that is, how the self-vehicle obtains a safe and reliable autonomous guest driving interactive strategy, is a major challenge faced by the autonomous guest driving decision planning field.
[0003] In view of this major challenge faced by the autonomous guest driving decision planning field, some research uses a deep neural network to infer the optimal interactive strategy of autonomous guest driving facing the blind area of the parking lot intersection, but the learning-based method has poor scene migration performance. Therefore, how to improve the safety and reliability of the autonomous guest driving interactive strategy has become a technical problem to be solved by the technical personnel in the field. SUMMARY
[0004] The present application provides an autonomous guest driving interactive decision method, device, system and medium, which constructs a full-sample tree based on a state transition model and an observation model, solves from a confident leaf node of the full-sample tree to a confident root node using a backward dynamic programming method, finally obtains an optimal interactive strategy, and uses the optimal interactive strategy to indicate the operation of the self-vehicle, which can improve the safety and reliability of autonomous guest driving.
[0005] In a first aspect, an embodiment of the present application provides an autonomous guest driving interactive decision method, comprising:
[0006] Modeling an interactive decision problem using a partially observable Markov decision process to obtain a modeling result;
[0007] According to the modeling result, a full-sample tree is constructed; wherein the full-sample tree includes a plurality of paths, and the paths include a confident root node, a confident internal node and a confident leaf node arranged from top to bottom;
[0008] A backward dynamic programming method is used to solve from the confident leaf node of the full-sample tree to the confident root node to determine an optimal interactive strategy, and the optimal interactive strategy includes an acceleration action, a maintenance action, a deceleration action or a parking action.
[0009] In the present application, a full-sample tree is constructed based on a state transition model and an observation model, a backward dynamic programming method is used to solve from a confident leaf node of the full-sample tree to a confident root node, finally an optimal interactive strategy is obtained, and the optimal interactive strategy is used to indicate the operation of the self-vehicle, which can improve the safety and reliability of autonomous guest driving.
[0010] In some embodiments, the modeling result comprises a state space, an observation space, an action space, a state transition model and an observation model; and the step of constructing the full-sample tree according to the modeling result comprises:
[0011] sampling the state space generated by using the real observation of the ego vehicle to the other vehicle as the confident root node of the full-sample tree multiple times to obtain multiple paths; wherein the confident leaf node is the deepest confident internal node, and the confident internal node is generated according to the state space, the observation space, the action space, the state transition model and the observation model.
[0012] In the present application, the state space generated by using the real observation of the ego vehicle to the other vehicle is sampled as the confident root node of the full-sample tree multiple times to obtain paths, and then the optimal interaction strategy is obtained, and the ego vehicle is instructed to run by using the optimal interaction strategy, which can improve the safety and reliability of autonomous driving for passengers.
[0013] In some embodiments, sampling the state space generated by using the real observation of the ego vehicle to the other vehicle as the confident root node of the full-sample tree once to obtain one path comprises:
[0014] taking the confident root node as a target node, generating a next state space by using the state transition model according to the target node and the action space;
[0015] determining a target observation space by using the observation model according to the next state space and the observation space;
[0016] determining the confident internal node according to the next state space and the target observation space, taking the confident internal node as a target node, and repeating the step of generating a next state space by using the state transition model according to the target node and the action space.
[0017] In the present application, the confident root node is taken as a target node, and a next state space is generated by using the state transition model together with the action space. A target observation space is obtained by using the observation model according to the next state space and the observation space, and finally a confident internal node is obtained. The step of generating a next state space is repeated, and then one path in the full-sample tree is obtained, which helps to finally obtain an optimal interaction strategy. The ego vehicle is instructed to run by using the optimal interaction strategy, which can improve the safety and reliability of autonomous driving for passengers.
[0018] In some embodiments, the modeling result further comprises a reward function.
[0019] In some embodiments, the step of solving the optimal interaction strategy by using the inverse dynamic programming method from the confident leaf node of the full-sample tree comprises:
[0020] pruning operation is performed on the full sampling tree to obtain a full sampling tree strategy tree, and the confident internal node in the full sampling tree strategy tree only contains one action branch;
[0021] If the depth of the full sampling tree strategy tree is not greater than a preset value, the optimal regular weighted discount value of the confident internal node is determined based on the full sampling tree and the return function.
[0022] According to the optimal regular weighted discount value of the confident internal node, the optimal regular weighted discount value of the confident root node is calculated.
[0023] According to the optimal regular weighted discount value of the confident internal node, the optimal regular weighted discount value of the confident root node is calculated.
[0024] According to the optimal regular weighted discount value of the confident root node, the optimal regular weighted discount value of the confident internal node, and the optimal regular weighted discount value of the confident leaf node, the optimal interaction strategy is determined.
[0025] In the present application, the full sampling number is pruned to obtain a full sampling number strategy tree, and the depth thereof is compared with a preset value to finally determine the optimal regular weighted discount value of the confident internal node, and then the optimal interaction strategy is obtained. The optimal interaction strategy is used to indicate the operation of the ego vehicle, which can improve the safety and reliability of autonomous taxi driving.
[0026] In some embodiments, the state space includes ego vehicle longitudinal position and speed information, and other vehicle longitudinal position and speed information.
[0027] In the present application, the multi-directional state space can be understood to obtain the optimal interaction strategy, and the optimal interaction strategy is used to indicate the operation of the ego vehicle, which can improve the safety and reliability of autonomous taxi driving.
[0028] In some embodiments, the action space includes possible driving actions of the ego vehicle and possible driving actions of the other vehicle; wherein the driving actions include acceleration actions, maintenance actions, deceleration actions, and parking actions.
[0029] In a second aspect, the embodiments of the present application also provide an autonomous taxi driving interaction decision device, comprising:
[0030] The modeling unit is configured to model the interaction decision problem by using a partially observable Markov decision process to obtain a modeling result.
[0031] The construction unit is configured to construct a full sampling tree according to the modeling result; wherein the full sampling tree includes a plurality of paths, and the paths include a confident root node, a confident internal node, and a confident leaf node arranged from top to bottom.
[0032] a solving unit configured to solve from a confidence leaf node of the full sampling tree to a confidence root node using a backward dynamic programming method to determine an optimal interaction strategy, the optimal interaction strategy including an acceleration action, a keep action, a deceleration action, or a stop action.
[0033] The device in the present application constructs a full sampling tree based on a state transition model and an observation model, solves from a confidence leaf node of the full sampling tree to a confidence root node using a backward dynamic programming method, finally obtains an optimal interaction strategy, and indicates the operation of the ego vehicle by using the optimal interaction strategy, thereby improving the safety and reliability of the autonomous guest driving car.
[0034] In a third aspect, the embodiments of the present application further provide an autonomous guest driving car interaction decision system, comprising:
[0035] a processor; and
[0036] a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the steps of the autonomous guest driving car interaction decision method.
[0037] The system in the present application constructs a full sampling tree based on a state transition model and an observation model, solves from a confidence leaf node of the full sampling tree to a confidence root node using a backward dynamic programming method, finally obtains an optimal interaction strategy, and indicates the operation of the ego vehicle by using the optimal interaction strategy, thereby improving the safety and reliability of the autonomous guest driving car.
[0038] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the autonomous guest driving car interaction decision method.
[0039] The readable storage medium in the present application constructs a full sampling tree based on a state transition model and an observation model, solves from a confidence leaf node of the full sampling tree to a confidence root node using a backward dynamic programming method, finally obtains an optimal interaction strategy, and indicates the operation of the ego vehicle by using the optimal interaction strategy, thereby improving the safety and reliability of the autonomous guest driving car. BRIEF DESCRIPTION OF DRAWINGS
[0040] Figure 1 An exemplary scenario schematic diagram provided according to some embodiments is shown;
[0041] Figure 2 An exemplary flowchart of an autonomous guest driving car interaction decision method provided according to some embodiments is shown;
[0042] Figure 3 An exemplary schematic diagram of a state transition model provided according to some embodiments is shown;
[0043] Figure 4 An exemplary schematic diagram of an observation model is shown according to some embodiments;
[0044] Figure 5 An exemplary schematic diagram of a standard tree is shown according to some embodiments;
[0045] Figure 6 An exemplary schematic diagram of a full volume sampling number is shown according to some embodiments;
[0046] Figure 7 An exemplary schematic diagram of a path composed of any one scene sequence combined with all action sequences is shown according to some embodiments;
[0047] Figure 8 An exemplary schematic diagram of a path composed of any one action sequence combined with all scene sequences is shown according to some embodiments;
[0048] Figure 9 An exemplary schematic diagram of a structure of an autonomous customer-to-vehicle interaction decision device is shown according to some embodiments. DETAILED DESCRIPTION
[0049] For the purpose of clarity and a thorough understanding of the application, the application will be described in greater detail with reference to the accompanying drawings, wherein:
[0050] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.
[0051] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.
[0052] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.
[0053] In the scenario where the obstacles around the parking lot intersection constitute the self-vehicle perception visual field blind area, how to ensure the safe operation of the self-vehicle and the other vehicle when the self-vehicle is in autonomous guest driving, that is, how the self-vehicle obtains a safe and reliable autonomous guest driving interaction strategy, is a major challenge in the field of autonomous guest driving decision planning.
[0054] Figure 1 An exemplary scenario schematic diagram provided according to some embodiments is shown, in which Figure 1 In the parking lot, a plurality of vehicles are parked in the parking spaces. Due to the existence of the parked vehicles, the self-vehicle 11 has a perception visual field blind area 12 when driving, which leads to the fact that the self-vehicle cannot directly observe the other vehicle 13 when the other vehicle enters the self-vehicle's perception visual field blind area, and thus the self-vehicle and the other vehicle may be in danger when the self-vehicle is in autonomous guest driving.
[0055] In view of this major challenge in the field of autonomous guest driving decision planning, some researches use deep neural networks to infer the optimal interaction strategy for autonomous guest driving facing the blind area of the parking lot intersection, but the scene migration of the learning-based method is poor. Therefore, how to improve the safety and reliability of the autonomous guest driving interaction strategy has become a technical problem to be solved by the technical personnel in the field.
[0056] To solve the above technical problem, the embodiments of the present application provide an autonomous guest driving interaction decision method, device, system and medium. In the method, a full-sample tree is constructed based on a state transition model and an observation model, and a reverse dynamic programming method is used to solve from a confident leaf node of the full-sample tree to a confident root node, and finally the best interaction strategy is obtained. The best interaction strategy is used to indicate the operation of the self-vehicle, which can improve the safety and reliability of autonomous guest driving.
[0057] Figure 2 An exemplary flowchart of an autonomous guest driving interaction decision method provided according to some embodiments is shown, which includes steps S100-S300.
[0058] It should be noted that the autonomous guest driving interaction decision method in the embodiments of the present application can be applied not only to the parking lot intersection blind area interaction scenario, but also to the city navigation driving no light control intersection blind area interaction scenario, etc.
[0059] In some embodiments, the system architecture of the autonomous guest driving interaction strategy method includes two blocks of problem modeling and problem solving, wherein the problem modeling block includes step S100, and the problem solving block includes steps S200 and S300.
[0060] S100, model the interaction decision problem by using a partially observable Markov decision process, and obtain a modeling result.
[0061] In the embodiments of the application, the modeling result includes a state space, an observation space, an action space, a state transition model, an observation model and a reward function.
[0062] Specifically, the modeling process of the autonomous driving interaction decision problem for the parking lot intersection blind area includes the following steps:
[0063] The state space is composed of the longitudinal position and speed information of the ego vehicle χ0=[s0 v0] T and the longitudinal position and speed information of the other vehicle χ1=[s1 v1] T The state space of the autonomous driving interaction decision problem for the parking lot intersection blind area is constructed as follows:
[0064] χ=[χ0,χ1] T (1)
[0065] In the embodiments of the application, the autonomous driving interaction decision only considers that the ego vehicle should perform acceleration, speed maintenance, deceleration and parking actions, so the state space is generated only according to the longitudinal position and speed information of the ego vehicle and the longitudinal position and speed information of the other vehicle.
[0066] In the embodiments of the application, before constructing the full-sample tree, the first state space is generated according to the longitudinal position and speed information of the ego vehicle and the longitudinal position and speed information of the other vehicle obtained when the ego vehicle actually observes the other vehicle. The other vehicle can be actually observed by the ego vehicle before it enters the non-perception blind area of the ego vehicle. The longitudinal position and speed information of the ego vehicle and the other vehicle are determined when the other vehicle is in the non-perception blind area of the ego vehicle, and the specific obtaining method can be obtained according to the real sensors installed on the ego vehicle.
[0067] Since there are multiple state spaces in the full-sample tree constructed subsequently, the remaining state spaces are determined according to the state transition model, which will be described in detail later.
[0068] The observation space is composed of the longitudinal position and speed information of the ego vehicle detected by the sensor o0=[s0 v0] T and the longitudinal position and speed information of the other vehicle detected by the virtual sensor o1=[s1 v1] T The observation space of the autonomous driving interaction decision problem for the parking lot intersection blind area is constructed as follows:
[0069] z=[o0,o1] T (2)
[0070] In the embodiments of the present application, the observation space is predicted, and multiple observation spaces can be predicted at one time. The parameters in each observation space are different, and each observation space has a corresponding possibility. In some embodiments, the possibility corresponding to the observation space represents the probability of the occurrence of the longitudinal position and speed information of the ego vehicle and the longitudinal position and speed information of the other vehicle in the observation space.
[0071] Action space: based on the driving actions that the ego vehicle and the other vehicle can take and the driving actions that the ego vehicle and the other vehicle can take The action space for the autonomous guest driving interaction decision problem facing the blind area of the parking lot intersection is constructed as:
[0072] A=[a0,a1] T (3)
[0073] In the formula, and represent the acceleration action, the speed maintaining action, the deceleration action and the parking action of the ego vehicle respectively, and represent the acceleration action, the speed maintaining action, the deceleration action and the parking action of the other vehicle respectively.
[0074] In the embodiments of the present application, the action space is randomly generated, and is used to simulate the driving actions that the ego vehicle and the other vehicle can take.
[0075] State transition model: the state transition model for the autonomous guest driving interaction decision problem facing the blind area of the parking lot intersection is constructed based on the kinematic model of the ego vehicle and the other vehicle, and is as follows:
[0076] T(χ′|χ,a)=P(χ′|χ,a)=P(χ′0|χ0,a0)P(χ′1|χ1,a1) (4)
[0077] In the formula, P(χ′0|χ0,a0) and P(χ′1|χ1,a1) are the state transition models of the ego vehicle and the other vehicle respectively, and can be expressed as:
[0078]
[0079] Exemplarily, Figure 3 An exemplary schematic diagram of a state transition model provided according to some embodiments is shown. The specific use method of the state transition model is described hereinafter, and will not be described here again.
[0080] Observation model: assuming that the sensor outputting the longitudinal position and speed information of the ego vehicle and the virtual sensor outputting the longitudinal position and speed information of the other vehicle are decoupled, the observation model for the autonomous guest driving interaction decision problem facing the blind area of the parking lot intersection can be expressed as:
[0081] O(z|χ',a) = P(z|χ',a) = P(o0|χ'0)P(o1|χ'1) (7)
[0082] Assuming that the sensor can accurately detect the longitudinal position and speed information of the ego vehicle, the observation model of the ego vehicle can be represented as:
[0083]
[0084] Assuming that the longitudinal position and speed information detected by the virtual sensor of the other vehicle has noise, the observation model of the other vehicle can be represented as:
[0085] P(o1|χ'1) = N(χ'1,σ) (9)
[0086] In the formula, N(χ'1,σ) represents a Gaussian distribution.
[0087] Exemplarily, Figure 4 An exemplary schematic diagram of an observation model according to some embodiments is shown. The specific use method of the observation model is described hereinafter and will not be repeated here.
[0088] Reward function: The reward function considering safety, comfort and efficiency can be described as:
[0089]
[0090] In the formula, 0≤λ≤1 is a weight coefficient.
[0091] S200, constructing a full-sample tree according to the modeling result; wherein the full-sample tree includes a plurality of paths, and the path includes a confidence root node, a confidence internal node and a confidence leaf node arranged from top to bottom.
[0092] Specifically, the full-sample tree is a sparse approximation of the standard tree, and contains all action sequences and sampled scene sequences. Compared with Figure 5 the standard tree and Figure 6 the full-sample tree, the full-sample tree performs pruning operation on the standard tree.
[0093] In order to ensure Figure 6 clarity and neatness, Figure 6 only the full-sample tree including four scene sequences obtained by four samplings and all action sequences is drawn. Due to the limitation of the length, Figure 6 the full-sample tree in the embodiment sets a certain depth. One scene sequence obtained by one sampling and all action sequences constitute a path. Figure 6All paths of the full-sampling tree include a confidence root node (b0), confidence internal nodes (b11, b12, b13 and b14) and confidence leaf nodes (b201, b204, b206, b208, b210, b211, b213, b214 and b215) arranged from top to bottom.
[0094] Figure 7 An example is shown to illustrate a path diagram of any one scene sequence combined with all action sequences according to some embodiments. As shown in Figure 7 The path of any one scene sequence combined with all action sequences obtained by sampling is fissioned at different action branches of the full-sampling tree confidence node. Figure 8 An example is shown to illustrate a path diagram of any one action sequence combined with all scene sequences according to some embodiments. As shown in Figure 8 The path of any one action sequence combined with all scene sequences obtained by sampling will not be fissioned, and the number of paths is equal to the number of all scene sequences obtained by sampling K. Therefore, if the depth of the full-sampling tree is D, and the number of actions is |A|, then the number of confidence leaf nodes of the full-sampling tree is in the order of O(|A| D K).
[0095] In some embodiments, the modeling result includes a state space, an observation space, an action space, a state transition model and an observation model; and the step of constructing the full-sampling tree according to the modeling result includes:
[0096] sampling the confidence root node of the full-sampling tree with the state space generated by using real observation of the ego vehicle to the other vehicle to obtain a plurality of paths; wherein the confidence leaf node is the deepest confidence internal node, and the confidence internal node is generated according to the state space, the observation space, the action space, the state transition model and the observation model.
[0097] In the embodiments of the present application, the number of sampling times is the same as the number of scene sequences.
[0098] In some embodiments, sampling the confidence root node of the full-sampling tree with the state space generated by using real observation of the ego vehicle to the other vehicle to obtain a path includes:
[0099] taking the confidence root node as a target node, and generating a next state space by using the state transition model according to the target node and the action space.
[0100] In the embodiments of the present application, the driving actions that the ego vehicle and the other vehicle can take are simulated by using the action space. First, a next state space is generated based on the state space and the state transition model generated when the ego vehicle actually observes the other vehicle, i.e., the longitudinal position and speed information of the ego vehicle and the other vehicle are predicted.
[0101] Referring to Figure 6 , in Figure 6 , the belief root node b0 extends different action branches (including an a1 branch corresponding to the action space A1 and an a2 branch corresponding to the action space A2). It should be noted that, Figure 6 The a1 in the belief space A1 is not related to the a1 in the action space A. Figure 6 The a1 in the belief space A1 includes the a0 and the a1 in the action space A, Figure 6 The a2 in the belief space A2 is the same.
[0102] According to the next state space and the observation space, the observation model is used to determine a target observation space.
[0103] In the embodiments of the present application, the observation model is used to filter the observation spaces with a required possibility from the observation space as the target observation space. A random number is generated by the observation model, which is a probability. Since the observation space has a corresponding possibility, the possibility of the observation space is compared with the random number. If the possibility is greater than the random number, the observation space with the greater possibility is determined as the target observation space. It should be noted that the target observation space can be multiple. Referring to Figure 6 , Figure 6 The z1 and the z2 in the belief space A1 and the belief space A2 are both target observation spaces.
[0104] According to the next state space and the target observation space, the belief internal node is determined, the belief internal node is taken as a target node, and the step of generating the next state space according to the target node and the action space by using the state transition model is repeatedly executed. In this way, the next state space is continuously generated, and a full sampling tree with a certain depth can be formed.
[0105] In some embodiments, the step of generating the next state space according to the target node and the action space by using the state transition model can be automatically stopped from being repeatedly executed, or a preset stopping depth can be set, and the repeatedly execution is stopped when the preset stopping depth is reached.
[0106] In the embodiments of the present application, the next state space and the target observation space are used to determine the belief internal node corresponding to the next state space. The belief internal node is the confidence degree determined by using the next state space and the target observation space, and the confidence degree can represent the reliable degree of the occurrence of the next state space.
[0107] For example,Figure 7 The path generation process in the action branch (a1 branch) side first, through the belief root node b0 and the action space a1, a corresponding next state space is generated by using the state transition model, and the next state space and the target observation space z1 are used to determine the belief internal node b11 corresponding to the next state space. The belief internal node b11 is the confidence of the next state space. Then, the belief internal node b11 is taken as the target node, and according to the target node b11 and the action space a1, a corresponding next state space is generated by using the state transition model. The next state space and the target observation space z1 are used to determine the belief internal node b201 corresponding to the next state space. In addition, the belief internal node b11 is taken as the target node, and according to the target node b11 and the action space a2, a corresponding next state space is generated by using the state transition model. The next state space and the target observation space z2 are used to determine the belief internal node b204 corresponding to the next state space.
[0108] For the action branch (a2 branch) side, through the belief root node b0 and the action space a2, a corresponding next state space is generated by using the state transition model, and the next state space and the target observation space z2 are used to determine the belief internal node b14 corresponding to the next state space. The belief internal node b14 is the confidence of the next state space. Then, the belief internal node b14 is taken as the target node, and according to the target node b14 and the action space a1, a corresponding next state space is generated by using the state transition model. The next state space and the target observation space z2 are used to determine the belief internal node b214 corresponding to the next state space. In addition, the belief internal node b14 is taken as the target node, and according to the target node b14 and the action space a2, a corresponding next state space is generated by using the state transition model. The next state space and the target observation space z1 are used to determine the belief internal node b215 corresponding to the next state space.
[0109] In some other embodiments, the confidence represented by the belief internal node can be determined according to the probability of reaching the belief internal node. In the embodiments of the present application, the probability of reaching the belief internal node of the full-sampling tree is approximately represented by the ratio of the number of particles of the belief node of the full-sampling tree to the number of particles of the belief root node of the full-sampling tree.
[0110] An example is shown in FIG. 6, where the belief root node b0 of the full-sampling tree is taken as the target node, and according to the target node b0 and the action space a1, a corresponding next state space is generated by using the state transition model. The next state space and the target observation space z1 are used to determine the belief internal node b11 corresponding to the next state space. In addition, the belief internal node b11 is taken as the target node, and according to the target node b11 and the action space a2, a corresponding next state space is generated by using the state transition model. The next state space and the target observation space z2 are used to determine the belief internal node b204 corresponding to the next state space. Figure 6 The number of particles of the belief root node of the full-sampling tree in the action branch (a1 branch) side can be the sampling times of the full-sampling tree. Figure 6 Only four path corresponding to four scene sequences obtained by four sampling times are drawn, so Figure 6The number of particles of the confidence root node of the full sampling tree in FIG. 11 is 4. Only 2 of the 4 samplings pass through the confidence node b11 of the full sampling tree, so the number of particles of the confidence node b11 of the full sampling tree can be considered to be 2, and the number of particles of the confidence node b12 is 2, the number of particles of the confidence node b13 is 1, and the number of particles of the confidence node b14 is 3. At this time, the probability of reaching the confidence node b11 of the full sampling tree is 2 / 4 = 50%, the probability of reaching the confidence node b12 of the full sampling tree is 2 / 4 = 50%, the probability of reaching the confidence node b13 of the full sampling tree is 1 / 4 = 25%, and the probability of reaching the confidence node b14 of the full sampling tree is 3 / 4 = 75%.
[0111] The sum of the probabilities of reaching all confidence child nodes of one action branch of a confidence parent node of the full sampling tree is equal to the probability of reaching the confidence parent node. Here, the confidence parent node is the upper level node of the confidence child node, and the confidence node b11 is the confidence parent node of the confidence leaf node b201, and the confidence root node b0 is the confidence parent node of the confidence node b11. The sum of the probabilities of the confidence child nodes b213 and b214 on one action branch (a1 branch) of the confidence parent node b14 is equal to the probability of reaching the confidence parent node b14. Therefore, in the process of calculating the value of the confidence parent node of the full sampling tree, the probability of reaching the confidence child node of the full sampling tree is used to represent the degree of influence of the value of the confidence child node of the full sampling tree on the value of the confidence parent node of the full sampling tree.
[0112] S300, the confidence leaf node of the full sampling tree is used to solve the confidence root node by the inverse dynamic programming method to determine the optimal interaction strategy, and the optimal interaction strategy includes the acceleration action, the maintenance action, the deceleration action or the parking action.
[0113] In the embodiment of the application, the optimal interaction strategy is determined by the inverse dynamic programming method, so that the ego vehicle drives according to the optimal interaction strategy.
[0114] In the embodiment of the application, if the pruning operation is performed on the full sampling tree, the confidence internal node of the full sampling tree only contains one action branch, and then a full sampling tree strategy tree is obtained after pruning. For example, as shown in FIG. 12, Figure 6 Figure 6 A full sampling tree strategy tree in FIG. 12 can include one action branch (a1 branch) of the confidence root node b0 and one action branch (a1 branch) of the confidence node b11.
[0115] For any strategy tree π of the full sampling tree, the true value V π (b0) of the value of the confidence root node b0 under the strategy tree π satisfies the following relationship with the estimated value at least with a probability of 1-τ.
[0116]
[0117] where |π| is the number of belief nodes of policy tree π, and the estimate value is calculated by using the reward function.
[0118] The right side of inequality (11) is maximized, and the approximate optimal policy of belief root node b0 in any policy tree π of the full-sample tree is
[0119]
[0120] Let π' represent any policy of belief root node b0, then the true value of belief root node b0 under the approximate optimal policy tree is and the true value of belief root node b0 under any policy tree π' is V π′ (b0). At least with probability 1-τ, the following relationship is satisfied
[0121]
[0122] According to inequality (13), the value V π′ (b0) of any policy tree π' under belief root node b0 can be used to approximate the true value of belief root node b0 under the approximate optimal policy tree . According to the right side of inequality (13), the fewer the number of belief nodes of any policy tree π', the smaller the approximation error, and the influence of the number of belief nodes of any policy tree π' on the approximation error can be suppressed by increasing the number of scene sequences of the full-sample tree. Further, the full-sample tree policy tree is taken as any policy tree on the right side of inequality (13), and a policy tree that maximizes the following optimization objective is selected from the full-sample tree policy tree set Π D as the optimal policy of belief root node b0.
[0123]
[0124] For the optimization objective described in equation (14), the regular weighted discounted value of belief node b under the full-sample tree policy tree π is defined as:
[0125]
[0126] where |Φ b | is the number of full-sample tree scene sequences of belief node b; γ is the discount factor; Δ(b) is the depth of belief node b in the full-sample tree policy tree π; π b is the subtree of the full-sample tree policy tree π, whose root node is belief node b; |Φ b|K represents the probability of reaching the belief node b.
[0127] According to formula (15), the regular weighted discounted value of the belief root node b0 under the action of the full sampling tree policy tree π can be expressed as
[0128]
[0129] In the formula, s φ is the initial state of the scene sequence φ; R(s φ , π(b0)) is the immediate reward of the belief root node b0 under the action of the scene sequence φ and the full sampling tree policy tree π; and CH(b0, π(b0)) is the set of belief child nodes generated by the belief root node b0 under the action of the scene sequence φ and the full sampling tree policy tree π.
[0130] According to formula (16), if the belief child node is used as the root of the policy sub-tree π b contains a large number of belief nodes, and the number of full sampling tree scene sequences through the belief child node is insufficient, the regular weighted discounted value of the belief child node is small, and the influence on the regular weighted discounted value of the belief root node is small. At the same time, according to formula (16), the regular weighted discounted value of the belief root node b0 under the action of the full sampling tree policy tree π can be expressed by its belief child nodes. If the belief child node obtains the optimal regular weighted discounted value under the action of the full sampling tree policy tree π, the belief root node b0 also obtains the optimal regular weighted discounted value under the action of the full sampling tree policy tree π. Therefore, the optimal regular weighted discounted value of the belief root node b0 is obtained by screening the policy space of the full sampling tree under the belief root node b0, and then the optimal policy of the belief root node b0 is obtained.
[0131] Based on the derivation and specific description of formula (11) to formula (16), in some embodiments, the step of obtaining the optimal interaction policy by using the inverse dynamic programming method from the belief leaf node of the full sampling tree comprises:
[0132] Pruning the full sampling tree to obtain a plurality of full sampling tree policy trees, wherein the belief internal nodes in the full sampling tree policy trees only contain one action branch.
[0133] In the embodiments of the present application, if the depth of the full sampling tree policy tree π is greater than R max / λ(1-γ)+1, then:
[0134]
[0135] According to the Bellman equation, the greater the depth of the belief child node, the smaller the influence on , and bounded. Substituting inequality (17) into inequality (18), we have
[0136] υ π (b0)≤-λ (18)
[0137] From inequalities (17) and (18), we can see that the factor λ|π| increases with the depth of the full-sampling tree policy tree π, and when the depth of the full-sampling tree policy tree π is greater than R max / λ(1-γ)+1, it dominates, so that the regularized weighted discounted value of the confidence root node b0 is less than the regularized weighted discounted value υ π0 (b0)≥0 under the full-sampling tree π0. Therefore, the full-sampling tree policy tree π cannot contain confidence child nodes with a depth greater than R max / λ(1-γ)+1.
[0138] Therefore, in the embodiments of the present application, if the depth of the full-sampling tree policy tree is greater than the preset value, the full-sampling tree policy tree is pruned, and the depth after pruning is not greater than the preset value.
[0139] If the depth of all the full-sampling tree policy trees is not greater than the preset value, the optimal regularized weighted discounted value of the confidence leaf node is determined based on the full-sampling tree and using the return function.
[0140] Specifically, according to the above analysis process deduced from formula (11) to formula (16), the optimal policy of the confidence root node b0 can be recursively solved from the confidence leaf node to the confidence root node b0 using the dynamic programming method shown in formula (17). Figure 6
[0141] wherein the optimal regularized weighted discounted value of the confidence leaf node can be expressed as
[0142]
[0143] wherein π0 is the full-sampling tree, and wherein is generated according to the return function.
[0144] The optimal regularized weighted discounted value of the confidence internal node is determined according to the optimal regularized weighted discounted value of the confidence leaf node.
[0145] In some embodiments, the optimal regularized weighted discounted value of the confidence internal node can be expressed as:
[0146]
[0147] wherein τ(b, a, z) represents the confidence child node of the confidence internal node b under the action branch a and the observation branch z, and wherein b represents the confidence of the confidence node, and b represents the driving action guiding the self-driving operation in the representative interaction strategy.
[0148] It should be noted that the parameters not specifically described in the embodiments of the present application are all calibration parameters.
[0149] According to the optimal regular weighted discount value of the confidence internal node, the optimal regular weighted discount value of the confidence root node is calculated.
[0150] In some embodiments, the optimal regular weighted discount value of the confidence root node is also calculated according to the optimal regular weighted discount value of the confidence internal node, and is calculated by formula (20), wherein the confidence internal node is taken as the confidence leaf node in the formula, and the optimal regular weighted discount value of the confidence root node is calculated.
[0151] Based on the full-quantity sampling tree π0, the optimal regular weighted discount values of the confidence leaf nodes b 201 , b 204 , b 206 , b 208 , b 210 , b 211 , b 213 , b 214 and b 215 can be obtained; the optimal regular weighted discount values of the confidence internal nodes b 11 , b 12 , b 13 and b 14 are calculated by formula (20); and the optimal regular weighted discount value of the confidence root node b0 is calculated again by formula (20).
[0152] According to the optimal regular weighted discount value of the confidence root node, the optimal regular weighted discount value of the confidence internal node and the optimal regular weighted discount value of the confidence leaf node, the best interaction strategy is determined.
[0153] In the embodiments of the present application, as Figure 6As shown, from the optimal regular weighted discount values calculated from the two action branches (a1 branch and a2 branch) of the confidence root node b0, the maximum optimal regular weighted discount value is screened out, for example, the optimal regular weighted discount value of the confidence root node calculated through the path corresponding to the action branch (a1 branch) is w, the optimal regular weighted discount value of the confidence root node calculated through the path corresponding to the action branch (a2 branch) is y, and in addition w is greater than y, then the maximum optimal regular weighted discount value of the subordinate confidence internal nodes (b11 and b12) of the confidence root node in the path corresponding to the action branch (a1 branch) is screened out, for example, the optimal regular weighted discount value corresponding to b11 is greater than the optimal regular weighted discount value corresponding to b12. Then the maximum optimal regular weighted discount value of the confidence leaf node corresponding to the path is continuously screened out from the action branches (a1 branch and a2 branch) corresponding to the maximum confidence internal node (b11), for example, it is determined that the optimal regular weighted discount value corresponding to the confidence leaf node b201 is greater than the optimal regular weighted discount value corresponding to the confidence leaf node b204. The best interaction strategy is determined according to the maximum optimal regular weighted discount value screened out from each node, and since the confidence root node corresponding to the action branch (a1 branch) has the maximum optimal regular weighted discount value in the embodiment of the present application, and the confidence internal node corresponding to the action branch (a1 branch) has the maximum optimal regular weighted discount value, it is determined that the best interaction strategy is to first drive according to the action space a1 to form the next state space, and then continue to drive according to the action space a1.
[0154] In the embodiment of the present application, since the best interaction strategy is determined according to the action space on the action branch corresponding to the maximum optimal regular weighted discount value, the self car can improve the safety and reliability of autonomous driving for passengers according to the best interaction strategy.
[0155] In the above embodiment, the autonomous guest driving car interaction decision method, device, system and medium are provided. The method is based on a state transition model and an observation model to construct a full sampling tree. The inverse dynamic programming method is used to solve from the confident leaf node to the confident root node of the full sampling tree to obtain the best interaction strategy. The best interaction strategy is used to indicate the operation of the ego vehicle, which can improve the safety and reliability of the autonomous guest driving car. The method comprises: modeling an interaction decision problem by using a partially observable Markov decision process to obtain a modeling result; constructing a full sampling tree according to the modeling result; wherein the full sampling tree comprises a plurality of paths, and the paths comprise a confident root node, a confident internal node and a confident leaf node arranged from top to bottom; and solving from the confident leaf node of the full sampling tree to the confident root node by using an inverse dynamic programming method to determine a best interaction strategy, wherein the best interaction strategy comprises an acceleration action, a maintenance action, a deceleration action or a parking action.
[0156] Further, as an implementation of the method shown in the above Figure 1 embodiments of the present application provide an autonomous guest driving car interaction decision device, Figure 9 An exemplary structure diagram of an autonomous guest driving car interaction decision device according to some embodiments is shown. As shown in Figure 9 The device comprises a modeling unit 901, a construction unit 902 and a solving unit 903.
[0157] The modeling unit is configured to model an interaction decision problem by using a partially observable Markov decision process to obtain a modeling result.
[0158] The construction unit is configured to construct a full sampling tree according to the modeling result; wherein the full sampling tree comprises a plurality of paths, and the paths comprise a confident root node, a confident internal node and a confident leaf node arranged from top to bottom.
[0159] The solving unit is configured to solve from the confident leaf node of the full sampling tree to the confident root node by using an inverse dynamic programming method to determine a best interaction strategy, wherein the best interaction strategy comprises an acceleration action, a maintenance action, a deceleration action or a parking action.
[0160] In a specific application scenario, the modeling result comprises a state space, an observation space, an action space, a state transition model and an observation model; and the construction unit specifically performs:
[0161] The state space generated by using the real observation of the ego vehicle to the other vehicle is sampled multiple times to obtain multiple paths, the confidence leaf node is the deepest confidence internal node, and the confidence internal node is generated according to the state space, the observation space, the action space, the state transition model and the observation model.
[0162] In a specific application scenario, the construction unit specifically performs:
[0163] The confidence root node is taken as a target node, and a next state space is generated by using the state transition model according to the target node and the action space;
[0164] A target observation space is determined by using the observation model according to the next state space and the observation space;
[0165] The confidence internal node is determined according to the next state space and the target observation space, the confidence internal node is taken as a target node, and the step of generating a next state space by using the state transition model according to the target node and the action space is repeatedly performed.
[0166] In a specific application scenario, the solving unit includes:
[0167] A pruning unit is configured to perform a pruning operation on the full-sample tree to obtain a full-sample tree policy tree, and the confidence internal node in the full-sample tree policy tree only includes one action branch;
[0168] A first determination unit is configured to determine an optimal regular weighted discount value of the confidence leaf node by using the return function based on the full-sample tree, if the depth of the full-sample tree policy tree is not greater than a preset value;
[0169] A second determination unit is configured to determine an optimal regular weighted discount value of the confidence internal node according to the optimal regular weighted discount value of the confidence leaf node;
[0170] A third determination unit is configured to calculate an optimal regular weighted discount value of the confidence root node according to the optimal regular weighted discount value of the confidence internal node;
[0171] A fourth determination unit is configured to determine an optimal interaction strategy according to the optimal regular weighted discount value of the confidence root node, the optimal regular weighted discount value of the confidence internal node and the optimal regular weighted discount value of the confidence leaf node.
[0172] According to an embodiment of the present application, a storage medium is provided, and the storage medium stores at least one executable instruction. The computer executable instruction can execute the autonomous customer driving interaction decision method in any method embodiment described above.
[0173] Based on such understanding, the technical scheme of the present application can be embodied in the form of a software product, which can be stored in a nonvolatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in various implementation scenarios of the present application.
[0174] The embodiment of the present application also provides an electronic device for autonomous customer driving interaction decision, which can be a computer, a smart phone, a tablet computer, a smart watch, a server, or a network device, etc. The entity device includes a storage medium and a processor; the storage medium is used to store a computer program; and the processor is used to execute the computer program to realize the autonomous customer driving interaction decision method.
[0175] Optionally, the electronic device can also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface can include a display screen (Display), an input unit such as a keyboard (Keyboard), etc. The optional user interface can also include a USB interface, a card reader interface, etc. The network interface can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
[0176] Those skilled in the art can understand that the structure of the electronic device provided by the embodiment does not constitute a limitation on the electronic device, and can include more or fewer components, or combine certain components, or different component arrangements.
[0177] The storage medium can also include an operating device and a network communication module. The operating device is a program for managing and saving hardware and software resources of the electronic device, supporting the running of information processing programs and other software and / or programs. The network communication module is used to realize the communication between the controls inside the storage medium, and the communication with other hardware and software in the entity device.
[0178] The embodiment of the present application also provides an autonomous customer driving interaction decision system, which includes:
[0179] a processor; and
[0180] a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the steps of the autonomous customer driving interaction decision method.
[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software with a necessary general hardware platform, or by hardware.
[0182] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of a preferred embodiment scenario, and the units or flows in the drawings are not necessarily essential for implementing the present application. Those skilled in the art can understand that the units in the devices in the embodiment scenario can be distributed in the devices in the embodiment scenario according to the description of the embodiment scenario, or can be changed and located in one or more devices different from the embodiment scenario. The units of the above embodiment scenario can be combined into one unit, or can be further split into multiple sub-units.
[0183] The above serial numbers of the present application only serve for description, and do not represent the advantages and disadvantages of the embodiment scenario. The above disclosure is only several specific embodiment scenarios of the present application, but the present application is not limited thereto, and any changes that those skilled in the art can think of should fall within the protection scope of the present application.
Claims
1. An autonomous valet driving interactive decision-making method, characterized in that, include: The interactive decision-making problem is modeled using a partially observable Markov decision process, and the modeling results are obtained. Based on the modeling results, a full sampling tree is constructed; wherein, the full sampling tree includes multiple paths, and each path includes a confidence root node, confidence internal nodes, and confidence leaf nodes arranged from top to bottom; Starting from the confidence leaf node of the full sampling tree, an inverse dynamic programming method is used to solve the problem up to the confidence root node to determine the optimal interaction strategy, which includes acceleration, holding, deceleration or stopping actions.
2. The method according to claim 1, characterized in that, The modeling results include a state space, an observation space, an action space, a state transition model, and an observation model; the step of constructing a full sampling tree based on the modeling results includes: Multiple paths are obtained by sampling multiple times using the state space generated when the vehicle actually observes other vehicles as the confidence root node of the full sampling tree; wherein, the confidence leaf node is the deepest confidence internal node, and the confidence internal node is generated based on the state space, observation space, action space, state transition model and observation model.
3. The method according to claim 2, characterized in that, Using the state space generated when the vehicle actually observes another vehicle as the confidence root node of the full sampling tree, a single sampling is performed to obtain a path including: Using the confidence root node as the target node, and based on the target node and the action space, the state transition model is used to generate the next state space; Based on the next state space and the observation space, the target observation space is determined using the observation model; Based on the next state space and the target observation space, the confidence internal node is determined, and the confidence internal node is used as the target node. The step of generating the next state space based on the target node and the action space using the state transition model is repeated.
4. The method according to claim 1, characterized in that, The modeling results also include a reward function.
5. The method according to claim 4, characterized in that, The step of using inverse dynamic programming to solve for the optimal interaction strategy starting from the confidence leaf nodes of the full sampling tree includes: The full sampling tree is pruned to obtain multiple full sampling tree strategy trees, and the confidence internal nodes in the full sampling tree strategy trees contain only one action branch. If the depth of all the full sampling tree strategy trees is not greater than a preset value, then based on the full sampling tree, the optimal regularized weighted discount value of the confidence leaf node is determined using the reward function. The optimal regularized weighted discount value of the confidence internal node is determined based on the optimal regularized weighted discount value of the confidence leaf node. The optimal regularized weighted discount value of the confidence root node is calculated based on the optimal regularized weighted discount value of the confidence internal nodes. The optimal interaction strategy is determined based on the optimal regularized weighted discount value of the root node, the optimal regularized weighted discount value of the internal nodes, and the optimal regularized weighted discount value of the leaf nodes.
6. The method according to claim 2, characterized in that, The state space includes the longitudinal position and speed information of the vehicle itself, as well as the longitudinal position and speed information of other vehicles.
7. The method according to claim 2, characterized in that, The action space includes driving actions that the vehicle may take, as well as driving actions that other vehicles may take; wherein, the driving actions include acceleration, holding, deceleration and stopping.
8. An autonomous valet driving interactive decision-making device, characterized in that, include: The modeling unit is used to model interactive decision problems using partially observable Markov decision processes and obtain modeling results. A construction unit is used to construct a full sampling tree based on the modeling results; wherein the full sampling tree includes multiple paths, and the paths include a confidence root node, confidence internal nodes and confidence leaf nodes arranged from top to bottom; The solution unit is used to solve the problem from the confidence leaf node of the full sampling tree to the confidence root node using an inverse dynamic programming method to determine the optimal interaction strategy, which includes acceleration, holding, deceleration or stopping actions.
9. An autonomous valet driving interactive decision-making system, comprising: processor; as well as A memory configured to store computer-executable instructions, characterized in that, when executed, the executable instructions cause the processor to perform the steps of the autonomous valet driving interactive decision-making method according to any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the autonomous valet driving interactive decision-making method according to any one of claims 1-7.
Citation Information
Patent Citations
Unmanned vehicle blind region turning planning method based on partially observable Markov model
CN113291318A
Automatic driving automobile lane changing decision control method considering uncertainty
CN115257746A
Trajectory planning method combining behavior estimation and backing time domain control
CN118025222A
Vehicle trajectory control using a tree search
US20230041975A1
