Autonomous driving decision-making method and system considering occlusion uncertainty

Abstract

The present invention provides an automatic driving decision-making method and system considering occlusion uncertainty, including: obtaining self-vehicle state information and environment vehicle state information; wherein, a virtual vehicle is introduced into the environment vehicle corresponding to the self-vehicle; based on the obtained state information of the self-vehicle and the environment vehicle, a multi-frame fusion module is used to obtain the state embedding of the self-vehicle and the state embedding of the environment vehicle containing time information; and based on the state embedding of the self-vehicle and the state embedding of the environment vehicle, a key vehicle selection module based on a cross-attention mechanism is used to obtain global state features; based on the obtained global state features, a pre-trained reinforcement learning model based on value distribution is used to obtain vehicle output actions, so as to realize automatic driving decision of the self-vehicle.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle driving decision-making, and in particular relates to an automatic driving decision-making method and system taking occlusion uncertainty into consideration. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In urban environments, environmental uncertainty caused by limited perception due to occlusion is a major obstacle to autonomous driving decision-making and planning. As a typical "long-tail problem", it restricts the development of autonomous driving. Human drivers can assess the risk of collision by predicting the movement of traffic participants that may appear in the occluded area, and make reasonable decisions, such as slowing down in advance or moving forward tentatively. However, many autonomous driving decision-making and planning algorithms only adjust their own behavior based on the known state of the environment and vehicle, ignoring the collision risk caused by the occlusion of obstacles, and making dangerous decisions and plans, such as maintaining the original speed or accelerating, which inevitably leads to traffic accidents.

[0004] Although vehicle-to-vehicle or other infrastructure (V2X) communication can be used to provide information about traffic participants in obstructed shadow areas, it is currently difficult to build the corresponding infrastructure in a short period of time; in terms of data calculation, as the complexity of the environment increases, methods based on POMDP solvers usually require a lot of computing work, affecting the real-time performance and efficiency of the model; distributed reinforcement learning methods are effective in dealing with the uncertainty brought by blocked areas, but do not consider the impact of understanding surrounding information on the decision-making model. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an autonomous driving decision-making method and system taking into account occlusion uncertainty. The scheme is based on the consideration of occlusion scenarios, by setting a virtual vehicle at the occlusion boundary, proposing and adopting a multi-frame fusion module for feature extraction, and combining a key vehicle selection module based on a cross-attention mechanism to obtain global fusion features, which effectively deepens the understanding of the environment in autonomous driving decision-making, thereby ensuring the safety and efficiency of autonomous driving vehicles under the uncertainty caused by occlusion.

[0006] According to a first aspect of an embodiment of the present invention, there is provided an automatic driving decision-making method considering occlusion uncertainty, comprising:

[0007] Acquire the self-vehicle state information and the environment vehicle state information; wherein, introduce a virtual vehicle into the environment vehicle corresponding to the self-vehicle;

[0008] Based on the state information of the ego vehicle and the surrounding vehicles, a multi-frame fusion module is used to obtain the state embedding of the ego vehicle and the state embedding of the surrounding vehicles containing time information; and based on the state embedding of the ego vehicle and the state embedding of the surrounding vehicles, a key vehicle selection module based on the cross attention mechanism is used to obtain the global state feature;

[0009] Based on the obtained global state features, a pre-trained reinforcement learning model based on value distribution is used to obtain the vehicle output action and realize the autonomous driving decision of the ego vehicle.

[0010] Furthermore, the virtual vehicle is set at the occlusion boundary of the own vehicle, wherein the occlusion boundary is a boundary formed by the intersection of a ray connecting the position of the own vehicle and the boundary point of the occlusion object and the center line of the lane where the environment vehicle is located.

[0011] Furthermore, the self-vehicle state information is represented by the self-vehicle speed at several consecutive moments and the self-vehicle position in the self-vehicle coordinate system; the environmental vehicle state information is represented by the relative position of the environmental vehicle to the self-vehicle and the environmental vehicle speed at several consecutive moments corresponding to several environmental vehicles.

[0012] Furthermore, the multi-frame fusion module is used to obtain the state embedding of the ego vehicle and the state embedding of the environment vehicles containing time information, wherein the multi-frame fusion module specifically uses the state information of the ego vehicle and the environment vehicles as the input of the fully connected network, and uses the output of the fully connected network as the input of the GRU layer to obtain the state embedding of the ego vehicle and the environment vehicles.

[0013] Furthermore, the key vehicle selection module based on the cross-attention mechanism is used to obtain global state features, wherein the key vehicle selection module specifically uses the state embedding of the self-vehicle, the state embedding of the environment vehicles and the mask representation of the environment vehicles as the input of the first cross-attention layer, and uses the output of the first cross-attention layer as the input of the second cross-attention layer to obtain global state features.

[0014] Furthermore, the reinforcement learning model based on value distribution specifically adopts the FQF model, which includes a quantile proposal network and a quantile value network. Based on the obtained global state characteristics, the quantile proposal network is used to obtain the adjustable quantile score corresponding to each time step; based on the obtained adjustable quantile score, the quantile value network is used to fit the corresponding adjustable quantile value.

[0015] Furthermore, the training of the reinforcement learning model based on value distribution adopts the following reward function:

[0016] R t =r s +r comf +r eff +rtime

[0017] Among them, r s is the security bonus, r com f For comfort bonus, r eff is the efficiency reward, r time Reward for time.

[0018] According to a second aspect of an embodiment of the present invention, there is provided an automatic driving decision-making system considering occlusion uncertainty, comprising:

[0019] A data acquisition unit, which is used to acquire the state information of the vehicle and the state information of the surrounding vehicles; wherein a virtual vehicle is introduced into the surrounding vehicles corresponding to the vehicle;

[0020] A feature extraction unit is used to obtain the state embedding of the ego vehicle and the state embedding of the environment vehicles containing time information by using a multi-frame fusion module based on the state information of the ego vehicle and the environment vehicles; and to obtain the global state feature by using a key vehicle selection module based on a cross attention mechanism based on the state embedding of the ego vehicle and the environment vehicles;

[0021] The decision unit is used to obtain the vehicle output action based on the obtained global state features and adopt a pre-trained reinforcement learning model based on value distribution to realize the automatic driving decision of the vehicle.

[0022] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, an automatic driving decision-making method considering occlusion uncertainty is implemented.

[0023] According to a fourth aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the automatic driving decision-making method considering occlusion uncertainty is implemented.

[0024] One or more of the above technical solutions have the following beneficial effects:

[0025] The present invention provides an automatic driving decision-making method and system taking into account occlusion uncertainty. The scheme is based on the consideration of occlusion scenarios, by setting a virtual vehicle at the occlusion boundary, adopting a multi-frame fusion module to extract features, and combining a key vehicle selection module based on a cross-attention mechanism to obtain global fusion features, thereby effectively deepening the understanding of the environment in automatic driving decision-making, thereby ensuring the safety and efficiency of automatic driving vehicles under the uncertainty caused by occlusion.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 This is a flow chart of an automatic driving decision-making method considering occlusion uncertainty described in an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the training simulation environment described in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0032] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0033] Embodiment 1

[0034] The purpose of this embodiment is to provide an autonomous driving decision-making method that takes occlusion uncertainty into account.

[0035] An autonomous driving decision-making method considering occlusion uncertainty, comprising:

[0036] Step 1: Obtain the state information of the vehicle and the state information of the surrounding vehicles; wherein a virtual vehicle is introduced into the surrounding vehicles corresponding to the vehicle;

[0037] The self-vehicle state information is represented by state observation values ​​at several consecutive moments; the environment vehicle state information is represented by state observation values ​​corresponding to several environment vehicles at several consecutive moments;

[0038] Specifically, the state of the environment vehicle at T-steps (i.e., T observation time steps) is used for the decision of the current step. The vehicle state set and the set of environmental vehicle states The state space at time t is composed of {tT, t-T+1, ..., t} representing the observation values ​​of T consecutive moments. Specifically:

[0039] The state set of the vehicle at time t represents the observation value at time t. The position of the vehicle is always the origin of the vehicle reference system. ego represents the speed of the vehicle at time t.

[0040] Similarly, for the environment vehicle, its state set at time t is:

[0041]

[0042]

[0043]

[0044] Where, the subscript env represents the environment vehicle, i represents the i-th environment vehicle, and n represents the total number of environment vehicles. The state space characteristics of the environment vehicle include the relative position of the environment vehicle to the ego vehicle And its own speed

[0045]

[0046] The representation of occlusion is crucial for decision-making in environments with occlusion. For occluded vehicles that may exist in the environment, this embodiment sets a virtual vehicle at the occlusion boundary so that the vehicle can cope with the impact of potential occluding vehicles. In order to enable the model to distinguish between potential occluded environmental vehicles (i.e., the first type), observed environmental vehicles (i.e., the second type), and non-existent environmental vehicles (i.e., the third type), the solution described in this embodiment sets a corresponding mask for each environmental vehicle. i t . is a matrix consisting of 0 or 1. When the environment vehicle does not exist, Set to 0, otherwise Set to 1.

[0047] If all T steps of the surrounding vehicle are of the second type, the original input is retained. Set to 1.

[0048] If all instances of T in the environment vehicles are of the third type, then The features in are set to 0. Set to 0.

[0049] If, in T steps, some steps of the environment vehicle belong to the first type and others belong to the second type, then The features at the first type of step in are set to 0, and the rest remain unchanged. Set to 1.

[0050] If, in T steps, some steps of the environment vehicle belong to the first type and others belong to the third type, then the The features in are set to 0. Set to 0.

[0051] The occlusion boundary refers to the boundary formed by the intersection of the ray connecting the position of the vehicle and the boundary point of the occlusion object with the center line of the lane where the environment vehicle is located; the occlusion object refers to a building.

[0052] Step 2: Based on the state information of the ego vehicle and the surrounding vehicles, a multi-frame fusion module is used to obtain the state embedding of the ego vehicle and the state embedding of the surrounding vehicles containing time information; and based on the state embedding of the ego vehicle and the state embedding of the surrounding vehicles, a key vehicle selection module based on the cross attention mechanism is used to obtain the global state features;

[0053] Specifically, based on the state information of the ego vehicle and the surrounding vehicles, the Multi-frame Stacking Module (MFM) is used to aggregate the stacked observations along the time dimension to obtain a state embedding containing time information. The MFM process is shown as follows:

[0054]

[0055]

[0056] Among them, Φ(·) represents the GRU layer, φ(·) represents the fully connected network (FC), d represents the GRU hidden layer unit size. n represents the number of vehicles in the environment.

[0057] Subsequently, the Critical-vehicle Selection Module (CSM) integrates the state embeddings of different vehicles, which utilizes the cross-attention mechanism.

[0058] In the CSM module, the cross attention mechanism is stacked twice to select the environmental vehicles to focus on when making the current decision. Layer normalization and residual connections are applied between each module. The representation of a single CSM is as follows:

[0059]

[0060] Among them, CA(·) represents the cross attention layer. Sent to downstream modules as global state features.

[0061] The specific network architecture is as follows: Figure 1 shown.

[0062] Step 3: Based on the obtained global state features, a pre-trained value distribution-based reinforcement learning model is used to obtain the vehicle output action;

[0063] Specifically, the reinforcement learning model based on value distribution adopts the FQF (Fully Parameterized Quantile Function) model, which includes FPN and QVN; the vehicle output action is obtained by processing the obtained global state features through FPN (fraction proposal network) and QVN (quantile value network) in turn. FPN estimates N adjustable quantile scores at each time step, and fits the corresponding N adjustable quantile values ​​through QVN. Among them, FPN and QVN networks are from the paper: Yang D, Zhao L, Lin Z, et al. Fully parameterized quantile function for distributional reinforcement learning [J]. Advances in neural information processing systems, 2019, 32.

[0064] Get quantiles from FPN:

[0065] The QVN expected reward value (which is used to select actions) can be expressed as:

[0066]

[0067] Among them, τ i represents the i-th quantile taken from FPN, τ i-1 <τ i ,τ 0 =0,τ N =1, is the actual quantile function, approximated by QVN, ω 2 are network parameters.

[0068] Select action according to QVN: a t = argmaxQ ω2 (S t ,a).

[0069] In terms of network updates, 1-Wasserstein is used to measure the gap between the approximate quantile function and the actual quantile function, and then the gradient descent method is used to minimize it:

[0070]

[0071] in ω 1 is the parameter of FPN.

[0072] Based on the appropriate choice of probability, the quantile function is trained by combining quantile regression and Bellman update of the distribution of the optimized probability. i ′ and τ j The weighted TD error of ′ at time t is:

[0073]

[0074] Among them, Z t+1 is (x t+1 ,a t+1 )’s action value random variable, Z t is (x t ,a t ) is an action-value random variable.

[0075] The loss calculation formula of QVN is:

[0076]

[0077] in, are two quantile fractions τ i ′ and τ j ′The weighted TD error at time t, is the minimum quantile Huber regression loss of the threshold κ, wherein the minimum quantile Huber regression loss comes from the paper: Dabney W, Rowland M, Bellemare M, et al. Distributional reinforcement learning with quantile regression [C] / / Proceedings of the AAAI conference on artificial intelligence. 2018, 32 (1).

[0078] In one or more implementations, the vehicle obtains state information and reward information by interacting with the environment. Specifically, in order to meet driving safety, the solution described in this embodiment sets a safety reward.

[0079]

[0080] The magnitude of acceleration reflects the impact of the strategy on comfort. The greater the acceleration, the lower the comfort; the smaller the acceleration, the higher the comfort.

[0081] r comf =-|a|

[0082] Where a represents the acceleration of the vehicle.

[0083] Expected speeds are set based on efficiency to encourage vehicles to pass through intersections safely and efficiently within a reasonable speed range.

[0084]

[0085] Among them, v represents the current absolute speed of the vehicle, v exp represents the desired speed of the vehicle.

[0086] At the same time, in order to encourage the agent to make safe rather than conservative strategies, a maximum time is set for training and a timeout penalty is added:

[0087]

[0088] In summary, autonomous vehicles may encounter the following situations when passing through intersections: 1) crossing an intersection; 2) collision; 3) no collision but timeout.

[0089] Finally, the reward function is designed as:

[0090] R t =r s +r comf +r eff +r time

[0091] Among them, r s represents the security reward, r com f represents the comfort reward, r eff represents efficiency reward, r time Represents time reward.

[0092] It should be noted that the reward function here is used to obtain the reward of the vehicle at each step as part of the TD error and is used to update the QVN.

[0093] The state information S obtained at the next moment t+1 The processing is as in step 2. The decision network stores the information obtained from the environment (S t ,mask t ,a t ,R t ,S t+1 ,mask t+1, done) to the experience replay pool D. The network randomly selects a batch of experiences M from the experience replay pool D at a certain frequency to update the overall network.

[0094] By repeating the above process for a certain number of rounds until the network converges, a trained model is obtained.

[0095] Finally, based on the obtained global state features, the vehicle output action is obtained using the trained model.

[0096] Embodiment 2

[0097] The purpose of this embodiment is to provide an autonomous driving decision-making system that takes occlusion uncertainty into account.

[0098] An autonomous driving decision system considering occlusion uncertainty, comprising:

[0099] A data acquisition unit, which is used to acquire the state information of the vehicle and the state information of the surrounding vehicles; wherein a virtual vehicle is introduced into the surrounding vehicles corresponding to the vehicle;

[0100] A feature extraction unit is used to obtain the state embedding of the ego vehicle and the state embedding of the environment vehicles containing time information by using a multi-frame fusion module based on the state information of the ego vehicle and the environment vehicles; and to obtain the global state feature by using a key vehicle selection module based on a cross attention mechanism based on the state embedding of the ego vehicle and the environment vehicles;

[0101] The decision unit is used to obtain the vehicle output action based on the obtained global state features and adopt a pre-trained reinforcement learning model based on value distribution to realize the automatic driving decision of the vehicle.

[0102] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0103] In further embodiments, there is also provided:

[0104] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0105] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0106] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0107] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0108] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0109] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An autonomous driving decision-making method considering occlusion uncertainty, characterized in that: include: Obtaining the state information of the ego vehicle and the state information of the environment vehicle; wherein, a virtual vehicle is introduced into the environment vehicle corresponding to the ego vehicle; the virtual vehicle is set at the occlusion boundary of the ego vehicle, wherein the occlusion boundary is the boundary formed by the intersection of the ray connecting the ego vehicle position and the boundary point of the occlusion object and the center line of the lane where the environment vehicle is located; Based on the obtained state information of the ego vehicle and the surrounding vehicles, a multi-frame fusion module is used to obtain the state embedding of the ego vehicle and the surrounding vehicles containing time information; wherein the multi-frame fusion module specifically comprises: using the state information of the ego vehicle and the surrounding vehicles as the input of the fully connected network respectively, and using the output of the fully connected network as the input of the GRU layer to obtain the state embedding of the ego vehicle and the surrounding vehicles; Based on the state embedding of the ego vehicle and the state embedding of the surrounding vehicles, a key vehicle selection module based on the cross attention mechanism is adopted to obtain the global state feature; wherein the key vehicle selection module specifically comprises: using the state embedding of the ego vehicle, the state embedding of the surrounding vehicles and the mask representation of the surrounding vehicles as the input of the first cross attention layer, and using the output of the first cross attention layer as the input of the second cross attention layer to obtain the global state feature; Based on the obtained global state features, a pre-trained reinforcement learning model based on value distribution is used to obtain the vehicle output action and realize the automatic driving decision of the self-vehicle; the reinforcement learning model based on value distribution specifically adopts the FQF model, which includes a quantile proposal network and a quantile value network. Based on the obtained global state features, the quantile proposal network is used to obtain the adjustable quantile score corresponding to each time step; based on the obtained adjustable quantile score, the quantile value network is used to fit the corresponding adjustable quantile value.

2. The automatic driving decision-making method considering occlusion uncertainty according to claim 1, characterized in that: The self-vehicle state information is represented by the self-vehicle speed at several consecutive moments and the self-vehicle position in the self-vehicle coordinate system; the environmental vehicle state information is represented by the relative position of the environmental vehicles to the self-vehicle and the environmental vehicle speed at several consecutive moments corresponding to several environmental vehicles.

3. The automatic driving decision-making method considering occlusion uncertainty according to claim 1, characterized in that: The training of the reinforcement learning model based on value distribution adopts the following reward function: in, Reward for safety, Reward for comfort, Reward for efficiency, Reward for time.

4. An autonomous driving decision system considering occlusion uncertainty, characterized in that: include: A data acquisition unit, which is used to acquire the state information of the ego vehicle and the state information of the environment vehicle; wherein a virtual vehicle is introduced into the environment vehicle corresponding to the ego vehicle; the virtual vehicle is set at the occlusion boundary of the ego vehicle, wherein the occlusion boundary is the boundary formed by the intersection of the ray connecting the ego vehicle position and the boundary point of the occlusion object and the center line of the lane where the environment vehicle is located; A feature extraction unit, which is used to obtain the state embedding of the self-vehicle and the state embedding of the environment vehicles containing time information by using a multi-frame fusion module based on the obtained state information of the self-vehicle and the environment vehicles; wherein the multi-frame fusion module specifically: uses the state information of the self-vehicle and the environment vehicles as the input of the fully connected network respectively, and uses the output of the fully connected network as the input of the GRU layer to obtain the state embedding of the self-vehicle and the environment vehicles; Based on the state embedding of the ego vehicle and the state embedding of the surrounding vehicles, a key vehicle selection module based on the cross attention mechanism is adopted to obtain the global state feature; wherein the key vehicle selection module specifically comprises: using the state embedding of the ego vehicle, the state embedding of the surrounding vehicles and the mask representation of the surrounding vehicles as the input of the first cross attention layer, and using the output of the first cross attention layer as the input of the second cross attention layer to obtain the global state feature; A decision unit is used to obtain a vehicle output action based on the obtained global state features and adopt a pre-trained reinforcement learning model based on value distribution to realize the automatic driving decision of the self-vehicle; the reinforcement learning model based on value distribution specifically adopts an FQF model, which includes a quantile proposal network and a quantile value network. Based on the obtained global state features, the quantile proposal network is used to obtain an adjustable quantile score corresponding to each time step; based on the obtained adjustable quantile score, the quantile value network is used to fit the corresponding adjustable quantile value.

5. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, an automatic driving decision-making method taking into account occlusion uncertainty is implemented as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, an automatic driving decision-making method taking into account occlusion uncertainty is implemented as described in any one of claims 1-3.

Images