Autonomous driving decision-making method and system considering interactive perception strategy

Abstract

The application relates to an automatic driving decision-making method and system considering an interactive perception strategy and belongs to the technical field of automatic driving. The application obtains self-vehicle state information and environment vehicle state information; based on the obtained self-vehicle state information and environment vehicle state information, a single-vehicle feature embedding module is used to obtain environment vehicle relative state information based on a self-vehicle coordinate system; and based on the environment vehicle relative state information, a multi-vehicle feature aggregation module is used to obtain interactive context information containing key vehicle information; and the self-vehicle state information is spliced to obtain global state features; based on the obtained global state features, a pre-trained reinforcement learning model is used to obtain a vehicle's lateral and longitudinal output control quantity, so that automatic driving decision-making of the self-vehicle is realized. The application unifies environment vehicle features to the self-vehicle coordinate system; the attention is used to screen out environment vehicles with the strongest conflict relationship with the self-vehicle; and a reinforcement learning architecture is used to realize action control and state value evaluation to accurately guide decision-making.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an autonomous driving decision-making method and system that takes into account interactive perception strategies. Background Technology

[0002] In complex urban road interaction scenarios, such as left turns at traffic light-free intersections or merging of multiple vehicles, the core challenge for autonomous vehicles lies in a deep understanding of the spatiotemporal topological interactions within dynamic traffic flows. Most existing autonomous driving decision-making algorithms employ a "full coverage" or simple feature stacking strategy, inputting all observed vehicle information into the decision model equally, ignoring the differences in the threat levels posed by different targets to the vehicle. This static aggregation approach cannot model the bidirectional, non-linear interactions between the vehicle and surrounding vehicles, making it difficult for the model to extract genuine conflict features from massive amounts of redundant information. Furthermore, it suffers from computational efficiency bottlenecks when handling variable-length interaction targets, resulting in poor robustness of the decision-making system in complex conflict environments, making it difficult to make decisions that balance safety and efficiency like human drivers.

[0003] Current autonomous driving interaction decision-making technologies face clear technical bottlenecks under complex conditions: existing algorithms mostly use the global environment state directly as input, lacking vehicle coordinate system transformation, which makes it impossible for the model to extract relative pose features with rotation and translation invariance, resulting in weak generalization ability; mainstream models mostly use average pooling or simple feature concatenation in the feature aggregation stage, lacking a weight allocation mechanism based on vehicle intent guidance, which makes the system unable to actively filter key conflict targets from dynamic traffic flow, and core interaction information is easily drowned out by background noise; when processing variable-length target sequences, existing technologies often rely on high-latency recurrent neural networks or redundant fixed-padding architectures, lacking mask-based feature blocking technology, making it difficult to solve the problem of unstable decision probability distribution caused by drastic fluctuations in the number of perceived targets while ensuring millisecond-level inference real-time performance. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this invention provides an autonomous driving decision-making method and system that considers interactive perception strategies. This invention unifies environmental features to the vehicle's central coordinate system and extracts relative pose features with rotation and translation invariance. Subsequently, an attention encoding network is used to filter out the environmental vehicles with the strongest conflict relationship with the vehicle, and masking techniques are combined to eliminate invalid observation interference, achieving accurate screening of core conflict targets. Finally, an actor-critic reinforcement learning architecture is used to achieve horizontally and vertically decoupled action control and state value assessment, effectively solving the technical bottlenecks of difficult identification of key risk targets and poor real-time decision-making in complex multi-vehicle interaction scenarios.

[0005] In a first aspect, the present invention provides an autonomous driving decision-making method considering an interactive perception strategy, comprising: The system collects real-time vehicle status information and all environmental vehicle status information for decision-making in the current step. The content of the vehicle status information and environmental vehicle status information includes the position, speed and heading angle of each vehicle at each moment. Based on the obtained vehicle state information and environmental vehicle state information, global state features are obtained by using a single-vehicle feature embedding module and a multi-vehicle feature aggregation module. Based on the obtained global state features, a pre-trained reinforcement learning model is used to obtain the lateral and longitudinal output control quantities of the vehicle, thereby realizing the autonomous driving decision-making of the vehicle. The lateral and longitudinal output control quantities include discrete longitudinal acceleration actions and discrete steering angle actions, which are combined to obtain the joint action strategy at each time step.

[0006] Furthermore, the vehicle status information at time t The included states are: the vehicle's position in the absolute coordinate system at time t. The speed of the vehicle at time t And the vehicle's heading angle at time t ; Environmental vehicle status information at time t The included states are: the position of vehicle i in the absolute coordinate system at time t. The speed of vehicle i in the environment at time t And the heading angle of vehicle i in the environment at time t. ; The complete set of environmental vehicle state information at time t is:

[0007] n represents the total number of environmental vehicles.

[0008] Furthermore, based on the obtained vehicle state information and environmental vehicle state information, a single-vehicle feature embedding module is used to obtain the relative state information of the environmental vehicles in the vehicle coordinate system, and environmental vehicle state embedding is obtained from the relative state information of the environmental vehicles; and based on the environmental vehicle state embedding, a multi-vehicle feature aggregation module is used to obtain interaction context information containing key vehicle information; the interaction context information and the vehicle state information are concatenated to obtain global state features.

[0009] Furthermore, based on the obtained vehicle state information and environmental vehicle state information, a single-vehicle feature embedding module is used to obtain the relative state information of the vehicle and the environment in the vehicle coordinate system, and the environmental vehicle state embedding is obtained from the relative state information of the environment, including: Based on the vehicle's status information and the environmental vehicle status information, the environmental vehicle status information is transformed into relative environmental vehicle status information based on the vehicle's coordinate system, as follows: ; in, The formula represents the coordinate transformation formula, which converts the environmental vehicle state information into the vehicle coordinate system. The vehicle coordinate system takes the vehicle position as the origin, the vehicle orientation as the positive x-axis, and the vehicle orientation to the left as the positive y-axis.

[0010] The single-vehicle feature extraction module is constructed using an MLP network, and the relative state information of the environment vehicles based on the vehicle's coordinate system is input into the single-vehicle feature extraction module to obtain the state embedding of each environment vehicle.

[0011] Furthermore, each environmental vehicle is assigned a corresponding mask, which is a matrix of 0s and 1s; when environmental vehicle i is not observable by the autonomous vehicle, the mask value in the mask is... Set to 0, otherwise Set to 1; By using masks, attention encoding is performed on observable environmental vehicles to obtain interactive context information containing key vehicle information.

[0012] Furthermore, based on the obtained global state features, a pre-trained reinforcement learning model is used to obtain the vehicle's lateral and longitudinal output control quantities, enabling autonomous driving decisions for the vehicle, including: The obtained global state features are processed through a policy network to extract low-dimensional latent feature representations for each decision time step t. Based on this low-dimensional latent feature representation, the probability distribution of discrete longitudinal acceleration actions is output via the first action head network. Based on this low-dimensional latent feature representation, the probability distribution of discrete steering angle actions is output via the second action head network; The conditional probability distribution of the joint action strategy is obtained from the probability distributions of the discrete longitudinal acceleration action and the discrete steering angle action. During the inference phase, sampling is performed based on the conditional probability distribution of the aforementioned joint action strategy, or the action with the highest probability is selected as the output control action.

[0013] Furthermore, based on the same global state features It is then fed into the value network of the reinforcement learning model. The state value at the corresponding time step is obtained by fitting: ,in, For value network parameters; Calculate the immediate reward for the trajectory determined by the output control action. And use instant rewards to construct discounted returns: ;in, Let k be the discount factor, and k be the k-th future step away from the current time t. The advantage function estimate is defined by combining state value and discounted return as follows: ; Calculate the probability ratio of the parameters of the new and old strategies; ;in, For the new strategy parameters, These are parameters for the old strategy; Obtain the truncated PPO agent objective using probability ratios and advantage function estimation: Obtain the truncated PPO agent objective using probability ratios and advantage function estimation: ; in, >0 indicates a truncation hyperparameter; Combining the PPO agent objective, the value function fitting error, and the policy entropy regularization term, the total loss is obtained: ; in, These are weighting coefficients. The value function fitting error, For policy entropy regularization, Let be the entropy of the policy distribution.

[0014] By minimizing the total loss using gradient descent, the policy network, action head network, and value network are jointly updated, thereby improving the performance of the vehicle control policy while ensuring the stability of policy updates.

[0015] Furthermore, instant rewards It consists of a weighted sum of the termination event reward and the process shape. ; The reward items for terminating the event are as follows: ; in, This indicates a collision with another vehicle; The intermediate rewards for termination events that do not distinguish between collision types are as follows: ; in, This is an indicator of whether a collision has occurred; 1 indicates that a collision has occurred. This is an indicator of whether the target area has been reached; 1 indicates that it has been reached. This is an indicator of whether this step is the final step; 1 indicates that it is the final step. Process shape item For targeted rewards Linear and upper-bounded bonus ,distance difference from heading Fractional decay positive reward term sum: .

[0016] Furthermore, the targeted rewards as follows: ; in, , Representing time respectively The horizontal and vertical coordinates of the vehicle's actual trajectory in the global coordinate system Indicates the agent's current heading angle. For the previous moment The heading angle; Additional bonuses that are linear and truncated at the upper bound: ; in, , Indicates time The horizontal coordinates given by the reinforcement learning agent in the global coordinate system. To enhance the distance between the decision trajectory and the real trajectory of the learning agent; Regarding distance difference from heading The fractional decaying positive reward term is as follows: ; in, They are respectively about distance difference from heading The fractional decay positive reward term.

[0017] In a second aspect, the present invention provides an apparatus for an autonomous driving decision-making method that considers interactive perception strategies, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit storing a computer program, and the computer program being executed by the processing unit to implement the autonomous driving decision-making method that considers interactive perception strategies.

[0018] A third aspect of the present invention provides an autonomous driving decision-making system considering an interactive perception strategy, comprising: The data acquisition unit is used to acquire vehicle status information and environmental vehicle status information. The feature extraction unit, based on the obtained vehicle state information and environmental vehicle state information, uses a single-vehicle feature embedding module and a multi-vehicle feature aggregation module to obtain global state features. The decision-making unit, based on the obtained global state features, uses a pre-trained reinforcement learning model to obtain the lateral and longitudinal output control quantities of the vehicle, thereby realizing the autonomous driving decision-making of the vehicle. The lateral and longitudinal output control quantities include discrete longitudinal acceleration actions and discrete steering angle actions, which are combined to obtain the joint action strategy at each time step.

[0019] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art: This invention provides an autonomous driving decision-making method and system that considers interactive perception strategies. Based on the consideration of left-turn scenarios at unprotected intersections, the scheme extracts information from both the driver vehicle and surrounding vehicles, proposes and employs a single-vehicle feature embedding module for feature extraction, and combines this with a proposed multi-vehicle feature aggregation module to obtain global state features. During this process, environmental features are unified to the driver vehicle's central coordinate system, and relative pose features with rotation and translation invariance are extracted. Subsequently, an attention encoding network is used to filter out the environmental vehicles with the strongest conflict relationship with the driver vehicle, and masking techniques are used to eliminate invalid observation interference, achieving accurate screening of core conflict targets. Finally, an actor-critic reinforcement learning architecture is used to achieve horizontally and vertically decoupled action control and state value assessment, effectively deepening the understanding of the environment in autonomous driving decision-making, thereby ensuring the safety and efficiency of autonomous vehicles in left-turn scenarios at unprotected intersections. This effectively solves the technical bottlenecks of difficult identification of key risk targets and poor real-time decision-making in complex multi-vehicle interaction scenarios. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0022] Figure 1 A flowchart of an autonomous driving decision-making method considering interactive perception strategies provided in an embodiment of the present invention; Figure 2 A flowchart for obtaining global state features provided in an embodiment of the present invention; Figure 3 A schematic diagram of a network for extracting global state features provided in an embodiment of the present invention; Figure 4 A schematic diagram of an autonomous driving decision-making system considering interactive perception strategies, provided as an embodiment of the present invention; Figure 5 This is a schematic diagram of an autonomous driving decision-making device that considers interactive perception strategies, provided as an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0025] Example 1 like Figure 1 As shown, the purpose of this embodiment is to provide an autonomous driving decision-making method that considers interactive perception strategies, including: Step 1: Obtain real-time vehicle status information and environmental vehicle status information; The vehicle status information is represented by the vehicle status observations; the environmental vehicle status information is represented by the environmental vehicle status observations corresponding to several environmental vehicles at the current moment.

[0026] Specifically, the vehicle's state information and the environmental vehicle state information at time t form the following state set: ,in, This is the vehicle's status information; the subscript ego represents the vehicle. This is a set of environmental vehicle state information, where the subscript `env` represents the environmental vehicle. Specifically, it contains the vehicle's state information at time `t`. The included states are: the vehicle's position in the absolute coordinate system at time t. The speed of the vehicle at time t And the vehicle's heading angle at time t Then the vehicle's state information at time t is represented as: .

[0027] Similarly, for a total of n environmental vehicles, the set of all environmental vehicle state information at time t is: ; For any environmental vehicle i among n environmental vehicles, its environmental vehicle state information at time t The included states are: the position of vehicle i in the absolute coordinate system at time t. The speed of vehicle i in the environment at time t And the heading angle of vehicle i in the environment at time t. The environmental vehicle state information of vehicle i at time t is represented as follows: .

[0028] Step 2: Based on the obtained vehicle state information and environmental vehicle state information, global state features are obtained by using a single-vehicle feature embedding module and a multi-vehicle feature aggregation module.

[0029] Specifically, based on the obtained vehicle state information and environmental vehicle state information, a single-vehicle feature embedding module is used to obtain the relative state information of the environmental vehicles in the vehicle coordinate system, and environmental vehicle state embedding is obtained from the relative state information of the environmental vehicles; and based on the environmental vehicle state embedding, a multi-vehicle feature aggregation module is used to obtain interaction context information containing key vehicle information; the interaction context information and the vehicle state information are concatenated to obtain global state features. For example... Figure 2 As shown, the process includes: Based on the obtained vehicle and environmental vehicle state information, the spatial features of the environmental vehicles relative to the vehicle at the current moment are extracted through coordinate transformation to obtain the relative state of the environmental vehicles. Specifically, based on the vehicle state information and the environmental vehicle state information, the environmental vehicle state information is converted into relative environmental vehicle state information based on the vehicle coordinate system through coordinate transformation, as follows: ; in, The formula represents the coordinate transformation formula, which converts the environmental vehicle state information into the vehicle coordinate system. The vehicle coordinate system takes the vehicle position as the origin, the vehicle orientation as the positive x-axis, and the vehicle orientation to the left as the positive y-axis.

[0030] The converted set of relative state information of environmental vehicles at time t integrates the relative state information of n environmental vehicles, and is represented as: ; The relative state information of any environmental vehicle i in the set of relative state information of environmental vehicles is represented as follows: ; Among them, the relative state information of the environment and the vehicle includes the relative state of the environment and the vehicle's parking space at time t. relative speed and relative heading angle .

[0031] The single-vehicle feature extraction module is constructed using an MLP network, and the relative state information of the environment vehicles based on the vehicle's coordinate system is input into the single-vehicle feature extraction module to obtain the state embedding of each environment vehicle: ; in, This represents a multilayer perceptron for the single-vehicle feature extraction module. , where n represents the number of vehicles in the environment, and d represents the size of the hidden layer unit of the multilayer perceptron in the single vehicle feature extraction module.

[0032] Subsequently, the multi-vehicle feature aggregation module based on the attention coding network integrates the state embeddings of different vehicles to obtain interactive context information containing key vehicle information.

[0033] In its specific implementation, this application designs a mask based on the observability of environmental vehicles to the vehicle itself. In the multi-vehicle feature aggregation module, vehicle observability is crucial for feature aggregation. To enable the model to distinguish between observable environmental vehicles (i.e., the first type) and unobservable environmental vehicles (i.e., the second type), this invention sets a corresponding mask for each environmental vehicle. The mask is a matrix composed of 0s and 1s; when environmental vehicle i is unobservable by the vehicle, the mask value in the mask... Set to 0, otherwise Set to 1. If any environmental vehicle i is of type 1 in all decision steps, then in each decision step, the environmental vehicle i's Set to 1. If all vehicles i in the environment are of type 2, Setting it to 0, the mask allows the environmental vehicle state information of environment vehicle i to be displayed. The feature in is set to 0.

[0034] Attention coding is performed on observable environmental vehicles to obtain interactive context information containing key vehicle information.

[0035] The multi-vehicle feature aggregation module is represented as follows: ; in, Indicates a masked network. This represents an attention encoding network. ; The global state feature is obtained by concatenating the interaction context information and the vehicle state information:

[0036] in, Indicates feature splicing, This information is sent as a global state feature to downstream modules. This application utilizes an attention-encoding network to filter out environmental vehicles with the strongest conflict relationship with the vehicle itself, and combines this with masking techniques to eliminate invalid observation interference, thereby achieving accurate screening of core conflict targets.

[0037] The specific implementation of the network architecture in step 2 is as follows: Figure 3 As shown.

[0038] Step 3: Based on the obtained global state features, a pre-trained reinforcement learning model is used to obtain the vehicle's lateral and longitudinal output control quantities. The lateral and longitudinal output control quantities include discrete longitudinal acceleration actions and discrete steering angle actions, which are combined to obtain the joint action strategy at each time step.

[0039] Specifically, the reinforcement learning model adopts an actor-evaluator architecture.

[0040] The obtained global state features are processed through a policy network to extract a low-dimensional latent feature representation for each decision time step t. Based on this low-dimensional latent feature representation, the probability distribution of the longitudinal acceleration discrete action is output through the first action head network, as follows: ; in, The global state characteristics at decision time step t, It is the discrete action of longitudinal acceleration at decision time step t. This represents the first action head network of the policy network. Let be the probability distribution of the discrete longitudinal acceleration action at decision time step t; Simultaneously, based on this low-dimensional latent feature representation, the probability distribution of discrete steering angle actions is output via the second action head network. ,as follows: ; in, It is the discrete action of the steering angle at decision time step t. The second action head network represents the policy network. Let be the probability distribution of the discrete longitudinal acceleration action at decision time step t; This yields the joint action strategy for each time step.

[0041] From the probability distributions of the longitudinal acceleration discrete action and the steering angle discrete action at decision time step t, the conditional probability distribution of the joint action strategy is obtained:

[0042] These are the policy network parameters.

[0043] During the inference phase, sampling or selecting the action with the highest probability is performed based on the conditional probability distribution of the aforementioned joint action strategy, such as: ; To obtain the output control action of the vehicle at that time step.

[0044] At the same time, based on the same global state features It is then fed into the value network of the reinforcement learning model. The state value at the corresponding time step is obtained by fitting: , in These are the parameters of the value network.

[0045] Calculate the immediate reward for the trajectory determined by the output control action. And use instant rewards to construct discounted returns:

[0046] in, Let t be the discount factor, and k be the k-th future step away from the current time t.

[0047] In one or more embodiments, the vehicle obtains status information and reward information by interacting with the environment. Specifically, in order to meet driving safety requirements, this embodiment sets up safety rewards.

[0048] This embodiment provides a reward function configuration strategy for an unprotected left-turn replay environment. This strategy guides a reinforcement learning model to closely approximate the real vehicle trajectory and safely complete the left turn while maintaining the environmental dynamics. The reward function configuration strategy outputs a scalar instantaneous reward at each decision time t. Instant rewards It consists of a weighted sum of the termination event reward and the process shape.

[0049] First, based on the current environmental state, the following intermediate variable is constructed: the Euclidean distance between the vehicle's actual trajectory position and the reinforcement learning agent's decision position. The deviation between the vehicle's actual heading and the agent's decided heading The indicator of whether a collision has occurred. The indicator showing whether the target area has been reached (left turn completed). And the indicator quantity indicating whether this step is the termination step. .

[0050] At the same time, based on the collision detection results, the collision type is encoded as ,in Indicates no collision. This indicates a collision with the lane boundary. This indicates a collision with another vehicle.

[0051] For clarity, this embodiment first defines an intermediate reward for termination events that does not distinguish between collision types based on intermediate variables of collision detection. as follows:

[0052] Termination event reward items that take into account the differences in different collision types for: ; That is, when a car-to-car collision occurs, the reward for terminating the event will be set to zero. When a lane boundary collision occurs ( Large negative penalties are retained during the process to allow for more targeted handling of vehicle-to-vehicle collision risks in subsequent safety modules.

[0053] The introduced process shape item This is used to provide fine-grained guidance for the intermediate behavior of an agent in non-terminating steps. Specifically, it includes the following aspects: When the actual trajectory of the vehicle is located in a key area of ​​the intersection (e.g., meeting the following conditions) and When, based on the heading change direction of the reinforcement learning agent in the current step and the previous step. Provide targeted rewards, the targeted rewards as follows: ; in, , Representing time respectively The horizontal and vertical coordinates of the vehicle's actual trajectory in the global coordinate system Indicates the agent's current heading angle. For the previous moment The heading angle. Through the above design, a positive reward is given when the heading change of the reinforcement learning agent conforms to the expected left turn direction, and a negative reward is given otherwise, thereby encouraging the agent to drive in the reasonable turning direction within the intersection.

[0054] When the vehicle's actual trajectory enters the middle section of the intersection (e.g., meeting the following conditions) and Furthermore, the distance between the decision trajectory of the reinforcement learning agent and the real trajectory is... close enough (e.g.) When, according to the reinforcement learning agent in Position in the axial direction Construct a linear, upper-bounded bonus: ; in, , Indicates time The reinforcement learning agent provides its lateral coordinates in the global coordinate system. This design allows the agent to achieve a higher reward by moving towards the target direction in the middle section of the intersection.

[0055] Globally, the process shape term also includes information about distance. difference from heading The fractional decaying positive reward term is as follows: ; in, The smaller The smaller the value, the larger the values ​​of the above two items are, and the values ​​decrease smoothly as the distance and attitude deviation increase, thereby encouraging the agent to stay close to the real vehicle position and maintain attitude alignment throughout the trajectory.

[0056] Based on the above, this embodiment will include the process shape item. Defined as the sum of its parts: ; In summary, in this embodiment, instant rewards... To terminate the event reward item With process shape item The sum is: .

[0057] It should be noted that the reward function here is used to obtain the reward for each step of the vehicle, which is used as part of the TD error to update the network parameters.

[0058] Through the above reward design, without changing the physical environment and replay mechanism, the reinforcement learning agent can be effectively guided to learn a safe and realistic unprotected left-turn strategy.

[0059] The advantage function estimate is defined by combining state value and discounted return as follows: ; Regarding strategy updates, old strategy parameters are introduced. Calculate the probability ratio: ; Obtain the truncated PPO agent objective using probability ratios and advantage function estimation: ; in, >0 indicates a truncation hyperparameter; Combining the PPO agent objective, the value function fitting error, and the policy entropy regularization term, the total loss is obtained:

[0060] in, These are weighting coefficients. The value function fitting error, For policy entropy regularization, Let be the entropy of the policy distribution.

[0061] By minimizing the total loss using gradient descent, the policy network, action head network, and value network are jointly updated, thereby improving the performance of the vehicle control strategy while ensuring the stability of policy updates. This invention achieves horizontally and vertically decoupled action control and state value assessment through an actor-commentator reinforcement learning architecture, effectively deepening the understanding of the environment in autonomous driving decision-making. This ensures the safety and efficiency of autonomous vehicles turning left at unprotected intersections, effectively solving the technical bottlenecks of difficult identification of key risk targets and poor real-time decision-making in complex multi-vehicle interaction scenarios.

[0062] Example 2 The purpose of this embodiment is to provide an autonomous driving decision-making system that considers interactive perception strategies, such as... Figure 4 As shown, it includes: The data acquisition unit is used to acquire vehicle status information and environmental vehicle status information. The feature extraction unit, based on the obtained vehicle state information and environmental vehicle state information, uses a single-vehicle feature embedding module to obtain environmental vehicle relative state information based on the vehicle coordinate system; and uses a multi-vehicle feature aggregation module based on the environmental vehicle relative state information to obtain interaction context information containing key vehicle information; and concatenates it with the vehicle state information to obtain global state features. The decision-making unit, based on the obtained global state features, uses a pre-trained reinforcement learning model to obtain the lateral and longitudinal output control quantities of the vehicle, thereby realizing the autonomous driving decision-making of the vehicle. It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0063] In more embodiments, such as Figure 5 As shown, an apparatus for an autonomous driving decision-making method considering an interaction perception strategy is also provided, comprising: at least one processing unit, the processing unit being connected to a storage unit via a bus unit, the storage unit storing a computer program, and the computer program being executed by the processing unit to implement the autonomous driving decision-making method considering an interaction perception strategy. For simplicity, further details are omitted here.

[0064] It should be understood that in this embodiment, the processing unit can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The storage unit may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the storage unit may also include non-volatile random access memory. For example, the storage unit may also store information about the device type.

[0066] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0067] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0068] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0071] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An autonomous driving decision-making method considering interactive perception strategies, characterized in that, include: Real-time collection of vehicle status information and all environmental and vehicle status information is used for decision-making in the current step; The content of the vehicle status information and the environmental vehicle status information includes the position, speed and heading angle of each vehicle at each time. Based on the obtained vehicle state information and environmental vehicle state information, global state features are obtained by using a single-vehicle feature embedding module and a multi-vehicle feature aggregation module. Based on the obtained global state features, a pre-trained reinforcement learning model is used to obtain the vehicle's lateral and longitudinal output control quantities, enabling autonomous driving decisions for the vehicle, including: The obtained global state features are processed through a policy network to extract a low-dimensional latent feature representation corresponding to each decision time step t. Based on this low-dimensional latent feature representation, the probability distribution of the longitudinal acceleration discrete action is output through a first action head network. Based on this low-dimensional latent feature representation, the probability distribution of the steering angle discrete action is output through a second action head network. The conditional probability distribution of the joint action policy is obtained from the probability distributions of the longitudinal acceleration discrete action and the steering angle discrete action. During the inference phase, sampling is performed based on the conditional probability distribution of the joint action policy, or the action with the highest probability is selected as the output control action. Based on the same global state features It is then fed into the value network of the reinforcement learning model. The state value at the corresponding time step is obtained by fitting: ,in, The parameters are the value network parameters; for the trajectory determined by the action, the immediate reward is calculated. And use instant rewards to construct discounted returns: ； in, Here, k is the discount factor, and k is the k-th future step away from the current time t; the instant reward It consists of a weighted sum of the termination event reward and the process shape. ； The reward items for terminating the event are as follows: ； in, This indicates a collision with another vehicle; The intermediate rewards for termination events that do not distinguish between collision types are as follows: ； in, This is an indicator of whether a collision has occurred; 1 indicates that a collision has occurred. This is an indicator of whether the target area has been reached; 1 indicates that it has been reached. This is an indicator of whether this step is the final step; 1 indicates that it is the final step. Process shape item For targeted rewards Linear and upper-bounded bonus ,distance difference from heading Fractional decay positive reward term sum: ； The advantage function estimate is defined by combining state value and discounted return as follows: ; Calculate the probability ratio of the parameters of the new and old strategies; ;in, For the new strategy parameters, These are parameters for the old strategy; Obtain the truncated PPO agent objective using probability ratios and advantage function estimation: ； in, >0 indicates a truncation hyperparameter; Combining the PPO agent objective, the value function fitting error, and the policy entropy regularization term, the total loss is obtained: ； in, These are weighting coefficients. The value function fitting error, For policy entropy regularization, The entropy of the policy distribution; By minimizing the total loss using gradient descent, the policy network, action head network, and value network are jointly updated, thereby improving the performance of the vehicle control policy while ensuring the stability of policy updates.

2. The autonomous driving decision-making method considering interactive perception strategies according to claim 1, characterized in that, Vehicle status information at time t The included states are: the vehicle's position in the absolute coordinate system at time t. The speed of the vehicle at time t And the vehicle's heading angle at time t ; Environmental vehicle status information at time t The included states are: the position of vehicle i in the absolute coordinate system at time t. The speed of vehicle i in the environment at time t And the heading angle of vehicle i in the environment at time t. ; The complete set of environmental vehicle state information at time t is: ； n represents the total number of environmental vehicles.

3. The autonomous driving decision-making method considering interactive perception strategies according to claim 1, characterized in that, Based on the obtained vehicle state information and environmental vehicle state information, a single-vehicle feature embedding module is used to obtain the relative state information of the environmental vehicles in the vehicle coordinate system, and the environmental vehicle state embedding is obtained from the relative state information of the environmental vehicles; and based on the environmental vehicle state embedding, a multi-vehicle feature aggregation module is used to obtain the interaction context information containing key vehicle information; the interaction context information and the vehicle state information are concatenated to obtain the global state features.

4. The autonomous driving decision-making method considering interactive perception strategies according to claim 3, characterized in that, Based on the obtained vehicle state information and environmental vehicle state information, a single-vehicle feature embedding module is used to obtain the relative environmental vehicle state information in the vehicle coordinate system, and the environmental vehicle state embedding is obtained from the relative environmental vehicle state information, including: Based on the vehicle's status information and the environmental vehicle status information, the environmental vehicle status information is transformed into relative environmental vehicle status information based on the vehicle's coordinate system, as follows: ； in, The coordinate transformation formula converts environmental vehicle state information into a vehicle coordinate system, wherein the vehicle coordinate system takes the vehicle position as the origin, the vehicle orientation as the positive x-axis direction, and the vehicle orientation to the left as the positive y-axis direction. A single-vehicle feature extraction module is constructed using an MLP network, and the relative state information of the environment vehicles based on the vehicle's coordinate system is input into the single-vehicle feature extraction module to obtain the state embedding of each environment vehicle.

5. The autonomous driving decision-making method considering interactive perception strategies according to claim 3, characterized in that, Each environmental vehicle is assigned a corresponding mask, which is a matrix of 0s and 1s; when environmental vehicle i is not observable by the autonomous vehicle, the mask value in the mask is... Set to 0, otherwise Set to 1; By using masks, attention encoding is performed on observable environmental vehicles to obtain interactive context information containing key vehicle information.

6. The autonomous driving decision-making method considering interactive perception strategies according to claim 1, characterized in that, The targeted reward as follows: ； in, , Representing time respectively The horizontal and vertical coordinates of the vehicle's actual trajectory in the global coordinate system Indicates the agent's current heading angle. For the previous moment The heading angle; The additional rewards for linearity and upper bound truncation are as follows: ； in, , Indicates time The horizontal coordinates given by the reinforcement learning agent in the global coordinate system. To enhance the distance between the decision trajectory and the real trajectory of the learning agent; Regarding distance difference from heading The fractional decaying positive reward term is as follows: ； in, They are respectively about distance difference from heading The fractional decay positive reward term.

7. An autonomous driving decision-making system considering interactive perception strategies, implementing the autonomous driving decision-making method considering interactive perception strategies as described in any one of claims 1-6, characterized in that, include: The data acquisition unit is used to acquire vehicle status information and environmental vehicle status information. The feature extraction unit, based on the obtained vehicle state information and environmental vehicle state information, uses a single-vehicle feature embedding module and a multi-vehicle feature aggregation module to obtain global state features. The decision-making unit, based on the obtained global state features, uses a pre-trained reinforcement learning model to obtain the vehicle's lateral and longitudinal output control quantities, realizing the autonomous driving decision-making of the vehicle, including: The obtained global state features are processed through a policy network to extract a low-dimensional latent feature representation corresponding to each decision time step t. Based on this low-dimensional latent feature representation, the probability distribution of the longitudinal acceleration discrete action is output through a first action head network. Based on this low-dimensional latent feature representation, the probability distribution of the steering angle discrete action is output through a second action head network. The conditional probability distribution of the joint action policy is obtained from the probability distributions of the longitudinal acceleration discrete action and the steering angle discrete action. During the inference phase, sampling is performed based on the conditional probability distribution of the joint action policy, or the action with the highest probability is selected as the output control action. Based on the same global state features It is then fed into the value network of the reinforcement learning model. The state value at the corresponding time step is obtained by fitting: ,in, The parameters are the value network parameters; for the trajectory determined by the action, the immediate reward is calculated. And use instant rewards to construct discounted returns: ； in, Here, k is the discount factor, and k is the k-th future step away from the current time t; the instant reward It consists of a weighted sum of the termination event reward and the process shape. ； The reward items for terminating the event are as follows: ； in, This indicates a collision with another vehicle; The intermediate rewards for termination events that do not distinguish between collision types are as follows: ； in, This is an indicator of whether a collision has occurred; 1 indicates that a collision has occurred. This is an indicator of whether the target area has been reached; 1 indicates that it has been reached. This is an indicator of whether this step is the final step; 1 indicates that it is the final step. Process shape item For targeted rewards Linear and upper-bounded bonus ,distance difference from heading Fractional decay positive reward term sum: ； The advantage function estimate is defined by combining state value and discounted return as follows: ; Calculate the probability ratio of the parameters of the new and old strategies; ;in, For the new strategy parameters, These are parameters for the old strategy; Obtain the truncated PPO agent objective using probability ratios and advantage function estimation: ； in, >0 indicates a truncation hyperparameter; Combining the PPO agent objective, the value function fitting error, and the policy entropy regularization term, the total loss is obtained: ； in, These are weighting coefficients. The value function fitting error, For policy entropy regularization, The entropy of the policy distribution; By minimizing the total loss using gradient descent, the policy network, action head network, and value network are jointly updated, thereby improving the performance of the vehicle control policy while ensuring the stability of policy updates.

Images