Enhanced negotiation decision-making method based on spatiotemporal directed graph multi-modal feature extraction network

Abstract

The application proposes a multi-modal feature extraction network based on space-time directed graph to enhance negotiation decision-making method, aiming at enhancing the negotiation decision-making algorithm of autonomous vehicles. The method constructs three kinds of graph structures including perspective graph, distance graph and collision time graph, comprehensively captures and describes the complex dynamic and asymmetric interaction relationship between vehicles. Each graph structure extracts features through graph attention network (GAT), fully excavates the deep features of different interaction prior information. Then, the extracted multi-modal features are fused through multi-layer perceptron (MLP) to form a unified feature representation. This comprehensive representation is further input into the reinforcement learning module based on the Actor-Critic framework to optimize the driving strategy. Through this innovative architectural design, the application can significantly improve the decision-making ability and safety performance of the autonomous driving system in complex and dynamic traffic environments, ensuring efficient and stable autonomous driving of vehicles under high-density traffic conditions.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and specifically to an autonomous driving negotiation decision-making method based on a multimodal feature extraction network of a spatiotemporal directed graph and a reinforcement learning framework, which is used to enhance the decision-making ability and safety of the autonomous driving system in complex traffic environments. Background Art

[0002] In future road traffic, autonomous vehicles must be able to coexist with human-driven vehicles, forming a mixed traffic environment. In order for people to accept autonomous vehicles, they must exhibit behaviors that are consistent with human driving expectations and follow social norms. This means that autonomous vehicles need to consider the social expectations of surrounding vehicles through their decision-making algorithms.

[0003] However, traditional autonomous driving methods are primarily based on hand-crafted heuristic rules, such as finite state machines (FSMs). These methods are often only applicable to specific use cases and require a significant amount of manpower to maintain the rule base to ensure safety. However, as the number of rules increases, new problems may arise, such as how to resolve new situations without forgetting old ones, and how to balance cost functions in complex scenarios with conflicting objectives (such as safety and efficiency). As a result, rule-based methods often lead to unnatural driving behaviors or fail completely in unexpected corner cases.

[0004] To address these issues, learning-based methods have gradually become a viable and popular solution in recent years. These methods efficiently extract driving knowledge from human driving data through imitation learning (IL). Although this paradigm is attractive, its widespread application still faces several significant limitations, such as the high cost of collecting large amounts of human driving data and the fact that imitation learning methods are particularly sensitive to dataset bias. In contrast, deep reinforcement learning (DRL) acquires knowledge through trial and error through interaction with the environment and maximizes the sum of expected future rewards. This method does not require expert labeling, thus eliminating the dataset bias problem in imitation learning. Although DRL has shown great application prospects in the field of autonomous driving, relying solely on DRL for interactive negotiation and cooperative driving will significantly increase the complexity of the system and make it difficult to accurately identify and handle complex relationships between vehicles.

[0005] To address the problem of vehicle interaction modeling, graph neural networks (GNNs) were introduced to capture complex social behaviors by representing vehicle features as node feature matrices and the mutual influences between vehicles as adjacency matrices. In this context, graph convolutional networks (GCNs) and graph attention networks (GATs) have been widely used to learn graph-structured data. On this basis, the graph reinforcement learning (GRL) method has gradually emerged, combining graph neural networks with deep reinforcement learning to enhance the system's ability to learn and adapt to vehicle interaction behaviors. By modeling complex traffic scenes as graph structures, GRL can effectively capture the topological relationships and mutual influences between vehicles and perform policy optimization based on this. The application of GRL methods in the field of autonomous driving, including trajectory prediction, charging navigation, traffic signal control, and traffic flow prediction, has demonstrated significant advantages.

[0006] However, despite certain progress, existing GRL methods still face major challenges in accurately modeling the interactions between vehicles and maintaining comprehensive state perception: on the one hand, how to achieve effective feature aggregation in the graph module to enhance downstream tasks related to negotiation decision-making remains a major challenge; on the other hand, focusing solely on feature extraction of the graph module ignores the overall perception of environmental features and task-related information, severely limiting the system's ability to make wise decisions in complex dynamic scenarios. Summary of the Invention

[0007] The purpose of the present invention is to address the problem of limited decision-making ability and safety of autonomous driving systems in complex traffic environments in the existing technology, and to provide an autonomous driving negotiation decision-making method based on a multimodal feature extraction network of a spatiotemporal directed graph and a reinforcement learning framework.

[0008] The technical solution adopted to achieve the purpose of the present invention is:

[0009] A method for autonomous driving negotiation decision-making based on a multimodal feature extraction network and reinforcement learning framework using a spatiotemporal directed graph includes the following steps:

[0010] Step 1: Construct a spatiotemporal directed graph model STDG consisting of the visual field graph VG, the distance graph DG, and the collision time graph CG;

[0011] Step 2: Input the visual field map VG, distance map DG and collision time map CG obtained in step 1 into the graph attention network GAT for calculation. Through the processing of GAT, the visual field map features are generated. Distance map features and collision time graph features

[0012] Step 3: Use the multi-layer perceptron MLP to transform the and Fusion is performed to obtain the fused graph feature H f ;

[0013] Step 4: Multimodal feature fusion:

[0014] The vehicle status and navigation information are processed using MLP to obtain the vehicle status and navigation features H2. The LiDAR data is processed using CNN and the processed features are flattened into a one-dimensional vector H flatten , H2, H flatten And H obtained in step 3 f Perform serial fusion to generate comprehensive feature representation S′ t ;

[0015] Step 5: The comprehensive feature representation S′ obtained in step 4 is t Input AC framework for policy optimization. In AC framework, the policy network Actor is based on S′ t Generate a specific driving strategy π θ (s′ t ); the evaluation network Critic evaluates the decisions generated by the policy network and guides the optimization of the policy network Actor by calculating rewards. Through continuous training and optimization, the policy network gradually learns the optimal driving strategy.

[0016] In the above technical solution, in step 1, the visual field map is defined represents the node matrix of the visual field graph, Represents the adjacency matrix of the visual field. Vehicle i is represented as a node Vehicle j is represented as a node If the node Located at the node Within the viewing angle range, from vehicle j node point to There is an edge Otherwise there is no edge side The presence or absence of is determined by the following formula:

[0017]

[0018] in:

[0019]

[0020] α represents the node To Node The angle between Representation node The direction of movement, d ij is a node To Node The direction vector, α v is the vehicle's viewing angle, Represents the set of all nodes within the maximum detection range of the vehicle;

[0021] Defining the distance map represents the distance graph node matrix, Represents the distance graph adjacency matrix, with vehicle i and vehicle j represented as nodes and The side Representation node To Node The distance is expressed by the following formula:

[0022]

[0023] Defining the collision time graph represents the collision time graph node matrix, Represents the collision time graph adjacency matrix. If there is an intersection η between the movement directions of two vehicles, there is a potential collision risk between them. Calculated by the following formula:

[0024]

[0025] ∈i and ∈j represent the predicted collision time of vehicle i and vehicle j to the intersection η.

[0026] In the above technical solution, in step 2, the visual field map Distance Map and collision time diagram The GAT calculation formula is as follows:

[0027]

[0028] H l Represents the node feature matrix of the lth layer, l is 1 or 2. When l is 1, H l-1 Equal to H0, that is, the three types of graph data in the STDG model and When l is 2, H l Equal to H2, the characteristics of each generated graph: and σ is a nonlinear activation function, A ij represents the value in the adjacency matrix, represents the set of neighbor nodes of node i, W lis the weight matrix of the lth layer, α ij is the attention coefficient.

[0029] In the above technical solution, α ij is calculated as follows:

[0030]

[0031] Among them, a l is a learnable weight vector, || represents vector concatenation, h i , h j With h k They represent the feature vectors of vehicle i, vehicle j, and vehicle k respectively, and LeakyReLU is the activation function.

[0032] In the above technical solution, in step 3,

[0033] In the above technical solution, in step 4, the vehicle state and navigation feature H2 are processed through two consecutive dense layers, which can be mathematically expressed as follows:

[0034] H1=σ(W1x+b1),

[0035] H2=σ(W2H1+b2),

[0036] Where x represents the input ego vehicle state and navigation data, W1 and W2 are weight matrices, b1 and b2 are bias terms, σ represents the activation function, H1 is the output of the first dense layer, and H2 is the output of the second dense layer, i.e., the ego vehicle state and navigation features.

[0037] In the above technical solution, in step 4, H flatten =Flatten(H conv ).

[0038] In the above technical solution, in step 4, S' t =Concatenate(H2,H f , H flatten ).

[0039] In the above technical solution, for continuous control, the driving strategy π θ (s′ t ) can be updated by the following gradient ascent algorithm:

[0040] θ new ←θ now +β·g(s′ t , a t θ now )

[0041] Among them, θ represents the parameters of the policy model, and the subscripts new and now represent the updated and the updated respectively. is the expected return An unbiased estimate of , β represents the learning rate, represents the state-value function, a t Represents the action at time t.

[0042] In the above technical solution, the state-value function The ω in ω can be updated using the temporal difference (TD) learning method by the following formula:

[0043]

[0044] Among them, ω represents the parameters of the value function model, and the subscript ne w and no w Respectively represent after update and before update, is the TD error, is the estimated value of the state at time t, is the TD target, is the gradient of ω, r t represents the instantaneous reward at time t, the γ discount factor, between 0 and 1, used to weigh the impact of future rewards, and a t Represents the action at time t.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This paper proposes an innovative method architecture that integrates a multimodal feature extraction network containing graph information into the Actor-Criti c This integration can effectively handle the complexity of high-dimensional environmental information and continuous vehicle actions, thereby providing more accurate input features for autonomous driving decisions.

[0047] 2. This paper introduces a spatiotemporal directed graph model into the GAT network for the first time to capture the complex interactions between autonomous vehicles and human-driven vehicles. The spatiotemporal directed graph model uses the view graph, distance graph, and collision time to Figure 3 A directed graph topology accurately describes the dynamic and asymmetric interaction relationships between vehicles, thereby enhancing the negotiation decision-making strategy for autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a multimodal feature extraction network based on spatiotemporal directed graph to enhance the autonomous driving negotiation decision algorithm architecture diagram;

[0049] Figure 2is a visual graph modeling schematic diagram;

[0050] Figure 3 is a distance graph modeling schematic diagram;

[0051] Figure 4 is a collision time graph modeling visual graph;

[0052] Figure 5 is a multi-modal feature extraction layer network structure diagram. DETAILED DESCRIPTION

[0053] The application will be further described below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and should not be used to limit the application.

[0054] A multi-modal feature extraction network based on a spatio-temporal directed graph enhances the automatic driving negotiation decision-making method, including the following steps:

[0055] Step 1, constructing a spatio-temporal directed graph (STDG) model, including three kinds of interaction prior graph models: visual graph VG, distance graph DG and collision time graph CG;

[0056] The spatio-temporal directed graph (STDG) model is an effective graph representation technique for modeling complex social behaviors between vehicles. Unlike traditional graph representation methods, the application models the interaction relationship in the scene as a directed graph by encoding the information of the visual field, distance and collision time. The STDG model includes three graph topologies: visual graph VG, distance graph DG and collision time graph CG, which are used to depict the dynamic and asymmetric interaction relationship between vehicles.

[0057] The visual graph VG is a graph model constructed according to the visual angle range of the vehicle. Each node represents a vehicle, and the existence and weight of the edge depend on whether other vehicles are within the visual angle range of the vehicle. The visual graph can capture the attention area of the vehicle to the surrounding environment in the actual driving process.

[0058] As shown in Figure 2 , when a vehicle i is within the visual angle range of another vehicle j, it will affect the behavior of vehicle j, and conversely, if vehicle k is not within the visual angle range of vehicle j, it will not affect the behavior of vehicle j. The visual graph VG is defined as VG represents the visual graph node matrix, VG represents the visual graph adjacency matrix. Vehicle i is represented as node Vehicle j is represented as node Only when vehicle i is within the maximum detection range of vehicle j is it included in VG, and the node attributes include relative speed and position and have been normalized. If node is within the maximum detection range of node Within the viewing angle range, from vehicle j node point to There is an edge Otherwise there is no edge side The presence or absence of is determined by the following formula:

[0059]

[0060] in:

[0061]

[0062] α represents the node To Node The angle between Representation node The direction of movement, d ij is a node To Node The direction vector, α v is the vehicle's viewing angle, Represents the set of all nodes within the maximum detection range of the vehicle.

[0063] The distance graph (DG) is a graph model built based on the relative distances between vehicles. The edge weights of the graph are determined by the actual distance between vehicles; the closer the distance, the greater the edge weight. The distance graph effectively reflects the physical proximity between vehicles and their mutual influence.

[0064] The distance graph DG is defined based on the physical distance between vehicles, such as Figure 3 As shown. Define the distance map represents the distance graph node matrix, Represents the distance graph adjacency matrix, with vehicle i and vehicle j represented as nodes and The side Representation node To Node The distance is expressed by the following formula:

[0065]

[0066] The collision time graph (CG) is a graphical model that calculates the collision time between vehicles. The weight of an edge is determined by the time difference between two vehicles on their paths before a collision occurs. The shorter the time difference, the greater the edge weight. The collision time graph helps predict and avoid potential collision risks.

[0067] CG is used to describe the interaction between vehicles that may collide in crowded scenes, such as Figure 4 As shown. Define the collision time diagram represents the collision time graph node matrix, Represents the adjacency matrix of the collision time graph. If there is an intersection η in the movement directions of two vehicles, there is a potential collision risk between them. Calculated by the following formula:

[0068]

[0069] ∈ i and∈ j represents the predicted collision time between vehicle i and vehicle j at the intersection η.

[0070] Step 2: Input the visual field graph VG, distance graph DG and collision time graph CG obtained in step 1 into the graph attention network (GAT) for calculation. GAT can adaptively aggregate the features of each node and its neighboring nodes through the self-attention mechanism, thereby effectively extracting the complex relationship information in the graph. Through the processing of GAT, the feature representation of each graph is generated, which are (Visual field features), (distance map features) and (Collision time graph features).

[0071] Each figure ( and The GAT calculation formula of ) is as follows:

[0072]

[0073] Among them H l Represents the node feature matrix of the lth (1 or 2) layer. When l is 1, H l-1 Equal to H0, that is, three kinds of graph data in the STDG model: and When l is 2, H l Equal to H2, the characteristics of each generated graph: and σ is a nonlinear activation function, A ij represents the value in the adjacency matrix, represents the set of neighbor nodes of node i, W l is the weight matrix of the lth layer, α ij is the attention coefficient, calculated by the following formula:

[0074]

[0075] Among them, a l is a learnable weight vector, || represents vector concatenation, h i , h j With h kThey represent the feature vectors of vehicle i, vehicle j, and vehicle k respectively, and LeakyReLU is the activation function.

[0076] Step 3, through the multi-layer perceptron (MLP) (Visual field features), (distance map features) and The fusion of the collision time graph features forms a unified representation Hf. This fused feature representation integrates information from different graph models, helping to fully reflect the complex interactions between vehicles and improve the accuracy of subsequent strategy generation.

[0077] The process formula of graph fusion is as follows:

[0078]

[0079] Among them H f It is the fused graph feature, which combines the diverse information in each individual graph, thereby enhancing the model's ability to make better decisions using multi-angle information.

[0080] Step 4: Multimodal feature fusion: In addition to the fusion of graph features, multimodal information from other data sources must also be integrated. First, the vehicle state and navigation information are processed using MLP to obtain the vehicle state and navigation features H2; at the same time, the LiDAR data is processed using a convolutional neutral network (CNN) and the processed features are flattened into a one-dimensional vector H flatten Finally, the vehicle state processed by MLP and the navigation feature H2 and the fused graph feature H obtained in step 3 are combined. f and the flattened lidar feature H flatten Perform serial fusion to generate comprehensive feature representation S′ t , as the core input for subsequent decision-making.

[0081] First, the ego vehicle state and navigation data contains information about the vehicle itself and its planned route. This data is first processed through two consecutive fully connected (dense) layers, which extract meaningful features and facilitate higher-level representations. Mathematically expressed as follows:

[0082] H1=σ(W1x+b1),

[0083] H2=σ(W2H1+b2),

[0084] Where x represents the input ego vehicle state and navigation data, W1 and W2 are weight matrices, b1 and b2 are bias terms, σ represents the activation function, H1 is the output of the first dense layer, and H2 is the output of the second dense layer, i.e., the ego vehicle state and navigation features.

[0085] LiDAR data is processed through convolutional (Conv) layers to extract spatial features from the 2D point cloud data. Specifically, the LiDAR data passes through two convolutional layers to capture the spatial hierarchy in the input data. The feature map generated by the convolutional layers is then converted to a 1D feature vector through a Flatten layer, making the data compatible with integration with other features. This flattened feature vector is represented as:

[0086] H flatten =Flatten(H conv )

[0087] Among them H conv Represents the lidar features after processing by the convolution (Conv) layer.

[0088] For the above vehicle state and navigation features H2, the fused graph features H f and the flattened lidar feature H flatten Fusion is performed to obtain the fused comprehensive feature S′ t express:

[0089] S′ t =Concatenate(H2,H f , H flatten )

[0090] The final concatenation layer integrates the diverse extracted features from all data sources into a comprehensive feature vector, denoted as S′ t .

[0091] Step 5: Strategy optimization based on the Actor-Critic (AC) framework: The fused comprehensive feature representation S′ t Enter the AC framework for policy optimization. In the AC framework, the policy network (Actor) is based on S′ t Generate a specific driving strategy π θ (s′ t The Critic network evaluates the decisions generated by the Policy Network and guides its optimization by calculating rewards. Through continuous training and optimization, the Policy Network gradually learns the optimal driving strategy.

[0092] The STDG model and the multimodal feature extraction network are integrated into the Actor-Critic (AC) learning framework. The STDG model integrates three types of prior graph information, which are combined with the multimodal feature extraction network to form an enhanced state representation S′ t The Actor-Critic (AC) algorithm consists of two main components: the Actor that learns the policy and the Critic that learns the value function. θ represents the parameters of the policy model, ω represents the parameters of the value function model, and the subscripts new and now represent the updated and the updated respectively. For continuous control, the driving policy π θ (s′ t ) can be updated by the following gradient ascent algorithm:

[0093] θ new ←θ now +β·g(s′ t , a t θ now )

[0094] in, is the expected return An unbiased estimate of , β represents the learning rate, a t Represents the action at time t.

[0095] State-Value Function The ω in ω can be updated using the temporal difference (TD) learning method by the following formula:

[0096]

[0097] in, is the TD error, is the estimated value of the state at time t, is the TD target, is the gradient of ω, r t represents the instantaneous reward at time t, the γ discount factor, between 0 and 1, used to weigh the impact of future rewards, and a t Represents the action at time t.

[0098] This enhanced architecture based on AC achieves effective perception and decision optimization of complex environments by combining multimodal feature extraction and graph representation technology, thereby improving the decision quality of autonomous driving systems in highly dynamic scenarios. Real-time decision-making is performed on the driving environment at each moment. Specifically, the policy network generates the corresponding action a t ,Longitudinal action includes the vehicle’s driving and braking control signals, and lateral action includes the steering control signal.

[0099] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as falling within the protection scope of the present application.

Claims

1. A self-driving negotiation decision-making method based on a multimodal feature extraction network of a spatiotemporal directed graph and a reinforcement learning framework, characterized by: The following steps are involved: Step 1: Construct a spatiotemporal directed graph model STDG consisting of the visual field graph VG, the distance graph DG, and the collision time graph CG; Step 2: Input the visual field map VG, distance map DG and collision time map CG obtained in step 1 into the graph attention network GAT for calculation. Through the processing of GAT, the visual field map features are generated. Distance map features and collision time graph features Step 3: Use the multi-layer perceptron MLP to transform the and Fusion is performed to obtain the fused graph feature H f ; Step 4: Multimodal feature fusion: The vehicle status and navigation information are processed using MLP to obtain the vehicle status and navigation features H2. The LiDAR data is processed using CNN and the processed features are flattened into a one-dimensional vector H flatten , H2, H flatten And H obtained in step 3 f Perform serial fusion to generate comprehensive feature representation S t ′ ; Step 5: The comprehensive feature representation S obtained in step 4 is t ′ Input AC framework for policy optimization. In AC framework, the policy network Actor is based on S t ′ Generate a specific driving strategy π θ (s t ′ ); the evaluation network Critic evaluates the decisions generated by the policy network and guides the optimization of the policy network Actor by calculating rewards. Through continuous training and optimization, the policy network gradually learns the optimal driving strategy; In step 1, define the visual field map represents the node matrix of the visual field graph, Represents the view graph adjacency matrix, vehicle i is represented as a node Vehicle j is represented as a node If the node Located at the node Within the viewing angle range, from vehicle j node point to There is an edge Otherwise there is no edge side The presence or absence of is determined by the following formula: in: α represents the node To Node The angle between Representation node The direction of movement, d ij is a node To Node The direction vector, α v is the vehicle's viewing angle, Represents the set of all nodes within the maximum detection range of the vehicle; Defining the distance map represents the distance graph node matrix, Represents the distance graph adjacency matrix, with vehicle i and vehicle j represented as nodes and Among them Representation node To Node The distance is expressed by the following formula: Defining the collision time graph represents the collision time graph node matrix, Represents the collision time graph adjacency matrix. If there is an intersection η between the movement directions of two vehicles, there is a potential collision risk between them. Calculated by the following formula: ∈ i and∈ j represents the predicted collision time between vehicle i and vehicle j at the intersection η; In step 4, the vehicle state and navigation feature H2 are processed through two consecutive dense layers, which can be expressed mathematically as follows: H1=σ(W1x+b1), H2=σ(W2H1+b2), Where x represents the input ego vehicle state and navigation data, W1 and W2 are weight matrices, b1 and b2 are bias terms, σ represents the activation function, H1 is the output of the first dense layer, and H2 is the output of the second dense layer, i.e., the ego vehicle state and navigation features.

2. The autonomous driving negotiation decision-making method according to claim 1, wherein: In step 2, the visual field map Distance Map and collision time diagram The GAT calculation formula is as follows: b l Represents the node feature matrix of the lth layer, l is 1 or 2. When l is 1, H l-1 Equal to H0, that is, the three types of graph data in the STDG model and When l is 2, H l Equal to H2, the characteristics of each generated graph: and σ is a nonlinear activation function, A ij represents the value in the adjacency matrix, represents the set of neighbor nodes of node i, W l is the weight matrix of the lth layer, α ij is the attention coefficient.

3. The autonomous driving negotiation decision-making method according to claim 2, wherein: α ij is calculated as follows: Among them, a l is a learnable weight vector, || represents vector concatenation, h i , h j With h k They represent the feature vectors of vehicle i, vehicle j, and vehicle k respectively, and LeakyReLU is the activation function.

4. The autonomous driving negotiation decision-making method according to claim 1, wherein: In the step 3, 5. The autonomous driving negotiation decision-making method according to claim 1, wherein: In step 4, H flatten =Flatten(H conv ).

6. The autonomous driving negotiation decision-making method according to claim 1, wherein: In step 4, S' t =Concatenate(H2,H f ,H flatten ).

7. The autonomous driving negotiation decision-making method according to claim 1, wherein: For continuous control, the driving policy π θ (s′ t ) can be updated by the following gradient ascent algorithm: i new ←θ now +β·g(s′ t ,a t ;θ now ) Among them, θ represents the parameters of the policy model, and the subscripts new and now represent the updated and the updated respectively. is the expected return An unbiased estimate of , β represents the learning rate, represents the state-value function, a t Represents the action at time t.

8. The autonomous driving negotiation decision-making method according to claim 7, wherein: State-Value Function The ω in ω can be updated using the temporal difference (TD) learning method by the following formula: Among them, η represents the learning rate, ω represents the parameters of the value function model, and the subscripts new and now represent after and before the update respectively. is the TD error, is the estimated value of the state at time t, is the TD target, is the gradient of ω, t t represents the instantaneous reward at time t, the γ discount factor, between 0 and 1, used to weigh the impact of future rewards, and a t Represents the action at time t.

Images