Multi-modal feature extraction network enhanced negotiation decision-making method based on space-time directed graph

Abstract

The invention provides a multi-modal feature extraction network enhanced negotiation decision-making method based on a space-time directed graph, and aims to enhance a negotiation decision-making algorithm of an automatic driving vehicle. According to the method, three graph structures including a view angle graph, a distance graph and a collision time graph are constructed to comprehensively capture and describe the complex dynamic and asymmetric interaction relation between vehicles. Each graph structure is subjected to feature extraction through a graph attention network (GAT), and deep features of different interactive prior information are fully mined. Then, the extracted multi-modal features are fused through a multi-layer perceptron (MLP), and unified feature representation is formed; the comprehensive representation is further input into a reinforcement learning module based on an Actor-Critic framework to be used for optimizing a driving strategy. Through the innovative architecture design, the decision-making ability and the safety performance of the automatic driving system can be remarkably improved in a complex and dynamic traffic environment, and it is ensured that efficient and stable autonomous driving of the vehicle is achieved under the high-density traffic condition.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and specifically to an autonomous driving negotiation decision-making method based on a multimodal feature extraction network of a spatiotemporal directed graph and a reinforcement learning framework, which is used to enhance the decision-making ability and safety of an autonomous driving system in a complex traffic environment. Background Art

[0002] In future road traffic, autonomous vehicles must be able to coexist with human-driven vehicles in a mixed traffic environment. In order for people to accept autonomous vehicles, they must behave in a manner consistent with human driving expectations and follow social norms, which means that autonomous vehicles need to consider the social expectations of surrounding vehicles through their decision-making algorithms.

[0003] However, traditional autonomous driving methods are mainly based on hand-designed heuristic rules, such as finite state machines (FSM), which are usually only applicable to specific usage scenarios and require a lot of manpower to maintain the rule base to ensure safety. However, as the number of rules increases, new problems may arise, such as how to solve new situations without forgetting old ones, and how to balance cost functions in complex scenarios with conflicting goals (such as safety and efficiency). As a result, rule-based methods often lead to unnatural driving behaviors or fail completely in unexpected extreme situations.

[0004] To address these issues, learning-based methods have gradually become a viable and popular solution in recent years. These methods efficiently extract driving knowledge from human driving data through imitation learning (IL). Although this paradigm is attractive, its widespread application still faces several significant limitations, such as the high cost of collecting large amounts of human driving data and the fact that imitation learning methods are particularly sensitive to dataset bias. In contrast, deep reinforcement learning (DRL) acquires knowledge through trial and error through interaction with the environment and maximizes the sum of expected future rewards. This method does not require expert annotation, thus eliminating the dataset bias problem in imitation learning. Although DRL has shown good application prospects in the field of autonomous driving, relying solely on DRL for interactive negotiation and cooperative driving will significantly increase the complexity of the system and make it difficult to accurately identify and handle complex relationships between vehicles.

[0005] In order to solve the problem of vehicle interaction modeling, graph neural networks (GNNs) were introduced to capture complex social behaviors by representing vehicle features as node feature matrices and representing the mutual influence between vehicles as adjacency matrices. In this context, graph convolutional networks (GCNs) and graph attention networks (GATs) are widely used in learning graph structured data. On this basis, the graph reinforcement learning (GRL) method has gradually emerged, combining graph neural networks with deep reinforcement learning to enhance the system's learning and adaptation capabilities to vehicle interaction behaviors. By modeling complex traffic scenes as graph structures, GRL can effectively capture the topological relationships and mutual influences between vehicles, and optimize strategies on this basis. The application of GRL methods in the field of autonomous driving includes trajectory prediction, charging navigation, traffic signal control, and traffic flow prediction, showing significant advantages.

[0006] However, despite certain progress, existing GRL methods still face major challenges in accurately modeling the interactions between vehicles and maintaining comprehensive state awareness: on the one hand, in the graph module, how to achieve effective feature aggregation to enhance downstream tasks related to negotiation decisions remains a major challenge; on the other hand, focusing only on feature extraction of the graph module ignores the overall perception of environmental features and task-related information, severely limiting the system's ability to make wise decisions in complex dynamic scenarios. Summary of the invention

[0007] The purpose of the present invention is to provide an autonomous driving negotiation decision-making method based on a multimodal feature extraction network of a spatiotemporal directed graph and a reinforcement learning framework to address the problem of limited decision-making ability and safety of autonomous driving systems in complex traffic environments in the prior art.

[0008] The technical solution adopted to achieve the purpose of the present invention is:

[0009] A method for autonomous driving negotiation decision-making based on a multimodal feature extraction network of a spatiotemporal directed graph and a reinforcement learning framework comprises the following steps:

[0010] Step 1, construct a spatiotemporal directed graph model STDG model consisting of a visual field graph VG, a distance graph DG, and a collision time graph CG;

[0011] Step 2: Input the visual field graph VG, distance graph DG and collision time graph CG obtained in step 1 into the graph attention network GAT for calculation. Through GAT processing, the visual field graph features are generated. Distance map features and collision time graph features

[0012] Step 3: Use the multi-layer perceptron MLP to transform the and Fusion is performed to obtain the fused graph feature H f ;

[0013] Step 4: Multimodal feature fusion:

[0014] The vehicle status and navigation information are processed using MLP to obtain the vehicle status and navigation features H2. The LiDAR data is processed using CNN and the processed features are flattened into a one-dimensional vector H flatten , H2, H flatten And H obtained in step 3 f Perform serial fusion to generate comprehensive feature representation S′ t ;

[0015] Step 5: The comprehensive feature representation S′ obtained in step 4 is t Input AC framework for policy optimization. In AC framework, the policy network Actor is based on S′ t Generate specific driving strategy π θ (s′ t ); the evaluation network Critic evaluates the decisions generated by the policy network and guides the optimization of the policy network Actor by calculating the rewards. Through continuous training and optimization, the policy network gradually learns the optimal driving strategy.

[0016] In the above technical solution, in step 1, the visual field map is defined represents the node matrix of the visual field graph, represents the adjacency matrix of the visual field graph. Vehicle i is represented as a node Vehicle j is represented as a node If the node Located at the node Within the visual range, from the vehicle j node point to There is an edge Otherwise there is no edge side The presence or absence of is determined by the following formula:

[0017]

[0018] in:

[0019]

[0020] α represents the node To Node The angle between Representation Node The moving direction, d ij Is a node To Node The direction vector, α v is the viewing angle of the vehicle, Represents the set of all nodes within the maximum detection range of the vehicle;

[0021] Defining the distance map represents the distance graph node matrix, Represents the distance graph adjacency matrix, with vehicle i and vehicle j represented as nodes and The side Representation Node To Node The distance is expressed by the following formula:

[0022]

[0023] Defining the collision time graph represents the collision time graph node matrix, Represents the adjacency matrix of the collision time graph. If there is an intersection η in the movement directions of two vehicles, there is a potential collision risk between them. Calculated by the following formula:

[0024]

[0025] ∈i and ∈j represent the predicted collision time of vehicle i and vehicle j to the intersection η.

[0026] In the above technical solution, in step 2, the visual field map Distance map and collision time diagram The GAT calculation formula is as follows:

[0027]

[0028] H l represents the node feature matrix of the lth layer, l is 1 or 2. When l is 1, H l-1 Equal to H0, that is, three types of graph data in the STDG model and When l is 2, H l Equal to H2, the characteristics of each generated graph are: and σ is a nonlinear activation function, A ij represents the values ​​in the adjacency matrix, represents the set of neighbor nodes of node i, W lis the weight matrix of the lth layer, α ij is the attention coefficient.

[0029] In the above technical solution, α ij The calculation method is as follows:

[0030]

[0031] Among them, a l is a learnable weight vector, || represents vector concatenation, h i ,h j With h k They represent the feature vectors of vehicle i, vehicle j and vehicle k respectively, and LeakyReLU is the activation function.

[0032] In the above technical solution, in step 3,

[0033] In the above technical solution, in step 4, the vehicle state and navigation feature H2 are processed through two consecutive dense layers, and the mathematical expression is as follows:

[0034] H1=σ(W1x+b1),

[0035] H2=σ(W2H1+b2),

[0036] Among them, x represents the input ego vehicle state and navigation data, W1 and W2 are weight matrices, b1 and b2 are bias terms, σ represents the activation function, H1 is the output of the first dense layer, and H2 is the output of the second dense layer, i.e., the ego vehicle state and navigation features.

[0037] In the above technical solution, in step 4, H flatten =Flatten(H conv ).

[0038] In the above technical solution, in step 4, S' t =Concatenate(H2,H f , H flatten ).

[0039] In the above technical solution, for continuous control, the driving strategy π θ (s′ t ) can be updated by the following gradient ascent algorithm:

[0040] θ new ←θ now +β·g(s′ t , a t θ now )

[0041] Among them, θ represents the parameters of the policy model, and the subscripts new and now represent after and before the update, respectively. is the expected return An unbiased estimate of , β represents the learning rate, represents the state-value function, a t Represents the action at time t.

[0042] In the above technical solution, the state-value function The ω in can be updated using the temporal difference (TD) learning method by the following formula:

[0043]

[0044] Among them, ω represents the parameters of the value function model, and the subscript ne w and no w Respectively represent after update and before update, is the TD error, is the estimated value of the state at time t, is the TD target, is the gradient of ω, r t represents the instantaneous reward at time t, the γ discount factor, between 0 and 1, used to weigh the impact of future rewards, and a t Represents the action at time t.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This paper proposes an innovative method architecture that integrates a multimodal feature extraction network containing graph information into the Actor-Criti c In the reinforcement learning framework, this integration can effectively handle the complexity of high-dimensional environmental information and continuous vehicle actions, thereby providing more accurate input features for autonomous driving decisions.

[0047] 2. This paper introduces a spatiotemporal directed graph model into the GAT network for the first time to capture the complex interaction between autonomous vehicles and human-driven vehicles. The spatiotemporal directed graph model uses the view graph, distance graph, and collision time to Figure 3 A directed graph topology accurately describes the dynamic and asymmetric interaction relationship between vehicles, thereby enhancing the negotiation decision-making strategy for autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a multimodal feature extraction network-enhanced autonomous driving negotiation decision algorithm architecture diagram based on spatiotemporal directed graph;

[0049] Figure 2It is a schematic diagram of the modeling of the visual field map;

[0050] Figure 3 It is a schematic diagram of distance graph modeling;

[0051] Figure 4 It is the collision time map modeling field of view map;

[0052] Figure 5 It is the network structure diagram of the multimodal feature extraction layer. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] A multimodal feature extraction network-enhanced autonomous driving negotiation decision method based on a spatiotemporal directed graph includes the following steps:

[0055] Step 1: construct a spatiotemporal directed graph (STDG) model, which includes three interaction prior graph models: the visual field graph VG, the distance graph DG, and the collision time graph CG;

[0056] The spatiotemporal directed graph (STDG) model is an effective graph representation technology for modeling complex social behaviors between vehicles. Different from traditional graph representation methods, the present invention models the interaction relationship in the scene as a directed graph by encoding the information of field of view, distance and collision time. The STDG model includes three graph topologies: field of view graph VG, distance graph DG and collision time graph CG, which are used to characterize the dynamic and asymmetric interaction relationship between vehicles.

[0057] The field of view graph VG is a graph model built based on the viewing range of the vehicle. Each node represents a vehicle, and the existence and weight of the edge depends on whether other vehicles are within the viewing range of the vehicle. The field of view graph can capture the area of ​​attention of the vehicle to the surrounding environment during actual driving.

[0058] like Figure 2 As shown in Figure 1, when a vehicle i is within the field of view of another vehicle j, it will affect the behavior of vehicle j. On the contrary, if vehicle k is not within the field of view of vehicle j, it will not affect the behavior of vehicle j. represents the node matrix of the visual field graph, represents the adjacency matrix of the visual field graph. Vehicle i is represented as a node Vehicle j is represented as a node Vehicle i is included in VG only if it is within the maximum detection range of vehicle j. The node attributes include relative speed and position, which are normalized. Located at the node Within the visual range, from the vehicle j node point to There is an edge Otherwise there is no edge side The presence or absence of is determined by the following formula:

[0059]

[0060] in:

[0061]

[0062] α represents the node To Node The angle between Representation Node The moving direction, d ij Is a node To Node The direction vector, α v is the viewing angle of the vehicle, Represents the set of all nodes within the maximum detection range of the vehicle.

[0063] The distance graph DG is a graph model built based on the relative distance between vehicles. The edge weights of the graph are determined by the actual distance between vehicles. The closer the distance, the greater the edge weight. The distance graph effectively reflects the physical proximity between vehicles and their mutual influence.

[0064] The distance graph DG is defined based on the physical distance between vehicles, such as Figure 3 As shown. Define the distance map represents the distance graph node matrix, Represents the distance graph adjacency matrix, with vehicle i and vehicle j represented as nodes and The side Representation Node To Node The distance is expressed by the following formula:

[0065]

[0066] The collision time graph CG is a graph model that calculates the collision time between vehicles. The weight of an edge is determined by the time difference between two vehicles that may collide on their paths. The shorter the time difference, the greater the weight of the edge. The collision time graph helps predict and avoid potential collision risks.

[0067] CG is used to describe the interaction between vehicles that may collide in crowded scenes, such as Figure 4 As shown. Define the collision time diagram represents the collision time graph node matrix, represents the adjacency matrix of the collision time graph. If there is an intersection η between the moving directions of two vehicles, there is a potential collision risk between them. Calculated by the following formula:

[0068]

[0069] ∈ i With ∈ j represents the predicted collision time between vehicle i and vehicle j at the intersection η.

[0070] Step 2: Input the visual field graph VG, distance graph DG, and collision time graph CG obtained in step 1 into the graph attention network (GAT) for calculation. GAT can adaptively aggregate the features of each node and its neighboring nodes through the self-attention mechanism, thereby effectively extracting the complex relationship information in the graph. Through the processing of GAT, the feature representation of each graph is generated, which are (Visual field features), (distance map feature) and (Collision time graph features).

[0071] Each figure ( and ) is calculated as follows:

[0072]

[0073] Among them, H l represents the node feature matrix of the lth (1 or 2) layer. When l is 1, H l-1 Equal to H0, that is, three kinds of graph data in the STDG model: and When l is 2, H l Equal to H2, the characteristics of each generated graph are: and σ is a nonlinear activation function, A ij represents the values ​​in the adjacency matrix, represents the set of neighbor nodes of node i, W l is the weight matrix of the lth layer, α ij is the attention coefficient, calculated by the following formula:

[0074]

[0075] Among them, a l is a learnable weight vector, || represents vector concatenation, h i ,h j With h kThey represent the feature vectors of vehicle i, vehicle j and vehicle k respectively, and LeakyReLU is the activation function.

[0076] Step 3: Multi-layer Perceptron (MLP) (Visual field features), (distance map feature) and The fusion of (collision time graph features) forms a unified representation Hf. This fused feature representation integrates the information of different graph models, which helps to fully reflect the complex interaction relationship between vehicles and improve the accuracy of subsequent strategy generation.

[0077] The process formula of graph fusion is as follows:

[0078]

[0079] Among them, H f It is the fused graph feature, which combines the diverse information in each individual graph, thereby enhancing the model's ability to use multi-angle information to make better decisions.

[0080] Step 4, multimodal feature fusion: In addition to the fusion of graph features, multimodal information from other data sources needs to be integrated. First, the vehicle status and navigation information are processed using MLP to obtain the vehicle status and navigation features H2; at the same time, the LiDAR data is processed using a convolutional neutral network (CNN), and the processed features are flattened into a one-dimensional vector H flatten Finally, the vehicle state and navigation features H2 processed by MLP and the fused graph features H obtained in step 3 are combined. f and the flattened lidar feature H flatten Perform serial fusion to generate comprehensive feature representation S′ t , as the core input for subsequent decision-making.

[0081] First, the ego vehicle state and navigation data contains information about the vehicle itself and its planned route. This data is first processed through two consecutive fully connected (dense) layers, which extract meaningful features and facilitate higher-level representations. The mathematical expression is as follows:

[0082] H1=σ(W1x+b1),

[0083] H2=σ(W2H1+b2),

[0084] Among them, x represents the input ego vehicle state and navigation data, W1 and W2 are weight matrices, b1 and b2 are bias terms, σ represents the activation function, H1 is the output of the first dense layer, and H2 is the output of the second dense layer, i.e., the ego vehicle state and navigation features.

[0085] LiDAR data is processed through convolutional (Conv) layers to extract spatial features from the 2D point cloud data. Specifically, the LiDAR data passes through two convolutional layers to capture the spatial hierarchy in the input data. The feature map generated by the convolutional layer is then converted to a 1D feature vector through a Flatten layer to make the data compatible with integration with other features. This flattened feature vector is represented as:

[0086] H flatten =Flatten(H conv )

[0087] Among them, H conv Represents the lidar features after processing by the convolution (Conv) layer.

[0088] For the above vehicle state and navigation features H2, the fused graph features H f and the flattened lidar feature H flatten Fusion is performed to obtain the fused comprehensive feature S′ t express:

[0089] S′ t =Concatenate(H2,H f , H flatten )

[0090] The final concatenation layer integrates the diverse extracted features from all data sources into a comprehensive feature vector, denoted as S′ t .

[0091] Step 5: Strategy optimization based on the Actor-Critic (AC) framework: The fused comprehensive feature representation S′ t Enter the AC framework for policy optimization. In the AC framework, the policy network (Actor) is based on S′ t Generate specific driving strategy π θ (s′ t ); the evaluation network (Critic) evaluates the decisions generated by the policy network and guides the optimization of the policy network by calculating the rewards. Through continuous training and optimization, the policy network gradually learns the optimal driving strategy.

[0092] The STDG model and the multimodal feature extraction network are integrated into the Actor-Critic (AC) learning framework. The STDG model integrates three types of prior graph information, which are combined with the multimodal feature extraction network to form an enhanced state representation S′ t The Actor-Critic (AC) algorithm consists of two main components: the Actor that learns the policy and the Critic that learns the value function. θ represents the parameters of the policy model, ω represents the parameters of the value function model, and the subscripts new and now represent the updated and before update, respectively. For continuous control, the driving policy π θ (s′ t ) can be updated by the following gradient ascent algorithm:

[0093] θ new ←θ now +β·g(s′ t , a t θ now )

[0094] in, is the expected return An unbiased estimate of , β represents the learning rate, a t Represents the action at time t.

[0095] State-Value Function The ω in can be updated using the temporal difference (TD) learning method by the following formula:

[0096]

[0097] in, is the TD error, is the estimated value of the state at time t, is the TD target, is the gradient of ω, r t represents the instantaneous reward at time t, the γ discount factor, between 0 and 1, used to weigh the impact of future rewards, and a t Represents the action at time t.

[0098] This AC-based enhanced architecture combines multimodal feature extraction and graph representation technology to achieve effective perception and decision optimization of complex environments, thereby improving the decision quality of autonomous driving systems in high-dynamic scenarios. Real-time decision-making is performed on the driving environment at each moment. Specifically, the policy network generates the corresponding action a t , the longitudinal action includes the vehicle's driving and braking control signals, and the lateral action is the steering control signal.

[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An autonomous driving negotiation decision method based on a multimodal feature extraction network and reinforcement learning framework based on a spatiotemporal directed graph, characterized in that: The following steps are involved: Step 1, construct a spatiotemporal directed graph model STDG model consisting of a visual field graph VG, a distance graph DG, and a collision time graph CG; Step 2: Input the visual field graph VG, distance graph DG and collision time graph CG obtained in step 1 into the graph attention network GAT for calculation. Through GAT processing, the visual field graph features are generated. Distance map features and collision time graph features Step 3: Use the multi-layer perceptron MLP to transform the and Fusion is performed to obtain the fused graph feature H f ; Step 4: Multimodal feature fusion: The vehicle status and navigation information are processed using MLP to obtain the vehicle status and navigation features H2. The LiDAR data is processed using CNN and the processed features are flattened into a one-dimensional vector H flatten , H2, H flatten And H obtained in step 3 f Perform serial fusion to generate comprehensive feature representation S′ t ; Step 5: The comprehensive feature representation S′ obtained in step 4 is t Input AC framework for policy optimization. In AC framework, the policy network Actor is based on S′ t Generate specific driving strategy π θ (s′ t ); the evaluation network Critic evaluates the decisions generated by the policy network and guides the optimization of the policy network Actor by calculating the rewards. Through continuous training and optimization, the policy network gradually learns the optimal driving strategy.

2. The autonomous driving negotiation decision method according to claim 1, characterized in that: In step 1, define the visual field map represents the node matrix of the visual field graph, represents the adjacency matrix of the visual field graph. Vehicle i is represented as a node Vehicle j is represented as a node If the node Located at the node Within the visual range, from the vehicle j node point to There is an edge Otherwise there is no edge side The presence or absence of is determined by the following formula: in: α represents the node To Node The angle between Representation Node The moving direction, d ij Is a node To Node The direction vector, α v is the viewing angle of the vehicle, Represents the set of all nodes within the maximum detection range of the vehicle; Defining the distance map represents the distance graph node matrix, Represents the distance graph adjacency matrix, with vehicle i and vehicle j represented as nodes and The side Representation Node To Node The distance is expressed by the following formula: Defining the collision time graph represents the collision time graph node matrix, Represents the adjacency matrix of the collision time graph. If there is an intersection η in the movement directions of two vehicles, there is a potential collision risk between them. Calculated by the following formula: ∈ i With ∈ j represents the predicted collision time between vehicle i and vehicle j at the intersection η.

3. The autonomous driving negotiation decision method according to claim 1, characterized in that: In step 2, the visual field map Distance map and collision time diagram The GAT calculation formula is as follows: H l represents the node feature matrix of the lth layer, l is 1 or 2. When l is 1, H l-1 Equal to H0, that is, three types of graph data in the STDG model and When l is 2, H l Equal to H2, the characteristics of each generated graph are: and σ is a nonlinear activation function, A ij represents the values ​​in the adjacency matrix, represents the set of neighbor nodes of node i, W l is the weight matrix of the lth layer, α ij is the attention coefficient.

4. The autonomous driving negotiation decision method according to claim 3, characterized in that: α ij The calculation method is as follows: Among them, a l is a learnable weight vector, || represents vector concatenation, h i ,h j With h k They represent the feature vectors of vehicle i, vehicle j and vehicle k respectively, and LeakyReLU is the activation function.

5. The autonomous driving negotiation decision method according to claim 1, characterized in that: In step 3, 6. The autonomous driving negotiation decision method according to claim 1, characterized in that: In step 4, the vehicle state and navigation feature H2 are processed through two consecutive dense layers, and the mathematical expression is as follows: H1=σ(W1x+b1), H2=σ(W2H1+b2), Among them, x represents the input ego vehicle state and navigation data, W1 and W2 are weight matrices, b1 and b2 are bias terms, σ represents the activation function, H1 is the output of the first dense layer, and H2 is the output of the second dense layer, i.e., the ego vehicle state and navigation features.

7. The autonomous driving negotiation decision method according to claim 1, characterized in that: In step 4, H flatten =Flatten(H conv ).

8. The autonomous driving negotiation decision method according to claim 1, characterized in that: In step 4, S′ t =Concatenate(H2,H f ,H flatten ).

9. The autonomous driving negotiation decision method according to claim 1, characterized in that: For continuous control, the driving policy π θ (s′ t ) can be updated by the following gradient ascent algorithm: i new ←θ now +β·g(s′ t ,a t ;θ now ) Among them, θ represents the parameters of the policy model, and the subscripts new and now represent after and before the update, respectively. is the expected return An unbiased estimate of , β represents the learning rate, represents the state-value function, a t Represents the action at time t.

10. The autonomous driving negotiation decision method according to claim 9, characterized in that: State-Value Function The ω in can be updated using the temporal difference (TD) learning method by the following formula: Among them, ω represents the parameters of the value function model, and the subscripts new and now represent after and before the update, respectively. is the TD error, is the estimated value of the state at time t, is the TD target, is the gradient of ω, r t represents the instantaneous reward at time t, the γ discount factor, between 0 and 1, used to weigh the impact of future rewards, and a t Represents the action at time t.

Images