Ramp merging multi-agent cooperative decision-making method based on space-time interaction capture and action priority

Abstract

This invention discloses a multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority, belonging to the field of intelligent transportation technology. The method includes: acquiring a sequence of observation information from multiple agents and obtaining a spatiotemporal interaction code through temporal encoding and self-attention interaction encoding; gating the spatiotemporal interaction code based on ramp merging features (TTMP); inputting the enhanced code into a policy priority ranking network, sampling the action generation order using Plackett-Luce, and then reordering the enhanced code, before inputting it into a decoding policy network with cross-attention to form a spatiotemporal interactive Plackett-Luce multi-agent reinforcement learning model. The model then autoregressively generates actions for each agent according to the reordered order and integrates them to form a joint action, generating vehicle control commands to control vehicles to perform collaborative driving in the ramp merging area. This invention can dynamically model the multi-vehicle interaction relationship and the action priority dependency between individual decisions in a strong interaction scenario of ramp merging, thereby improving the stability of collaborative decision-making and traffic efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation systems, specifically to a multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority. Background Technology

[0002] As connected vehicles (CAVs) enhance their capabilities in vehicle perception and vehicle-to-infrastructure (V2I) communication, multi-vehicle collaborative decision-making at bottleneck sections such as ramp merging has become a key technology for improving traffic efficiency and safety. In mixed traffic conditions where CAVs coexist with human-driven vehicles, ramp vehicles merging into the main line trigger frequent following, yielding, and lane-changing interactions. The relative positions, speeds, and lane relationships between vehicles change rapidly within a short period, resulting in a scenario characterized by strong spatiotemporal coupling, strong interaction dependence, and concentrated decision-making conflicts.

[0003] The relevant technologies mainly employ multi-agent deep reinforcement learning models to learn multi-vehicle cooperative strategies through data-driven approaches, and fuse multi-vehicle information through an interaction modeling module. However, in ramp merging scenarios, the cooperative effect of decision-making highly depends on whether the model can effectively capture the spatiotemporal interaction relationships of multiple vehicles, identify the key interaction objects and interaction intensities that play a dominant role in merging safety and efficiency, and whether it can form a reasonable action generation priority based on this, so that the decisions of key vehicles are determined first, thereby providing conditional decision-making basis for subsequent vehicles. The current technical approach to the interaction modeling process of mixed traffic flow ramp merging is insufficient in highlighting key interaction information, and it is difficult to continuously focus on the vehicle relationships most sensitive to cooperation under the conflict situation near the merging point, resulting in low efficiency in utilizing interaction information during the strategy learning process. In addition, the joint action generation or sequential decision-making framework adopts a fixed or implicit action generation order, lacking explicit modeling of the dependency relationship between action generation order and cooperative effect, making it difficult to further transform the captured interaction intensity into learnable decision priorities, thus limiting the convergence stability and final performance of the multi-vehicle cooperative strategy.

[0004] Therefore, there is an urgent need for a collaborative decision-making method for multi-agent systems in mixed traffic ramp merging scenarios, which can enable each agent to effectively focus on the preceding key actions and interaction codes in the optimal order during the generation process, thereby comprehensively improving the stability, security and traffic efficiency of collaborative decision-making. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority. This method can form spatiotemporal interaction codes in the observation sequence and highlight key merging interactions, and learn the priority order of action generation based on the interaction codes.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-agent collaborative decision-making method for hybrid traffic ramp merging based on spatiotemporal interaction enhancement and action priority, specifically including the following steps: S1. Construct a multi-agent CAVs system, obtain the observation information sequence of the multi-agents participating in the collaboration to form the original observation feature vector, stack the observation feature vectors to obtain the single-vehicle observation matrix; based on the single-vehicle observation matrix, stack it according to the agent dimension to form a joint observation tensor, input the joint observation tensor sequence into the temporal coding network to obtain the multi-agent temporal representation; In the scenario of this invention, the set of controllable agents in the multi-agent CAVs reinforcement learning system is as follows: The total number of controllable intelligent agents is Among them, individual controllable intelligent agents For any time t, the individual Acquire information about neighboring vehicles within the visible range of the vehicle's perception and / or V2X communication, and integrate the observations according to the nearest visible vehicle index k. This includes information about individual controllable intelligent agents. The original observed feature vector constructed at time t satisfies the following expression: in, The location of the visible vehicle is indicated; For planar position; For velocity components; This is the heading angle.

[0007] individual intelligent agents At time t After stacking the observation vectors of the nearest visible vehicles according to index k to obtain the single-vehicle observation matrix, then... The individual vehicle observation matrices of each controllable agent are stacked along the agent dimension to form a joint observation tensor: In time For each recursive step, the joint observation tensor at each time step is... The inner elements are sequentially input into the temporal coding network, and the LSTM state is updated recursively over time to obtain the temporal representation. The temporal representations of all controllable agents are then concatenated according to a preset embedding dimension to output the multi-agent temporal representation: This completes the output of step S1: multi-agent temporal representation. .

[0008] S2. Based on the multi-agent temporal representation, the interaction attention between multiple agents is calculated through a self-attention interaction coding network, and the spatiotemporal interaction coding is output. Multi-agent temporal representation matrix The input is processed by a self-attention interactive encoding network, and the query is obtained through a linear mapping. ,key Sum A matrix, expressed as follows: in, For multi-agent temporal representation, To query weight, For key weights, Value weights; Construct a scaled dot product attention score, as shown in the following expression: in, The first , Row vectors of agents; This is the scaling item. Among them, The first , Row vectors of agents; A logit score is given to the attention of the vehicle. For vehicles "Strength of correlation"; denominator This is a scaling term used to stabilize the magnitude of the dot product; The attention weights are obtained by performing row normalization along index j, as shown in the following expression: The self-attention output is obtained by weighted convergence of the value vectors. in, For the first Aggregate representation of individual agents (weighted aggregation of all) ), ,overall , This represents the value vector of the j-th agent.

[0009] Subsequently, residual connections and layer normalization are performed on the attention output, and the multi-agent spatiotemporal interaction encoding is obtained through a feedforward network: in, For attention mechanism operators, For layer normalization operators, For a fully connected multilayer perceptron, then This serves as the multi-agent spatiotemporal interaction code output from step S2.

[0010] S3. Based on the ramp merging characteristics, perform gated fusion enhancement on the spatiotemporal interaction code to obtain the spatiotemporal interaction enhanced code; The specific characteristic of the ramp merging is the time to reach the merging point; The location of the confluence point is predetermined in the vertical coordinate system. At any moment Extract the first The longitudinal position and longitudinal speed of the controllable intelligent vehicle are: And calculate the arrival time at the confluence point, the expression is: in, Used to suppress instability caused by extremely low speeds. The amplitude of the vehicle's longitudinal velocity component can be taken as either the value of the longitudinal approach velocity or the equivalent longitudinal approach velocity defined by the implementation system. The location of the confluence point in the vertical coordinate system. The longitudinal position of the controllable intelligent vehicle.

[0011] With bicycle For reference, construct the vehicle and the first The arrival time difference of the nearest vehicles: in, For the first The arrival time of nearby vehicles at the confluence point This refers to the time it takes for the vehicle's ego to reach the merge point; For each neighboring vehicle Calculate the exponential temperature weighting: in, The preset temperature parameter is used to adjust the "time proximity sensitivity".

[0012] Organize the weights of all neighboring agents into a weight vector according to index j: Will The input consists of two MLP layers, generating channel-wise gated vectors. The activation functions used are ReLU and Sigmoid, respectively. In a preferred implementation, to limit the magnitude of the gating gain, the gating vector can be clipped to obtain... The expression is as follows: in, The preset maximum gain is set for each agent. The pruned gating vector The interaction coding is applied to each channel, and residual fidelity fusion is employed. Enhanced encoding is obtained: in, This is the residual fidelity coefficient; All intelligent agents The current structure is obtained by stacking agents according to their dimensions. Spatiotemporal interaction enhancement coding at any given moment: in, This is the Hadamard term-by-term product.

[0013] S4. Input the spatiotemporal interaction enhancement encoding strategy into the priority ranking network, and use Plackett-Luce sampling to obtain the action generation order. And according to the order of action generation Reorder the spatiotemporal interaction enhancement codes and output the reordered spatiotemporal interaction enhancement codes. This invention incorporates spatiotemporal interaction enhancement coding. Input policy priority ranking network In this embodiment, the network consists of a multilayer perceptron scoring network, which outputs a comparable preference score for each agent and forms a preference score vector. This explicitly transforms "interaction enhancement features" into "quantifiable basis for decision priority".

[0014] Furthermore, the Plackett-Luce (PL) sampling method is used to determine the action generation sequence. It is represented as a sampled permutation random variable and modeled according to log-parameterized order probabilities: in, For at any time In order The Middle The preference score of the agent in the position. For at any time In order The Middle The preference score of an agent is digit, where n is the total number of agents.

[0015] The optimization action generation sorting sequence is formed by extracting each element sequentially and then removing them from the candidate set. Then, the spatiotemporal interaction enhancement coding is reordered to construct the current... Spatiotemporal interaction enhancement coding after time-ordering This is then used as the state input for the subsequent decoding network.

[0016] In this invention, random sampling is used during the training phase to obtain... To enhance strategy exploration; the reasoning phase uses a deterministic approach to generate the order, such as sorting preferences by score from largest to smallest to obtain the order at time step [time]. Action generation order ,correspond For at any time The preference score vector.

[0017] S5. Input the reordered spatiotemporal interaction enhancement code into the decoding policy network containing self-attention and cross-attention mechanisms to construct the spatiotemporal interaction Plackett-Luce multi-agent reinforcement learning SIPL-MARL model, and follow the order of action generation. The joint policy is calculated in an autoregressive manner to generate the actions of each agent. When generating subsequent agent actions, the decoding policy network uses a cross-attention mechanism to effectively focus on the reordered enhanced encoding and the action information of the preceding agent, and then generates a decision action, integrates and outputs the joint action. The reordered spatiotemporal interaction enhancement encoder is input into the decoder-actor network. Structurally, this network includes a masked self-attention module and a cross-attention module. The masked self-attention module applies causal constraints to the generated preceding actions, ensuring that subsequent actions depend only on their preceding actions. The cross-attention module... The element vectors in the algorithm serve as keys and values, while the decoded hidden state is used as the query. This allows each action generation step to simultaneously focus on the reordered key interaction codes and the determined preceding actions. This effectively focuses on the state observation interaction codes and the decisions of preceding CAVs during the action decision generation process.

[0018] Specifically, the decoding process is implemented as follows: Self-regression unfolds: On Generate action sequences sequentially. and the preceding action sequence Embedded as an input mask for action context self-attention; simultaneously, cross-attention reading. Establish conditional states and output the current action distribution of the agent. Therefore, the joint strategy can be written as: , After one round of decoding, the combined actions are obtained. Then it is converted into a joint action under the original sequence. Output.

[0019] S6. Generate vehicle cluster control commands based on joint actions to control vehicles in the ramp merging area to perform cooperative driving, and collect trajectory experience samples in the training iteration environment interaction and store them in the experience pool. S7. Sample experience data from the experience pool and calculate the advantage estimate through the value assessment network; calculate the policy loss, value loss and ranking loss based on the advantage estimation function; weight and fuse the three losses into a total loss, and jointly update the parameters of the temporal coding network, self-attention interactive coding network, policy priority ranking network and cross-attention decoding network through backpropagation.

[0020] The spatiotemporal interactive value assessment network (SI-Critic) is used to output state values. The advantage estimation function is constructed as follows: , in, A tribute to Monte Carlo.

[0021] The Decoder-Actor network updates its output using a pruning target based on probability ratios: [the text abruptly ends here, likely due to an incomplete sentence or missing information.] Construct importance sampling ratio And in A pruning optimization objective is formed under a pruning threshold, thereby achieving stable PPO-style updates. This network uses a pruning objective based on probability ratios for updates, where the probability ratios satisfy: , in, To stabilize training, a target decoding strategy is introduced, and Updated asynchronously.

[0022] The network loss calculation expression for the decoding strategy is as follows: , in, For the trajectory length, The preset clipping threshold is denoted by m, where m is the position index in the sequence.

[0023] In this embodiment, the policy prioritization network can be stably updated as the policy is learned, by introducing advantage estimation. The relevant ranking update mechanism employs a probability ratio pruning method to suppress training oscillations. The ranking probability ratio is defined as: , in, This represents the forward mapping of the policy priority ranking network. To stabilize training, a target network strategy is introduced to prioritize the network; the network parameters of both are updated asynchronously. This is based on the pruning target pair. Updates are performed to achieve a learnable coupling between order probabilities and collaborative benefits. The loss expression for the policy priority ranking network is as follows: .

[0024] The spatiotemporal interaction value assessment network is updated using mean squared error loss, as shown in the following expression: In this invention, the losses of each network in the model are weighted and fused into a total loss function during training and updating. satisfy: in, To preset the network loss weight coefficients for the decoding strategy, To pre-determine the network weight coefficients for assessing the value of spatiotemporal interaction, The network weights are prioritized according to the preset strategy; and the Adam optimizer is used to optimize the network parameters of the decoding strategy. Spatiotemporal Interaction Value Network Parameters and sorting module network parameters Total loss The backpropagation update.

[0025] Compared with the prior art, the present invention has the following beneficial effects: This invention addresses the issues of insufficient interaction modeling and difficulty in highlighting key vehicle relationships in mixed traffic ramp merging scenarios by extracting spatiotemporal interaction features and employing gating enhancement mechanisms. It achieves dynamic focusing on multi-vehicle interaction relationships and effective enhancement of key interaction information. Furthermore, through the synergistic effect of Plackett-Luce priority ranking and cross-attention decoding networks, this invention solves the problems of fixed action generation order and implicit decision dependencies in traditional multi-agent reinforcement learning methods. It achieves explicit action priority learning and conditional sequential decision-making based on interaction intensity, thereby improving the decision stability, traffic efficiency, and scenario adaptability of multi-vehicle collaboration. This invention enables safer, more efficient, and more robust multi-agent collaborative driving in highly interactive ramp merging environments. Attached Figure Description

[0026] Figure 1 The flowchart is a multi-agent collaborative decision-making method for traffic ramp merging based on spatiotemporal interaction enhancement and action priority according to the present invention. Figure 2 This is a schematic diagram of the spatiotemporal interaction feature enhancement method in this invention; Figure 3This is a schematic diagram of the SIPL-MARL framework for strategy priority optimization under spatiotemporal interaction representation in this invention; Figure 4 The training curve of the present invention compared to the standard baseline algorithm is shown below, wherein... Figure 4 (a) is a reward curve graph of the training process. Figure 4 (b) is a graph of the average velocity during the training process; Figure 5 The training loss curves for the Plackett-Luce score evaluation network and the Decoder-Actor network of this invention are shown below. Figure 5 (a) is a graph showing the loss curve of the PL sorting network during the training process. Figure 5 (b) is a graph showing the loss curve of the Decoder-Actor network during the training process; Figure 6 This is a test box plot of the present invention deployed in a specific traffic flow simulation, wherein, Figure 6 (a) is a graph showing the post-deployment test reward value. Figure 6 (b) is a graph showing the average speed of deployment testing. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figures 1-6 A multi-agent collaborative decision-making method for traffic ramp merging based on spatiotemporal interaction capture and action priority is proposed. The specific implementation method includes the following steps: S1. Construct a multi-agent CAVs system, obtain the observation information sequence of the multi-agents participating in the collaboration to form the original observation feature vector, stack the observation feature vectors to obtain the single-vehicle observation matrix; based on the single-vehicle observation matrix, stack it according to the agent dimension to form a joint observation tensor, input the joint observation tensor sequence into the temporal coding network to obtain the multi-agent temporal representation matrix. In the scenario of this invention, the set of controllable agents in the multi-agent CAVs reinforcement learning system is as follows: The total number of controllable intelligent agents is Among them, individual controllable intelligent agents For any time t, the individual Acquire information about neighboring vehicles within the visible range of the agent's onboard perception and / or V2X communication, and integrate the observations according to the nearest visible vehicle index k. This includes information about individual agents. The original observed feature vector constructed at time t satisfies the following expression: in, The location of the visible vehicle is indicated; For planar position; For velocity components; This is the heading angle.

[0029] individual intelligent agents At time t The single-vehicle observation matrix is ​​obtained by stacking the visible vehicle observation vectors according to their index k: in, For the observation feature dimension, This represents the number of vehicles visible in the vicinity of the original observation space.

[0030] Furthermore, The individual vehicle observation matrices of each controllable agent are stacked along the agent dimension to form a joint observation tensor: To simulate the temporal dynamic evolution across multiple decision steps during ramp merging, at time... For each recursive step, the joint observation tensor at each time step is... The inner elements are sequentially input into the time-series coding network, and the LSTM state is updated sequentially over time. The LSTM gating update is as follows: in, For the Sigmoid function, Let be the temporal input vector sequence of any intelligent agent. It is the hyperbolic tangent function. For Hadamard's term-by-term product; For trainable parameters, For the LSTM trainable weight matrix, This is the trainable bias term for the LSTM. The current hidden state is taken. As a temporal representation, and after linear projection compression to a fixed embedding dimension D (D=64 in this embodiment), the expression is: in, This is the weight matrix of the linear projection layer, used to compress the hidden state to a fixed dimension of 64. For the bias term of the linear projection layer, Hide the current state; By embedding the temporal representations of all controllable agents and concatenating them according to the agent dimension, a multi-agent temporal representation is obtained: This completes the output of step S1: multi-agent temporal representation. .

[0031] S2. Based on multi-agent temporal representation, the interaction attention between multiple agents is calculated through a self-attention interaction coding network, and then the spatiotemporal interaction coding of multiple agents is output through a feedforward neural network. In step S2, the multi-agent temporal representation matrix obtained in step S1 is... The input is processed by a self-attention interactive encoding network, and the query is obtained through a linear mapping. ,key Sum A matrix, expressed as follows: in, For multi-agent temporal representation, To query weight, For key weights, Value weights; Construct a scaled dot product attention score, as shown in the following expression: in, The first , Row vectors of agents; This is the scaling item. Among them, The first , Row vectors of agents; A logit score is given to the attention of the vehicle. For vehicles "Strength of correlation"; denominator This is a scaling term used to stabilize the magnitude of the dot product; The attention weights are obtained by performing row normalization along index j, as shown in the following expression: The self-attention output is obtained by weighted convergence of the value vectors. in, For the first Aggregate representation of individual agents (weighted aggregation of all) ), ,overall , This represents the value vector of the j-th agent.

[0032] Subsequently, residual connections and layer normalization are performed on the attention output, and the multi-agent spatiotemporal interaction encoding is obtained through a feedforward network: in, For attention mechanism operators, For layer normalization operators, For a fully connected multilayer perceptron, then This serves as the multi-agent spatiotemporal interaction code output from step S2.

[0033] S3. Based on the ramp merging characteristics, perform gated fusion enhancement on the spatiotemporal interaction code to obtain the spatiotemporal interaction enhanced code; The specific feature of the ramp merging is the arrival time to the merging point (TTMP) constructed by the merging point. After calculating the arrival time difference based on TTMP, a weight vector is calculated. The weight vector is input into a two-layer MLP and a gating vector is output. The gating vector is then pruned to obtain a pruned gating vector. The core objective of step S3 is to highlight the most sensitive key interactions near the merging point in the multi-agent spatiotemporal interaction code obtained in step S2, so as to facilitate subsequent priority learning and action generation. In this embodiment, the arrival time difference (TTMP) is used as the ramp merging feature to construct the arrival time difference and generate a gating vector, and gain modulation and residual fidelity fusion are performed on the interaction code.

[0034] After integration and joint observation, the data will be transformed into spatiotemporal interaction enhanced coding through the spatiotemporal interaction feature enhancement module and steps S1, S2, and S3, as follows: Figure 2 As shown.

[0035] The location of the confluence point is predetermined in the vertical coordinate system. At any moment Extract the first The longitudinal position and longitudinal speed of the controllable intelligent vehicle are: And calculate the arrival time at the confluence point, the expression is: in, Used to suppress instability caused by extremely low speeds. The amplitude of the vehicle's longitudinal velocity component can be taken as either the value of the longitudinal approach velocity or the equivalent longitudinal approach velocity defined by the implementation system. The location of the confluence point in the vertical coordinate system. The longitudinal position of the controllable intelligent vehicle.

[0036] With bicycle For reference, construct the vehicle and the first The arrival time difference of the nearest vehicles: in, For the first The arrival time of nearby vehicles at the confluence point This refers to the time it takes for the vehicle's ego to reach the merge point; For each neighboring vehicle Calculate the exponential temperature weighting: in, The preset temperature parameter is used to adjust the "time proximity sensitivity".

[0037] Organize the weights of all neighboring agents into a weight vector according to index j: Will The input consists of two MLP layers, generating channel-wise gated vectors. The activation functions used are ReLU and Sigmoid, respectively. In a preferred implementation, to limit the magnitude of the gating gain, the gating vector can be clipped to obtain... The expression is as follows: in, The preset maximum gain is set for each agent. The pruned gating vector The interaction coding is applied to each channel, and residual fidelity fusion is employed. Enhanced encoding is obtained: in, This is the residual fidelity coefficient; All intelligent agents The current structure is obtained by stacking agents according to their dimensions. Spatiotemporal interaction enhancement coding at any given moment: in, For the Hadamard term-by-term product. The spatiotemporal interaction enhancement code output from step S3. This will be used for subsequent action generation priority learning and value assessment.

[0038] S4. Input the spatiotemporal interaction enhancement encoding into the strategy priority ranking network to obtain the action generation order. And according to the order of action generation Reorder the spatiotemporal interaction enhancement codes and output the reordered spatiotemporal interaction enhancement codes. The policy priority ranking network outputs preference scores through a multi-layer MLP scoring network and generates action sequences based on Plackett-Luce model sampling. .

[0039] Based on spatiotemporal interaction-enhanced encoding, a multi-agent reinforcement learning framework for policy priority optimization under spatiotemporal interaction representation is constructed. This framework outputs semantic decision actions and completes the training of its own network, corresponding to steps S4 to S7. Figure 3 As shown, specifically: In step S4, the spatiotemporal interaction enhancement encoding is performed. Input policy priority ranking network In this embodiment, the network consists of a multilayer perceptron scoring network, which outputs a comparable preference score for each agent and forms a preference score vector. This explicitly transforms "interaction enhancement features" into "quantifiable basis for decision priority".

[0040] Furthermore, step S4 uses the Plackett-Luce (PL) sampling method to generate the action sequence. It is represented as a sampled permutation random variable and modeled according to log-parameterized order probabilities: in, For at any time In order The Middle The preference score of the agent in the position. For at any time In order The Middle The preference score of an agent, where n is the total number of agents; in this invention... Let n be the total number of agents, where n represents the same quantity in mathematical formulas such as Plackett-Luce. Both n and n have the same value, and the choice of sign follows the conventions of each field.

[0041] In the specific implementation process, the optimization action generation sorting sequence can be formed by extracting each element one by one and then removing them from the candidate set. Then, the spatiotemporal interaction enhancement coding is reordered to construct the current... Spatiotemporal interaction enhancement coding after time-ordering This is then used as the state input for the subsequent decoding network.

[0042] The order of generation will differ between the training and inference phases: the training phase will use random sampling. To enhance strategy exploration; the reasoning phase uses a deterministic approach to generate the order, such as sorting preferences by score from largest to smallest to obtain the order at time step [time]. Action generation order ,correspond For at any time The preference score vector.

[0043] S5, will The input cross-attention decoding network forms a spatiotemporal interactive Plackett-Luce multi-agent reinforcement learning (SIPL-MARL) model, and follows the order of action generation. The joint policy is calculated in an autoregressive manner to generate the actions of each agent. When generating subsequent agent actions, the decoding policy network generates decision actions by effectively focusing on the reordered enhanced encoding and the action information of the preceding agent using a cross-attention mechanism, and then integrates them into a joint action. In step S5, The input decoding policy network (Decoder-Actor) structurally comprises a masked self-attention module and a cross-attention module. The masked self-attention module applies causal constraints to the generated preceding actions, ensuring that subsequent actions depend only on their preceding actions. The cross-attention module... The element vectors in the algorithm serve as keys and values, while the decoded hidden state is used as the query. This allows each action generation step to simultaneously focus on the reordered key interaction codes and the determined preceding actions. This effectively focuses on the state observation interaction codes and the decisions of preceding CAVs during the action decision generation process.

[0044] Specifically, the decoding process is implemented as follows: Self-regression unfolds: On Generate action sequences sequentially. and the preceding action sequence Embedded as an input mask for action context self-attention; simultaneously, cross-attention reading. Establish conditional states and output the current action distribution of the agent. Therefore, the joint strategy can be written as: , After one round of decoding, the combined actions are obtained. Then it is converted into a joint action under the original sequence. Output.

[0045] S6. Generate vehicle cluster control commands based on joint actions to control vehicles in the ramp merging area to perform cooperative driving, and collect trajectory experience samples in the training iteration environment interaction and store them in the experience pool. In this implementation example, step S6 will be based on the combined action obtained in step S5. The system generates high-level decision-making commands for vehicle control, with a semantic set specifically including: acceleration, deceleration, lane keeping, left lane change, and right lane change. By employing a hierarchical interface of "high-level decision-making—low-level executable control," the low-level control module translates these commands into physically executable longitudinal acceleration and steering angle commands, thus forming the control input to the vehicle's execution layer. To support the training of the aforementioned decoding strategy, this embodiment collects trajectory samples, including the current reward value, through interaction with the environment during training iterations. Joint observation tensor at continuous time points , and joint actions The empirical samples at each time step are at least... The format is stored in the experience buffer.

[0046] S7. Sample experience data from the experience pool and calculate the advantage estimate through the value assessment network; calculate the policy loss, value loss and ranking loss based on the advantage estimation function; weight and fuse the three losses into a total loss, and jointly update the parameters of the temporal coding network, self-attention interactive coding network, policy priority ranking network and cross-attention decoding network through backpropagation.

[0047] This invention uses experience extracted according to a preset trajectory sampling step size to calculate the advantage estimate and update the neural network of the algorithm. In this embodiment, the sampling step size is set to 100 steps. A spatiotemporal interactive value assessment network (SI-Critic) is used to output the state value. The advantage estimation function is constructed as follows: , in, A tribute to Monte Carlo.

[0048] The Decoder-Actor network updates its output using a pruning target based on probability ratios: [the text abruptly ends here, likely due to an incomplete sentence or missing information.] Construct importance sampling ratio And in A pruning optimization objective is formed under a pruning threshold, thereby achieving stable PPO-style updates. This network uses a pruning objective based on probability ratios for updates, where the probability ratios satisfy: , in, To stabilize training, a target decoding strategy is introduced, and Updated asynchronously.

[0049] The network loss calculation expression for the decoding strategy is as follows: , in, For the trajectory length, The preset clipping threshold is denoted by m, where m is the position index in the sequence.

[0050] In this embodiment, the policy prioritization network can be stably updated as the policy is learned, by introducing advantage estimation. The relevant ranking update mechanism employs a probability ratio pruning method to suppress training oscillations. The ranking probability ratio is defined as: , in, This represents the forward mapping of the policy priority ranking network. To stabilize training, a target network strategy is introduced to prioritize the network; the network parameters of both are updated asynchronously. This is based on the pruning target pair. Updates are performed to achieve a learnable coupling between order probabilities and collaborative benefits. The loss expression for the policy priority ranking network is as follows: .

[0051] The spatiotemporal interaction value assessment network is updated using mean squared error loss, as shown in the following expression: In this implementation, the losses of each network in the model are weighted and fused into a total loss function during training and updates. satisfy: in, To preset the network loss weight coefficients for the decoding strategy, To pre-determine the network weight coefficients for assessing the value of spatiotemporal interaction, The network weights are prioritized according to the preset strategy; and the Adam optimizer is used to optimize the network parameters of the decoding strategy. Spatiotemporal Interaction Value Network Parameters and sorting module network parameters Total loss The backpropagation update.

[0052] For the verification of this embodiment, a mixed traffic ramp merging scenario can be constructed on the Highway simulation platform. After collecting experience and training the agent with parameters as shown in Table 1, the algorithm agent is deployed under different traffic flow densities to test the collaborative strategy. The experimental results are used to compare and evaluate safety and traffic efficiency, thereby verifying the effectiveness of the collaborative decision-making chain of interaction enhancement—priority ranking—cross-attention decoding under complex interaction conditions. The preset parameters of this embodiment are shown in Table 1.

[0053] Table 1: Hyperparameters for SIPL-MARL Model Training The preset weights are designed primarily to maintain the stability of network updates. Since the policy priority ranking network significantly alters policy generation, its update magnitude is appropriately reduced in this embodiment to minimize its impact on training stability. Training examples of the proposed SIPL-MARL algorithm and the baseline algorithm are provided below. Figure 4 As shown. Regarding training rewards, SIPL-MARL continuously increases round rewards through trial and error in the early stages of training, reaching a plateau around 2500 episodes. The average reward during training is 133.13. See [link / reference]. Figure 4 (a) Regarding scene traffic efficiency, the SIPL-MARL's speed index stabilized after approximately 5000 episodes, with an average speed reaching 27.28 m / s and remaining around 28 m / s for an extended period. See [link to relevant documentation]. Figure 4 (b). Based on the above results, SIPL-MARL demonstrates a combined advantage in the training phase, with higher expected return, faster convergence speed, and higher average speed, showing better co-efficiency compared to other baseline algorithms.

[0054] Figure 5 This example demonstrates the statistical visualization of the losses of each sub-network during training: training was repeated using different random seeds, and the loss data were averaged and calculated for each seed, then plotted as curves. The results show that the pruning target loss of the PL ranking network exhibits a convergence trajectory of "rapidly decreasing first, then stabilizing." (See [link to relevant documentation]). Figure 5 (a); meanwhile, after fusing ranking results, spatiotemporal interactive coding, and Critic advantage guidance, the loss of the Decoder-Actor network gradually approaches 0 after approximately 14,000 episodes, see [link to relevant documentation]. Figure 5 (b) demonstrates that it still has stable convergence and good training robustness under different random seed conditions.

[0055] This example deploys the trained model in a scenario with 5-8 CAVs and 2-4 HDVs. The MARL model will need to coordinate multiple CAVs simultaneously during the merging process to test the model's ability to coordinate actions in a multi-agent environment. Reward value data and ensemble velocity of different models are collected and plotted. Figure 6 , Figure 6 (a) is a graph showing the post-deployment test reward value. Figure 6 (b) Deployment test average velocity plot and organize the data and standard deviation results into Table 2.

[0056] Table 2: Model Deployment Rewards and Average Speed In the table, SIPL-MARL achieved average reward and average speed of 122.14 and 27.53 m / s respectively in a multi-CAV deployment scenario. Moreover, the reward and speed distributions were more concentrated and the variance was lower, demonstrating its stability and robustness in a highly interactive and random environment. By integrating spatiotemporal interaction capture, PL sequence optimization, and sequential decision conditional generation mechanisms, SIPL-MARL demonstrated more stable policy output and stronger migration and expansion capabilities in test environments with different traffic densities and different mixing ratios.

[0057] It should be noted that the above are merely preferred embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority, characterized in that, Includes the following steps: S1. Construct a multi-agent CAVs system, obtain the observation information sequence of the multi-agents participating in the collaboration to form the original observation feature vector, stack the observation feature vectors to obtain the single-vehicle observation matrix; based on the single-vehicle observation matrix, stack it according to the agent dimension to form a joint observation tensor, input the joint observation tensor sequence into the temporal coding network to obtain the multi-agent temporal representation matrix. S2. Based on the multi-agent temporal representation, the interaction attention between multiple agents is calculated through a self-attention interaction coding network, and the spatiotemporal interaction coding is output. S3. Based on the ramp merging characteristics, perform gated fusion enhancement on the spatiotemporal interaction code to obtain the spatiotemporal interaction enhanced code; The specific feature of the ramp merging is the arrival time to merge point (TTMP) of the merging point construction. S4. Input the spatiotemporal interaction enhancement encoding into the strategy priority ranking network and sample to obtain the action generation order. And according to the order of action generation Reorder the spatiotemporal interaction enhancement codes and output the reordered spatiotemporal interaction enhancement codes. The policy priority ranking network outputs preference scores through an MLP scoring network and generates action sequences based on Plackett-Luce model sampling. ; The order of action generation The training phase uses random sampling to generate the data, while the inference phase uses a deterministic approach. The deterministic approach involves selecting the data in descending order of preference scores. S5. Input the reordered spatiotemporal interaction enhancement code into the decoding strategy network containing self-attention and cross-attention mechanisms, according to the order of action generation. The joint policy is calculated in an autoregressive manner to generate the actions of each agent. When generating subsequent agent actions, the decoding policy network generates decision actions by effectively focusing on the reordered enhanced encoding and the action information of the preceding agent using a cross-attention mechanism. The joint actions are then integrated and output to complete the construction of the spatiotemporal interactive Plackett-Luce multi-agent reinforcement learning SIPL-MARL model. S6. Vehicle cluster control commands based on joint actions are used to control vehicles within the system to perform cooperative driving in the ramp merging area, and to collect trajectory experience samples in the training iteration environment interaction and store them in the experience pool. The trajectory experience sample specifically includes: the reward value at the current moment, the joint observation tensor, and the joint action; S7. Sample experience data from the experience pool and calculate the advantage estimate through the spatiotemporal interaction value assessment network; calculate the value loss, policy loss and ranking loss based on the advantage estimation function; weight and fuse the three losses into a total loss, and jointly update the parameters of the spatiotemporal interaction value assessment network, decoding policy network and policy priority ranking network through backpropagation.

2. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In S1, the expression for the original observed feature vector is as follows: in, This indicates that a location marker is present for the visible vehicle. , Indicates the planar position of the visible vehicle. , Represents the velocity components of visible vehicles. Indicates the heading angle of the visible vehicle; The expression for the joint observation tensor is as follows: Where t is time. This represents the number of vehicles visible in the vicinity of the original observation space. For the observation feature dimension, The total number of controllable intelligent agents.

3. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In S1, the temporal coding network uses a long short-term memory network (LSTM) to encode the joint observation tensor. When performing gating updates, the hidden state at the current time is used as the temporal representation, and the temporal representation is spliced ​​together according to a preset embedding dimension to output the multi-agent temporal representation.

4. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In step S2, the interaction attention between multiple agents is calculated through a self-attention interaction coding network, and the expression for the spatiotemporal interaction coding of the multiple agents is output as follows: , in, For attention mechanism operators, Representation layer normalization operator, It is a fully connected multilayer sensor. Encoding for spatiotemporal interaction, This is a temporal representation of multi-agent systems.

5. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In S3, the calculation expression for the arrival time to the confluence point (TTMP) is: in, Used to suppress instability caused by extremely low speeds. To obtain the amplitude of the vehicle's longitudinal velocity component, The location of the confluence point in the vertical coordinate system. The longitudinal position of a controllable intelligent vehicle; Building autonomous vehicles and the first The arrival time difference of the three neighboring agents is expressed as: in, For the first The arrival time of nearby vehicles at the confluence point This refers to the time it takes for the vehicle's ego to reach the merge point; The weight vector is calculated based on ΔTTMP for each neighboring agent. The exponential temperature-weighted average is calculated using the following expression: in, Preset temperature parameters; The weight vector is input into an MLP with a preset number of layers. The activation functions of the MLP are ReLU and Sigmoid, respectively, and the output is a channel-wise gated vector. ; To limit the magnitude of the gating gain, the gating vector is clipped to obtain... The calculation expression is as follows: in, This is the preset maximum gain.

6. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In S3, the gated fusion enhancement adopts a residual-fidelity fusion method to make the enhanced coding highlight the characteristics of the ramp merging, and the specific method of obtaining the spatiotemporal interactive enhanced coding is as follows: The enhanced coding is obtained by residual-fidelity fusion, and the expression is as follows: in, The residual fidelity coefficient is used. To prune the gating vector, This represents Hadamard element-wise multiplication. Encoding for spatiotemporal interaction, ; Enhanced coding The spatiotemporal interaction enhancement encoding is obtained by stacking the total number of agents, and the expression is as follows: in, The total number of controllable intelligent agents.

7. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In step S4, the policy priority ranking network uses the Plackett-Luce sampling method to obtain the action generation order. In this context, the policy prioritization network is composed of an MLP, which uses spatiotemporal interaction to enhance coding. As input, the output is the preference score vector of each agent. The action sequence is generated by sampling based on the log-parameterized Plackett-Luce model. The expression is as follows: ； in, For at any time In order The Middle Each agent's preference score For at any time In order The Middle The preference score of an agent, where n is the total number of agents. and Index the agent.

8. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In S5, the decoding strategy network includes a mask self-attention module and a cross-attention module; The mask self-attention module imposes causal constraints on the generated preceding actions, so that subsequent actions depend only on the preceding actions. The cross-attention module uses the reordered spatiotemporal interaction enhancement code as the key and value, and the previous decision action code as the query to generate attention output; The expression for generating the actions of each agent using the joint policy in an autoregressive manner is as follows: ， in, Indicates the order at time t The decision-making actions of the generated intelligent agent. Indicates according to Joint decision actions generated sequentially This is an enhanced encoding of spatiotemporal interaction after reordering, where k is the index of the number of nearby visible vehicles. Let t be the time index and n be the total number of agents, representing the order in which actions are generated. This represents the current action distribution of the agent.

9. The multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 1, characterized in that, In S7, the total loss calculation expression is as follows: in, To preset the network loss weight coefficients for the decoding strategy, For decoding strategy network loss, To pre-determine the network weight coefficients for assessing the value of spatiotemporal interaction, To assess network loss for spatiotemporal interaction value, The network weight coefficients are sorted according to the preset strategy priority. Rank the network loss according to policy priority.

10. A multi-agent collaborative decision-making method for ramp merging based on spatiotemporal interaction capture and action priority as described in claim 9, characterized in that, The expression for calculating the network loss in the spatiotemporal interaction value assessment is as follows: in, Where t is the trajectory length and t is the time index. In return for Monte Carlo, Let i be the state value, i be the agent index, and n be the total number of agents. The expression for calculating the network loss of the decoding strategy is as follows: ， in, Here, m is the preset clipping threshold, and m is the position index in the sequence. For the advantage estimation function, Generate probability ratios for autoregressive actions that satisfy the following expression: in, For the target decoding policy network, For the current decoding policy network, Encodes to enhance spatiotemporal interactions after reordering. For the first m One generated action sequence, For m generated action sequences; The expression for calculating network loss by prioritizing policies is as follows: in, The order probability ratio satisfies the following expression: in, Prioritize the network for the current policy. Prioritize the target policy network.

Images