An end-to-end cooperative vehicle-infrastructure integrated driving method and system with planning phase delay fusion

Abstract

The application belongs to the technical field of intelligent transportation and automatic driving, and discloses a kind of end-to-end vehicle-road cooperative driving method and system of planning phase delay fusion, comprising: constructing vehicle-road cooperative end-to-end driving network;Double-flow heterogeneous processing and historical time aggregation are carried out on the bird's eye view BEV space features of vehicle end and roadside;The spatial features compressed into vehicle end Token and roadside Token after double-flow aggregation are respectively;The target navigation intention is embedded, vehicle end Token, roadside Token and learnable planning query Token are spliced into a unified sequence, and input into a self-recurrent decoder, and under the constraint of hybrid causal mask, the adaptive fusion weight of vehicle end and roadside Token is learned through self-attention mechanism to generate future waypoint sequence;Two-stage training is carried out using training target containing waypoint regression loss and multi-expert model auxiliary loss.The application more directly optimizes closed-loop planning and control performance and improves communication utilization efficiency.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and autonomous driving technology, and particularly relates to an end-to-end vehicle-road cooperative driving method and system with delayed fusion in the planning stage. Background Technology

[0002] Vehicle-to-everything (V2X) cooperative autonomous driving can alleviate problems such as limited field of view of a single vehicle, blind spots caused by occlusion, and unpredictable behavior of traffic participants by sharing information between roadside units and vehicles. Existing V2X cooperative driving methods mostly adopt a two-stage paradigm of "perception stage fusion + single-vehicle planning": first, the features of multiple agents are fused in the perception / detection stage, and then the fused result is input into the single-vehicle planning module.

[0003] However, the above paradigm has at least the following shortcomings:

[0004] Mismatch in fusion timing: Fusion strategies in the perception phase are usually optimized around objectives such as detection / segmentation, and often use fixed or weakly adaptive weights, which makes it difficult to meet the changing importance of "scene-related" information in the planning task (for example, information on the roadside is critical at occluded intersections, but its value decreases on open roads).

[0005] Information redundancy and bottlenecks: Roadside areas have a wide field of view and a fixed perspective, which often contain a lot of background information. This information contributes little to the planning task, leading to a waste of communication and computing resources and creating an information bottleneck before planning.

[0006] Insufficient heterogeneous representation capability: There are heterogeneities between the vehicle and the roadside in terms of perspective, noise, and time delay. A single shared representation network is prone to "pattern averaging" and it is difficult to perform specialized modeling for different driving maneuvers and scenarios.

[0007] Therefore, there is a need for an end-to-end cooperative driving solution that can postpone the fusion of multi-agent information to the planning stage and can adaptively learn the fusion weights during the planning process, while taking into account both communication efficiency and robustness. Summary of the Invention

[0008] The core of this invention lies in proposing a "planning-phase delayed fusion" vehicle-to-everything (V2X) cooperative driving method and system for planning tasks: Unlike the fixed or weakly adaptive fusion of features of multiple agents in the perception phase, this invention postpones the fusion timing to the planning phase and learns scene-related adaptive fusion weights in the autoregressive decoding process, thereby more directly optimizing closed-loop planning and control performance and improving communication utilization efficiency.

[0009] The specific technical solution is as follows:

[0010] An end-to-end vehicle-road cooperative driving method with delayed fusion during the planning phase includes the following steps:

[0011] S1. Construct a vehicle-road cooperative end-to-end driving network to receive observation information from at least one vehicle-side agent and at least one roadside agent under bandwidth constraints;

[0012] S2. Perform dual-stream heterogeneous processing and historical time-series aggregation on the bird's-eye view BEV spatial features of the vehicle end and roadside, among which the roadside features are subjected to planning-oriented channel purification to filter out planning-irrelevant background information;

[0013] S3. The spatial features after the aggregation of the two streams are compressed into vehicle-side tokens and roadside tokens respectively. The roadside tokens are generated using a multi-expert model (MoE) structure to improve the representation ability of heterogeneous features and different maneuvering modes.

[0014] S4. The target navigation intent embedding, vehicle token, roadside token, and learnable planning query token are concatenated into a unified sequence and input into an autoregressive decoder. Under the constraint of hybrid causal mask, the adaptive fusion weights of vehicle and roadside tokens are learned through a self-attention mechanism to generate a future waypoint sequence.

[0015] S5. A two-stage training objective is adopted, which includes waypoint regression loss and multi-expert model-assisted loss, so that the network can directly optimize the planning task performance and maintain communication efficiency.

[0016] Furthermore, step S2 includes the following sub-steps:

[0017] S21. Obtain observation information from at least one vehicle-side intelligent agent and generate vehicle-side bird's-eye view features;

[0018] S22. Acquire observation information from at least one roadside agent and generate roadside bird's-eye view features;

[0019] S23. Complete message interaction between the vehicle and the roadside under bandwidth constraints, wherein the bandwidth constraints are achieved by limiting the number of tokens exchanged, the token dimension, or the message byte size, so that the communication load of each sending agent meets the preset bandwidth limit.

[0020] Furthermore, step S3 includes the following sub-steps:

[0021] S31. Perform historical temporal aggregation on the spatial features of the bird's-eye view images of the vehicle end and the roadside respectively. The historical temporal aggregation adopts a spatiotemporal joint coding network motion network. The input is the BEV features of the historical T-frame bird's-eye view image and occupancy or semantic raster information. The compressed motion perception features are output through three-dimensional convolution or equivalent temporal operators.

[0022] S32. In the roadside branch, perform channel cleanup on the roadside bird's-eye view features before temporal aggregation to suppress planning-irrelevant background information.

[0023] Furthermore, step S4 includes the following sub-steps:

[0024] S41. A tokenizer is used to perform token compression on the spatial features after temporal aggregation. This involves flattening the spatial grid features and adding positional encoding, then... A learnable query vector is obtained by multi-head attention pooling. One Token;

[0025] S42. A multi-expert model (MoE) structure is introduced in the roadside branch. For each token, the routing network selects Top-K experts to participate in the calculation, and the expert collapse or route collapse is suppressed by load balancing loss. The routing network also introduces maneuver mode guidance supervision. The maneuver mode is obtained by clustering the trajectory curvature or turning change, which is used to apply soft constraints to the route selection in the early stage of training and gradually weakens as the training progresses.

[0026] S43. Perform channel purification on the roadside bird's-eye view features. The channel purification includes: calculating the activation statistics of each channel of the roadside bird's-eye view features; selecting the Top-α% channels according to the activation statistics and retaining their values, and setting the remaining channels to zero to obtain purified features; inputting the purified features into a temporal aggregation network to obtain roadside motion perception features oriented towards planning, where α is a proportional parameter less than 1.

[0027] Furthermore, step S5 includes the following sub-steps:

[0028] S51. Concatenate the target navigation intent embedding, vehicle-side token, roadside token, and learnable planning query token into a unified sequence;

[0029] S52. Input the unified sequence into the autoregressive decoder and set a hybrid causal mask so that the target embedding and the context area where the dual-stream token is located enable bidirectional attention, while the planning area where the planning query token is located enables causal attention;

[0030] S53. Generate a future waypoint sequence based on the output of the autoregressive decoder.

[0031] The present invention also provides an end-to-end vehicle-road cooperative driving system with delayed fusion during the planning phase, comprising:

[0032] Multi-agent perception module: used to collect sensor data from the vehicle and roadside and generate bird's-eye view BEV features;

[0033] Dual-stream heterogeneous processing module: used to perform temporal aggregation of vehicle-side and roadside features respectively, including a roadside channel purification unit;

[0034] Token compression module: used to generate vehicle-side tokens and roadside tokens containing a multi-expert model (MoE) structure;

[0035] The planning phase fusion and decoding module is used to input the target embedding, dual-stream token, and planning query token into the autoregressive decoder and output future waypoints.

[0036] Training and Update Module: Used to train and update the system using waypoint regression loss and multi-expert model MoE auxiliary loss.

[0037] The technical effects of this invention are as follows:

[0038] 1. Delayed fusion during the planning phase: The fusion of features of multiple agents is postponed from the perception phase to the planning phase, so that the fusion weight is determined by the planning context and the historical planning state, thereby improving the collaborative effectiveness in key scenarios such as obscured intersections.

[0039] 2. Dual-stream heterogeneous feature processing and channel purification: For the heterogeneous perspectives and redundant backgrounds of the vehicle and roadside, a dual-stream processing pipeline is constructed, and channel purification is performed in the roadside branch to suppress planning-irrelevant background information and improve the density of effective communication information.

[0040] 3. Token Compression and Multi-Expert Model Enhanced Token Segmenter: Spatial features are compressed into compact tokens by the token segmenter to meet bandwidth constraints, and a multi-expert model (MoE) is introduced in the roadside branch to enhance the specialized representation capability for different traffic scenarios / driving maneuvering modes.

[0041] 4. Autoregressive Decoding Planning Based on Transformer: The target navigation intent, vehicle token, roadside token and planning query token are concatenated into a unified sequence. An autoregressive decoder with hybrid causal mask constraints is used to generate future waypoints, achieving end-to-end planning optimization and robust collaboration. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the overall framework of the present invention;

[0043] Figure 2 This is a comparative diagram of delayed fusion during the planning phase and fusion during the perception phase.

[0044] Figure 3 A schematic diagram of the structure of the word segmenter enhanced by the multi-expert model;

[0045] Figure 4 This is a schematic diagram of an autoregressive decoder and a hybrid causal mask. Detailed Implementation

[0046] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0047] Operator and symbol definition (for formula description)

[0048] To facilitate a formal description of the attention mechanism, positional encoding, and multi-expert (MoE) model structure of this invention, this section provides commonly used operator notation (which does not constitute a limitation on specific implementations):

[0049] Location coding: ,in This represents the position index matrix or its embedding matrix. The output is the positional embedding in the same dimension as the feature.

[0050] Bullish Attention: ,in , , These are respectively Query, Key, and Value.

[0051] Self-attention: , it is special case.

[0052] Cross attention: , it is The general form is (the query comes from one branch, and the key / value comes from another branch).

[0053] Feedforward networks: It is used to perform non-linear transformations on tokens.

[0054] Routed networks: It is used to output expert selection probabilities and perform Top-K routing in a multi-expert model structure.

[0055] In one example implementation, multi-head attention can be calculated as follows:

[0056]

[0057] in For the number of heads, For learnable parameters, The scaling factor dimension.

[0058] In one example implementation, the feedforward network can be written as:

[0059]

[0060] in It is a non-linear activation function.

[0061] In one example implementation, the routing for the multi-expert model (MoE) can be written as:

[0062]

[0063] Select the expert set corresponding to the Top-K probabilities. And perform weighted aggregation:

[0064]

[0065] in Indicates the first The transformation function corresponding to each expert network.

[0066] The specific method of this embodiment includes:

[0067] S1. Construct a planning-oriented end-to-end vehicle-to-infrastructure (V2X) cooperative driving architecture

[0068] like Figure 1 , Figure 2 As shown, this invention centers on the planning task, shifting the fusion of multiple agents from the traditional perception stage to the planning stage. The system includes vehicle-side agents and roadside agents, which output their respective bird's-eye view features and historical time-series information, and exchange compact messages (Tokens) under bandwidth constraints.

[0069] S2. Dual-stream heterogeneous treatment and historical time-series aggregation (including roadside channel purification)

[0070] like Figure 1 , Figure 3 As shown, this invention employs a dual-flow structure:

[0071] Vehicle-side stream: Time-series aggregation of features and historical information from the vehicle-side bird's-eye view to obtain motion perception features;

[0072] Roadside flow: First, the features of the roadside bird's-eye view are purified by channel purification, and then time-series aggregation is performed to obtain roadside motion perception features oriented towards planning.

[0073] S21. Roadside access road purification

[0074] Given that roadside features contain a large amount of background information unrelated to the current planning, this invention introduces a channel purification strategy before roadside time-series aggregation:

[0075] Calculate the global activation statistics for each channel (e.g., average over the spatial dimension).

[0076] Select and retain the channel with the highest activation rate (Top-α%), and set the other channels to zero.

[0077] The purified roadside features are input into a temporal aggregation network.

[0078] This strategy can suppress redundant backgrounds and improve the representation concentration of planning-related areas without significantly increasing computation.

[0079] S22. Temporal Aggregation Network

[0080] Temporal aggregation networks (such as motion networks) receive history The frame bird's-eye view features and occupancy / semantic grid are combined with spatiotemporal joint encoding to output compressed motion-aware features, enabling token generation to possess historical motion trend information.

[0081] S3. Heterogeneous Tokenization with Token Compression and Multi-Expert Model Enhancement

[0082] like Figure 3 As shown, this invention compresses the aggregated spatial features into compact tokens for communication and subsequent fusion.

[0083] S31. Attention Pooling Tokenizer

[0084] After performing dimensionality reduction projection, flattening, and adding position encoding to the spatial features, Each learnable query vector is pooled through multi-head attention to obtain a fixed number of tokens, thereby compressing large-scale spatial features into a small-scale token set and improving communication efficiency.

[0085] In one example implementation, let the flattened spatial features be: The location code is Then the position-encoded input is:

[0086]

[0087] Let the learnable query matrix of the word segmenter be... (Include (if there are query vectors), then the Token can be obtained as follows:

[0088]

[0089] in To output a set of tokens.

[0090] S32. Multi-expert model enhanced word segmenter (for roadside or heterogeneous scenarios)

[0091] To adapt to the heterogeneous perspectives of the vehicle and the roadside, as well as different scenarios / maneuvering modes, this invention introduces a multi-expert model structure in the feedforward / projection part of the word segmenter:

[0092] Set up multiple expert networks, and the routing network selects the Top-K experts to participate in the calculation for each Token;

[0093] Introducing load balancing loss mitigation to address uneven expert usage;

[0094] Introduce maneuver pattern guidance supervision (such as maneuver categories obtained from trajectory curvature clustering) to guide route learning in the early stages of training, and gradually reduce the guidance intensity as training anneals.

[0095] This design improves model capacity and specialization capabilities, avoiding the "compromise representation" of a single network in multimodal scenarios.

[0096] In one example implementation, for any token First, the expert probability is obtained through the routing network:

[0097]

[0098] Select Top-K experts The experts transformed the token (e.g., by...). constitute):

[0099]

[0100] in Indicates the first Each expert has a corresponding feedforward network. Through this structure, different experts can form specialized processing paths for different traffic scenarios or driving maneuvers.

[0101] S4. Delayed Fusion in the Planning Phase: Autoregressive Decoder and Hybrid Causal Mask

[0102] like Figure 4 As shown, the present invention performs delayed fusion during the planning stage.

[0103] S41. Token Sequence Construction

[0104] The target navigation intent, vehicle-side token, roadside token, and learnable planning query token are concatenated into a unified sequence in a fixed order:

[0105]

[0106] S42. Hybrid Causal Mask

[0107] Set a hybrid causal mask in the decoder attention:

[0108] The context area (target embedding + vehicle-side token + roadside token) allows for bidirectional attention to fully understand the scenario;

[0109] The planning area (future waypoint token) adopts causal attention, which makes each future waypoint only focus on historical waypoints and the entire context, thereby ensuring the temporal consistency of trajectory generation.

[0110] S43. Adaptive Fusion and Waypoint Generation

[0111] The autoregressive decoder dynamically aggregates vehicle-side and roadside token information on the planning query token using a self-attention mechanism, adaptively learning the importance weights of both, and outputting a sequence of future waypoints (e.g., 2D offsets or absolute coordinates). This mechanism allows the fusion weights to change with the scene, avoiding mismatches caused by fixed fusion in complex scenarios.

[0112] In one example implementation, the autoregressive decoder is a Transformer-based autoregressive decoding network (which may be a parallel autoregressive implementation), comprising several decoding layers; each decoding layer may include an attention sublayer and a feedforward sublayer, and is coupled with residual connections and normalization operators (e.g., layer normalization) to improve training stability. To avoid unnecessary limitations on specific implementations, this invention provides a formal description using attention and feedforward operators.

[0113] In one example implementation, the decoder receives a sequence of tokens as input to a certain layer. Self-attention can be used in the context region:

[0114]

[0115] Inquiry about planning areas Under the constraint of hybrid causal masking, attention aggregation of "context key / value" and "historical planning token" can be equivalently represented as:

[0116]

[0117] The output will be passed through a linear layer or Projection yields future waypoints:

[0118]

[0119] in This is the prediction result for the future waypoint sequence.

[0120] Furthermore, in one example implementation, the single-layer decoding structure can be abstracted as follows (without specifying the exact normalization order):

[0121]

[0122] The first formula corresponds to the attention sub-layer (which can be self-attention or cross-attention in a specific region), and the second formula corresponds to the feedforward sub-layer. By stacking multiple layers, deep fusion of multi-agent contexts and layer-by-layer refinement of planning tokens can be achieved.

[0123] In one example implementation, hybrid causal masking can be equivalently understood as adding the calculation of attention weights to the mask matrix. For example, for any attention sublayer:

[0124]

[0125] The context area corresponds to Bidirectional visibility is allowed, corresponding to the planning area. Satisfying causal visibility ensures the temporal consistency of future waypoint generation.

[0126] In one example implementation, the autoregressive generation can be implemented in a "stepwise generation" or "parallel autoregressive" manner: during stepwise generation, the first... Each waypoint depends only on the context and Historical waypoints; in parallel autoregression, multiple future waypoints can be output in a single forward propagation, but their attention dependencies are still constrained by a hybrid causal mask to satisfy the autoregressive condition. The above description is used to fully disclose the essence of "achieving adaptive fusion and generating waypoints through attention in the planning phase," without limiting the specific network depth, dimensions, or engineering implementation details.

[0127] S5. Training Objectives and Two-Stage Training Strategy

[0128] This invention employs a training method primarily focused on planning objectives:

[0129] Principal loss: Future waypoint regression loss;

[0130] Auxiliary losses: MoE routing guidance loss (annealable) and load balancing loss.

[0131] Two-stage training may include:

[0132] Phase 1: Freeze the pre-trained bird's-eye view encoder and train only the temporal aggregator, word segmenter, and decoder;

[0133] Phase 2: Optional end-to-end fine-tuning, using a smaller learning rate to balance stability and performance improvement.

[0134] Example 1: Collaborative Planning Scenario for Obstructed Intersections

[0135] In scenarios involving obstructed intersections and targets with low line-of-sight (such as pedestrians / cyclists crossing the road), roadside agents can detect potential conflict targets earlier. This invention improves the weight of roadside tokens at intersections through adaptive fusion during the planning phase, outputting a trajectory that yields first and then turns; after passing through the conflict zone, it automatically reduces the weight of roadside tokens, restoring smooth tracking primarily driven by the vehicle, thereby achieving higher safety and completion rates.

[0136] Example 2: Robustness to Communication Noise and Delay

[0137] In the event of noise or increased communication latency in roadside pose estimation, the autoregressive decoder of this invention can reduce the weight of inconsistent or expired roadside tokens through an attention mechanism, and rely more on vehicle-side tokens to generate trajectories, thus having more stable errors and fewer violations / collisions compared to fixed fusion schemes.

[0138] This invention can be applied to:

[0139] 1. Cooperative planning and control module in vehicle-road cooperative autonomous driving system (Level 3-L5 autonomous driving);

[0140] 2. Intelligent transportation infrastructure with roadside perception capabilities integrated with vehicle-side systems;

[0141] 3. Collaborative driving baseline / components for end-to-end autonomous driving R&D and simulation evaluation platform;

[0142] 4. Multi-agent perception-planning integrated deployment scenario under communication-constrained conditions.

[0143] Key points of deployment:

[0144] Message exchange using tokens facilitates controllable communication overhead under bandwidth constraints.

[0145] Delayed fusion during the planning phase enables the system to have stronger adaptability and security in complex scenarios such as occlusion, congestion, and intersections.

Claims

1. An end-to-end vehicle-road cooperative driving method with delayed fusion during the planning phase, characterized in that, Includes the following steps: S1. Construct a vehicle-road cooperative end-to-end driving network to receive observation information from at least one vehicle-side agent and at least one roadside agent under bandwidth constraints; S2. Perform dual-stream heterogeneous processing and historical time-series aggregation on the bird's-eye view BEV spatial features of the vehicle end and roadside, among which the roadside features are subjected to planning-oriented channel purification to filter out planning-irrelevant background information; S3. The spatial features after the aggregation of the two streams are compressed into vehicle-side tokens and roadside tokens respectively. The roadside tokens are generated using a multi-expert model (MoE) structure to improve the representation ability of heterogeneous features and different maneuvering modes. S4. The target navigation intent embedding, vehicle token, roadside token, and learnable planning query token are concatenated into a unified sequence and input into an autoregressive decoder. Under the constraint of hybrid causal mask, the adaptive fusion weights of vehicle and roadside tokens are learned through a self-attention mechanism to generate a future waypoint sequence. Step S4 includes the following sub-steps: S41. A tokenizer is used to perform token compression on the spatial features after temporal aggregation. This involves flattening the spatial grid features and adding positional encoding, then... A learnable query vector is obtained by multi-head attention pooling. One Token; S42. A multi-expert model (MoE) structure is introduced in the roadside branch. For each token, the top-K experts are selected through the routing network to participate in the calculation, and the expert collapse or route collapse is suppressed through load balancing loss. The routing network also introduces maneuver mode guidance supervision. The maneuver mode is obtained by clustering the trajectory curvature or turning change, which is used to apply soft constraints to the route selection in the early stage of training and gradually weakens as the training progresses. S43. Perform channel purification on the roadside bird's-eye view features. The channel purification includes: calculating the activation statistics of each channel of the roadside bird's-eye view features; selecting the Top-α% channels according to the activation statistics and retaining their values, and setting the remaining channels to zero to obtain purified features; inputting the purified features into a temporal aggregation network to obtain roadside motion perception features oriented towards planning, where α is a proportional parameter less than 1. S5. Two-stage training is performed using a training objective that includes waypoint regression loss and multi-expert model-assisted loss, enabling the network to directly optimize planning task performance while maintaining communication efficiency. Step S5 includes the following sub-steps: S51. Concatenate the target navigation intent embedding, vehicle-side token, roadside token, and learnable planning query token into a unified sequence; S52. Input the unified sequence into the autoregressive decoder and set a hybrid causal mask so that the target embedding and the context area where the dual-stream token is located enable bidirectional attention, while the planning area where the planning query token is located enables causal attention; S53. Generate a future waypoint sequence based on the output of the autoregressive decoder.

2. The end-to-end vehicle-road cooperative driving method with delayed fusion during the planning phase according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21. Obtain observation information from at least one vehicle-side intelligent agent and generate vehicle-side bird's-eye view features; S22. Acquire observation information from at least one roadside agent and generate roadside bird's-eye view features; S23. Complete message interaction between the vehicle and the roadside under bandwidth constraints, wherein the bandwidth constraints are achieved by limiting the number of tokens exchanged, the token dimension, or the message byte size, so that the communication load of each sending agent meets the preset bandwidth limit. .

3. The end-to-end vehicle-road cooperative driving method with delayed fusion during the planning phase according to claim 1, characterized in that: Step S3 includes the following sub-steps: S31. Perform historical temporal aggregation on the spatial features of the bird's-eye view images from the vehicle and roadside respectively. The historical temporal aggregation adopts a spatiotemporal joint coding network motion network, with historical data as the input. The frame bird's-eye view BEV features and occupancy or semantic raster information are used to output compressed motion-aware features through 3D convolution or equivalent temporal operators; S32. In the roadside branch, perform channel cleanup on the roadside bird's-eye view features before temporal aggregation to suppress planning-irrelevant background information.

4. An end-to-end vehicle-road cooperative driving system with delayed fusion during the planning phase, characterized in that, An end-to-end vehicle-road cooperative driving method for implementing a planning-phase delayed fusion as described in any one of claims 1 to 3; the system includes: Multi-agent perception module: used to collect sensor data from the vehicle and roadside and generate bird's-eye view BEV features; Dual-stream heterogeneous processing module: used to perform temporal aggregation of vehicle-side and roadside features respectively, including a roadside channel purification unit; Token compression module: used to generate vehicle-side tokens and roadside tokens containing a multi-expert model (MoE) structure; The planning phase fusion and decoding module is used to input the target embedding, dual-stream token, and planning query token into the autoregressive decoder and output future waypoints. Training and Update Module: Used to train and update the system using waypoint regression loss and multi-expert model MoE auxiliary loss.

Images