An end-to-end automatic driving method and system based on multi-modal attention fusion

Abstract

This invention relates to the fields of intelligent transportation and artificial intelligence technology, and discloses an end-to-end autonomous driving method and system based on multimodal attention fusion. The end-to-end autonomous driving method based on multimodal attention fusion includes: acquiring multimodal data; generating initial semantic text and calibrating the initial semantic text to form calibrated semantic text; performing feature processing on environmental images and point cloud data acquired by multi-view cameras, aligning the calibrated semantic text and first BEV features across modalities, and performing spatiotemporal feature modeling; generating candidate trajectories, quantifying the collision risk of each candidate trajectory with dynamic obstacles, selecting low-risk trajectories, and guiding the optimization of low-risk trajectories through rule masks; and mapping the optimized trajectory to control commands. This invention utilizes the CLIP cross-modal alignment mechanism to deeply fuse text features with BEV geometric features, improving the recognition accuracy of key areas.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and artificial intelligence technology, and more specifically, to an end-to-end autonomous driving method and system based on multimodal attention fusion. Background Technology

[0002] In the field of autonomous driving, environmental perception is a core prerequisite for achieving safe autonomous driving. The key lies in the accurate perception of the three-dimensional geometric structure and semantic information of objects in traffic scenarios. Among the current mainstream technical approaches, perception models based on bird's-eye view (BEV) (such as Lift-Splat-Shoot) can construct environmental geometric representations through multi-view image projection, but they lack explicit semantic annotations. This leads to fuzzy recognition of the intentions of traffic participants such as pedestrians and vehicles, and makes it difficult to accurately capture the correlation between key semantic commands such as "stop at a red light" and "pedestrian crossing the street" and their spatial locations.

[0003] Existing multimodal fusion methods (such as the cross-attention mechanism based on Transformer) attempt to combine visual images and text commands, but they adopt a fixed weight allocation strategy, which cannot dynamically adapt to the modal emphasis requirements in complex scenarios (such as intersections needing to rely on text commands first, while highways need to focus on visual perception), resulting in low feature fusion efficiency and limited model generalization ability.

[0004] Furthermore, most end-to-end autonomous driving systems only focus on the features of the current frame and ignore the temporal correlation of historical trajectories, which makes the prediction of the motion trend of dynamic obstacles (such as vehicles changing lanes and pedestrians suddenly crossing) inaccurate. The lack of spatiotemporal continuity modeling directly affects the safety and rationality of trajectory planning.

[0005] To address the aforementioned issues, traditional solutions, lacking deep coupling between semantic information and geometric features, dynamic modal weight adjustment mechanisms, and spatiotemporal sequence modeling capabilities, struggle to meet the high-precision decision-making requirements in complex urban traffic scenarios. Therefore, there is an urgent need to propose an end-to-end autonomous driving method that integrates semantic enhancement, dynamic attention mechanisms, and spatiotemporal continuity to improve the system's understanding and response capabilities in complex scenarios. Summary of the Invention

[0006] This invention provides an end-to-end autonomous driving method and system based on multimodal attention fusion, which solves the technical problems of insufficient semantic understanding, low modal fusion efficiency and lack of spatiotemporal continuity in related technologies.

[0007] This invention provides an end-to-end autonomous driving method based on multimodal attention fusion, comprising:

[0008] Real-time acquisition of high-definition images of the road ahead and driving intention data to generate initial semantic text; calibration of the initial semantic text to form semantic text to be processed.

[0009] The first BEV feature is generated by performing feature processing on environmental images acquired by multi-view cameras and point cloud data acquired by LiDAR. The CLIP model is used to perform cross-modal alignment between the semantic text to be processed and the first BEV feature, and spatiotemporal feature modeling is performed to output the second BEV feature that integrates spatiotemporal information.

[0010] The second BEV feature is input into a conditional variational autoencoder to generate candidate trajectories. A probability occupancy field is constructed based on a Gaussian mixture model. The collision risk of each candidate trajectory with dynamic obstacles is quantified, low-risk trajectories are selected, a rule mask is generated based on the semantic text to be processed, and the optimization of the low-risk trajectories is guided by the rule mask.

[0011] The optimized trajectory is mapped into control commands to achieve end-to-end autonomous driving control.

[0012] Furthermore, real-time acquisition of high-definition images of the road ahead and driving intention data to generate initial semantic text, followed by calibration of the initial semantic text to form the semantic text to be processed, including:

[0013] High-definition images of the road ahead and driving intention data are collected via a forward-facing camera and a CAN bus, respectively.

[0014] The GroundTruth tag information is recorded synchronously during the data collection process.

[0015] The high-resolution image of the road ahead, driving intention data, and GroundTruth labels are input into the BLIP-2 model to generate initial semantic text. The initial semantic text is then semantically calibrated using an MLP network to suppress the illusion problem of the visual language model, and the output is unprocessed semantic text containing key semantic entities and their spatial locations.

[0016] Furthermore, the semantic calibration process is as follows:

[0017] ;

[0018] in This represents the calibrated text semantic feature vector; It is the ReLU activation function; and These represent the weight matrices of the first fully connected layer and the second fully connected layer, respectively. Represents the feature vector of the input image; and These represent the first-layer bias vector and the second-layer bias vector, respectively.

[0019] Furthermore, the cross-modal alignment of the semantic text to be processed and the first BEV feature using the CLIP model includes: encoding the semantic text to be processed using the text encoder of the CLIP model to generate a high-dimensional text feature vector; reducing the dimensionality of the text feature to 77 dimensions through a linear layer, freezing the CLIP visual encoder and retaining only the text encoder parameters, and mapping the dimensionality-reduced text feature to the BEV feature space; and using a learnable residual modulation module to perform weighted fusion of the dimensionality-reduced text feature and the first BEV feature at the feature channel level to achieve explicit alignment of semantic and geometric features.

[0020] Furthermore, the spatiotemporal feature modeling includes: transforming the historical third BEV features of the previous 3 frames to the current coordinate system through a relative transformation matrix to eliminate spatial displacement deviation between different frames; inputting the aligned historical third BEV features of the previous 3 frames and the current third BEV features into a 6-layer spatiotemporal Transformer, with each layer employing an 8-head self-attention mechanism to capture spatiotemporal dependencies layer by layer, and outputting a second BEV feature that fuses spatiotemporal information.

[0021] Furthermore, the candidate trajectory is generated by using a conditional variational autoencoder to generate 10 sets of candidate trajectories, and the trajectory diversity is constrained by KL divergence loss to cover driving intention.

[0022] Furthermore, the text features are reduced to 77 dimensions using a linear layer:

[0023] ;

[0024] in This represents a high-dimensional feature vector generated after the calibrated semantic text is encoded by the CLIP text encoder. This represents the 77-dimensional text feature vector output after dimensionality reduction; This represents the linear projection weight matrix, with dimension 1. ; This represents the linear projection bias vector, which has 77 dimensions. This represents the original feature dimension output by the CLIP text encoder.

[0025] Furthermore, the process of weighted fusion of the dimensionality-reduced text features and the first BEV feature at the feature channel level can be represented as follows:

[0026] ;

[0027] in Indicates the third BEV characteristic, Scaling factor The residual weight matrix is... This represents the 77-dimensional text feature vector output after dimensionality reduction. Represents the Hadamard product. Presentation layer normalization operation.

[0028] Furthermore, the optimization of the low-risk trajectory guided by the rule mask includes generating a rule mask based on the semantic text to be processed. The rule mask uses a neural network to convert the regulatory constraint information in the semantic text to be processed into quantifiable mask values. Based on the generated rule mask, the trajectory parameters of the low-risk trajectory are dynamically adjusted.

[0029] ;

[0030] ;

[0031] in This represents the curvature after dynamic adjustment under the influence of the rule mask. For the original curvature, For safety curvature, For the original speed, The speed is dynamically adjusted under the rule mask. Represents the rule mask.

[0032] This invention provides an end-to-end autonomous driving system based on multimodal attention fusion, used to execute the aforementioned end-to-end autonomous driving method based on multimodal attention fusion, comprising:

[0033] The data acquisition and visual semantic feature generation module is used to acquire high-definition images of the road ahead and driving intention data in real time and generate initial semantic text, and to calibrate the initial semantic text to form semantic text to be processed.

[0034] The BEV feature generation and spatiotemporal fusion module is used to perform feature processing on environmental images collected by multi-view cameras and point cloud data collected by lidar to generate the first BEV feature. The CLIP model is used to perform cross-modal alignment of the semantic text to be processed and the first BEV feature, and to perform spatiotemporal feature modeling to output the second BEV feature that fuses spatiotemporal information.

[0035] The multimodal trajectory prediction and safety planning module is used to input the second BEV feature into the conditional variational autoencoder to generate candidate trajectories, construct a probability occupancy field based on a Gaussian mixture model, quantify the collision risk between each candidate trajectory and dynamic obstacles, filter out low-risk trajectories, generate a rule mask based on the semantic text to be processed, and guide the optimization of the low-risk trajectories through the rule mask.

[0036] The control command output module is used to map the optimized trajectory into control commands to achieve end-to-end autonomous driving control.

[0037] The beneficial effects of this invention are: by generating explicit semantic annotations through the BLIP-2 model and the MLP calibration layer, the semantic ambiguity problem of traditional BEV models is solved;

[0038] By leveraging the CLIP cross-modal alignment mechanism, text features are deeply fused with BEV geometric features to improve the recognition accuracy of key areas (such as pedestrians and traffic signals).

[0039] Multimodal candidate trajectories are generated based on CVAE, and collision risk is quantitatively assessed by combining probabilistic occupancy fields. The trajectory parameters are dynamically adjusted through a semantic text-driven rule masking engine to ensure driving compliance and safety. Attached Figure Description

[0040] Figure 1 This is a flowchart of an end-to-end autonomous driving method based on multimodal attention fusion in this invention;

[0041] Figure 2 This is a framework diagram of the semantic enhancement module based on VLM driver of the present invention;

[0042] Figure 3 This is a framework diagram of the trajectory generation strategy module based on probabilistic programming and dynamic rule constraints of the present invention;

[0043] Figure 4 This is a diagram showing the actual vehicle operation effect of the invention. Detailed Implementation

[0044] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0045] At least one embodiment of the present invention discloses an end-to-end autonomous driving method based on multimodal attention fusion, such as... Figure 1 As shown, it includes:

[0046] Step 1: Real-time acquisition of high-definition images of the road ahead and driving intention data;

[0047] Step 1.1, Image Data Acquisition;

[0048] For the The frame is formed by simultaneously capturing multi-view images of the target scene from eight surround-view cameras and stitching them together. Frame multi-view image Its dimensions are ,in Indicates the image height (in pixels); 3 represents the image width (pixels); 3 represents the number of RGB channels.

[0049] The multi-view camera setup includes eight perspectives: front, left front, right front, left side, right side, rear, left rear, and right rear, enabling full 360-degree coverage of the vehicle's surroundings.

[0050] Meanwhile, a dedicated forward-looking camera acquires high-definition images of the road ahead, with a resolution of 1920×1080 pixels, for the precise identification of key targets such as traffic signals, pedestrians, and lane lines. The feature vectors generated from the high-definition images of the road ahead acquired by the forward-looking camera are preprocessed by a visual encoder. Visual input used for semantic text generation.

[0051] Step 1.2, Point cloud data acquisition;

[0052] The lidar synchronously acquires point cloud data and outputs the first... Frame point cloud sequence:

[0053] ;

[0054] in Indicates the first Frame point cloud sequence, Indicates the number of point clouds; This represents the number of point cloud information points at a single point, including coordinates. and reflection intensity ; Represents the real number field;

[0055] The point cloud data uses a 64-line lidar with a scanning frequency of 10Hz, a ranging range of 200 meters, and an angular resolution of 0.1 degrees, providing the system with accurate spatial geometric information.

[0056] The original point cloud is feature-encoded using a 3D convolutional network to generate a point cloud feature volume that includes spatial structure and reflection properties. The focus is on enhancing the feature representation of distant obstacles (such as vehicles 100 meters away) and low-reflectivity targets (such as pedestrians and bicycles), among which... Represents point cloud feature volumes; Represents the real number field.

[0057] Step 1.3, Dynamic adaptation of sensor modes;

[0058] A modal weight adaptive module is introduced to monitor the environmental adaptability of multi-view cameras in real time. This module dynamically calculates the confidence weights of each sensor by analyzing metrics such as image clarity of the forward-looking camera in strong light and heavy rain scenarios, and the confidence level of the side-looking camera in recognizing nearby obstacles. An attention mechanism automatically assigns fusion weights to the features of each sensor, ensuring optimal perception performance under different environmental conditions.

[0059] Step 1.4, labeling vehicle bus data and information;

[0060] In addition, the system also simultaneously collects onboard bus data and annotation information to provide contextual support for subsequent semantic generation. Specifically, this includes driving intention data, which is collected in real time via the onboard CAN bus, including vehicle steering angle. acceleration Speed Vehicle operating status parameters reflect the current driving intention and serve as contextual supplements for semantic text generation; GroundTruth tags are annotation information recorded synchronously during the data collection phase, including manually annotated semantic descriptions, real trajectories of dynamic obstacles, and compliant driving trajectories, used for supervised calibration during model training.

[0061] Step 2: Generate initial semantic text, and calibrate the initial semantic text to form semantic text to be processed;

[0062] This step enables entity extraction and text generation at the primary semantic layer, laying the foundation for subsequent behavior inference and intent prediction.

[0063] Step 2.1, Initial text generation;

[0064] Initial semantic text is generated based on the BLIP-2 model in the Visual Language Model (VLM). The input to the BLIP-2 model includes front view image features. Driving intention data and Ground Truth tags ;

[0065] Initial semantic text output Includes descriptions of key traffic entities in the scene, such as "There is a pedestrian crossing the zebra crossing 10 meters ahead".

[0066] Step 2.2, semantic calibration;

[0067] The initial semantic text is processed using a Multi-Layer Perceptron (MLP) network. Calibration is performed to suppress the "illusion" problem of visual language models, and the output is the semantic text to be processed containing key semantic entities and their spatial locations. Let the MLP input be image features. The semantic calibration process is as follows:

[0068] ;

[0069] in This represents the calibrated text semantic feature vector; It is the ReLU activation function; and These represent the weight matrices of the first fully connected layer and the second fully connected layer, respectively. and These represent the first-layer bias vector and the second-layer bias vector, respectively.

[0070] Specifically, , , , ,in, Represents the feature vector of the input image Dimensions This represents the dimension of the first hidden unit in the MLP layer, i.e., the dimension of the intermediate feature space. Represents the calibrated text semantic feature vector The dimension, that is, the dimension of the output features of the second layer of the MLP network.

[0071] Calibrated text semantic feature vector Dimension of the input image feature vector MLP first layer hidden unit dimension The mapping relationship that together constitutes the feature dimensions: through the first fully connected layer... Mapped to Then, through a second fully connected layer, Mapped to Ultimately, this achieves dimensional adaptation from image features to text semantic features, providing a foundation for dimensional consistency in subsequent cross-modal fusion of text features and BEV geometric features. Specifically, the calibrated semantic text contains key entities (such as "pedestrian" and "red light") and their precise spatial locations (such as "20 meters to the left").

[0072] Step 3: Based on the environmental images collected by the multi-view camera and the point cloud data collected by the lidar, perform feature processing to generate the first BEV feature. Use the CLIP model to perform cross-modal alignment between the semantic text to be processed and the first BEV feature, and perform spatiotemporal feature modeling to output the second BEV feature that integrates spatiotemporal information.

[0073] This step achieves deep coupling between visual geometric features and semantic features, constructing a unified BEV representation space.

[0074] Step 3.1, Feature processing;

[0075] Environmental images captured by multi-view cameras undergo feature extraction via a visual encoder. First, basic features are extracted using shared convolutional layers. Then, features are extracted from the environmental images captured by the multi-view cameras through three convolutional layers. The layer output is:

[0076] ;

[0077] in Indicates the first The output feature map of each convolutional layer has its dimensions gradually compressed with each convolutional operation (finally, after 3 convolutional layers, the output dimension is...). ), containing multi-view images via the first Abstract features after layer processing (such as edges, textures, local semantics, etc.). Indicates the convolutional layer number ( ), corresponding to a layer of 3 shared convolutional layers; This represents the activation function, used to introduce nonlinear feature transformations to enhance the network's ability to fit complex patterns; Indicates the first The output feature map of the first convolutional layer is used as the first... The input of the layer, and when hour, That is, the first data collected in step 1.1 Frame multi-view image (dimension) ); Indicates the first The weight matrix of the convolutional layer has dimensions of . ( The kernel size is [size]. Input the number of channels. (Number of output channels), used to extract local features at different scales; Indicates the first The bias vector of a convolutional layer, with dimension and The number of channels is consistent, which is used to compensate for the offset of the convolution operation and improve the accuracy of feature extraction.

[0078] After extraction using three convolutional layers, deformable convolution is used to enhance edge details, and the output feature map is then generated. ,in Represents the feature map; Represents the real number field;

[0079] Receive deformable convolution-enhanced feature maps As input, a 3D feature volume is generated using a monocular depth model. 3D feature volume via camera intrinsic parameter matrix Projected into the BEV space, the resolution is Visual BEV features are generated through attention aggregation. ;

[0080] After environmental image feature processing acquired by multi-view cameras, the point cloud feature volume is then processed using the lidar extrinsic matrix. Projected into BEV space (resolution consistent with visual BEV characteristics, both being...) Generate point cloud BEV features ;

[0081] Introducing cross-modal attention weight matrix Dynamically adjust visual BEV features BEV features of point cloud The fusion ratio is expressed as:

[0082] ;

[0083] in This represents the cross-modal attention weight matrix, which is dynamically updated based on the environment confidence level. and These represent visual BEV features and point cloud visual BEV features, respectively. This represents the first BEV feature.

[0084] Step 3.2, cross-modal alignment of text features;

[0085] The semantic text is encoded using the CLIP model's text encoder, and the semantic text to be processed is... High-dimensional features are output by CLIP text encoder , Dimensions (The CLIP model predefines text feature dimensions, such as 512 or 768 dimensions), containing deep semantic information of the semantic text. This represents the original feature dimension (predefined parameter of the model) output by the CLIP text encoder and is The source of the number of columns in a matrix. The original feature dimension is the output of the CLIP text encoder in the real number field.

[0086] The text features are reduced to 77 dimensions using a linear layer. The CLIP visual encoder is frozen, retaining only the text encoder parameters. The dimensionality-reduced text features are then mapped to the BEV feature space. The high-dimensional text features output by the CLIP text encoder are converted into 77-dimensional feature vectors. The formula for mapping and adapting text features to the BEV feature space is as follows:

[0087] ;

[0088] in This represents a high-dimensional feature vector generated after the calibrated semantic text is encoded by the CLIP text encoder. This represents the 77-dimensional text feature vector output after dimensionality reduction; This represents the linear projection weight matrix, with dimension 1. ; This represents the linear projection bias vector, which has 77 dimensions. This represents the original feature dimension output by the CLIP text encoder.

[0089] Cross-modal alignment of semantic and geometric features can be achieved by fusing the visual BEV features with the subsequent residual modulation module; High-dimensional... The dimensionality is reduced to 77 dimensions to achieve adaptation of the feature dimension to the BEV feature space; It is used to compensate for the offset during the linear transformation process and improve the accuracy of the representation of the features after dimensionality reduction.

[0090] By using a learnable residual modulation module, text features are mapped to the BEV feature space, thus realizing the mapping between semantic features and the first BEV feature. Cross-modal weighted fusion, which fuses text features with visual BEV features, is specifically represented as follows:

[0091] ;

[0092] in Indicates the third BEV characteristic, Scaling factor The residual weight matrix is... This represents the 77-dimensional text feature vector output after dimensionality reduction. Represents the Hadamard product. Presentation layer normalization operation.

[0093] Step 3.3, Spatiotemporal feature modeling;

[0094] Introducing a spatiotemporal Transformer to transform the current third BEV feature The third BEV feature from the previous three historical frames is aligned to the current coordinate system using a relative transformation matrix (4×4). Spatiotemporal cross-attention is calculated through six Transformer layers (eight self-attention heads per layer) to capture the motion trends and spatiotemporal dependencies of vehicles and pedestrians.

[0095] Historical feature alignment: Third BEV feature from the first 3 frames Through relative transformation matrix Align to the current coordinate system:

[0096] ;

[0097] in Indicates the first The original history of the frame, third BEV feature ( (corresponding to the previous 1 frame, the previous 2 frames, and the previous 3 frames respectively). Indicates the first Frame to current frame (the first frame) The relative transformation matrix of the frame, with dimension . (Homogeneous coordinate transformation matrix), which includes spatial transformation parameters such as translation and rotation, is used to correct the positional deviation between historical features and the current frame; Indicates the first after alignment The historical third BEV feature of the frame has been transformed to the current frame coordinate system and is on the same spatial reference as the current BEV feature, so it can be directly involved in spatiotemporal correlation calculation.

[0098] During spatiotemporal Tranformer processing, the current BEV features are concatenated with aligned historical features to form the input sequence.

[0099] ;

[0100] in The input sequence of the Transformer is represented by the third BEV feature. and the aligned first 3 frames of history third BEV feature , and It is pieced together, with dimensions of (4 indicates 4 frames of features,) , , These are the number of feature channels, height, and width, respectively. Indicates the third BEV characteristic; , and These represent the historical third BEV features of the previous 1 frame, the previous 2 frames, and the previous 3 frames, respectively;

[0101] Through 6 layers of Tranformer, with 8 heads of self-attention per layer:

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] in Indicates attention head index and Each layer has eight parallel attention heads, used to capture feature associations from different subspaces. , , They represent the first The query, key, and value matrices for each attention head are respectively composed of the input sequence. With learnable weight matrix , , Multiplying them together, we get the dimension as follows: ( For sequence length, (feature dimensions for each attention head) , , They represent the first The query, key, and value projection matrices of each attention head are used to map the input features to the attention subspace, with dimensions of [missing information]. ( for (Number of channels); Indicates the first The attention weight matrix of each attention head, through and transpose of dot product, scaling (divided by) The values ​​are obtained by combining the Softmax activation and the larger the value, the stronger the correlation between the corresponding location features. This represents the feature dimension of each attention head, used to scale the dot product result and avoid numerical overflow or gradient vanishing due to excessive dimensionality. Indicates the transpose symbol; This indicates a splicing operation, combining the outputs of 8 attention heads. Concatenate according to channel dimension; This represents the output projection matrix, used to map the concatenated features to the target dimension; The motion trend features represented by the spatiotemporal Transformer output are integrated with the spatiotemporal dependencies of each frame and position in the input sequence. Finally, after being stacked with 6 Transformer layers, the output BEV features contain long-term temporal dynamic information.

[0108] The output of this step is the second BEV feature, which integrates spatiotemporal information, and serves as the basic environmental representation for trajectory prediction and planning.

[0109] Step 4: Input the second BEV feature into the conditional variational autoencoder to generate candidate trajectories, construct a probability occupancy field based on the Gaussian mixture model, quantify the collision risk of each candidate trajectory with dynamic obstacles, filter out low-risk trajectories, generate a rule mask based on the semantic text to be processed, and guide the optimization of the low-risk trajectories through the rule mask.

[0110] This step implements a trajectory generation strategy based on probabilistic programming and dynamic rule constraints, and at the same time constitutes a high-level intent layer in a multi-level semantic reasoning framework. It integrates scene context and historical trajectory data to predict the long-term intent of traffic entities, ensuring the diversity, safety and regulatory compliance of trajectories.

[0111] Step 4.1, candidate trajectory generation;

[0112] Ten candidate trajectories are generated using a Conditional Variational Autoencoder (CVAE). KL divergence loss is used to constrain trajectory diversity, covering various possible driving intentions. Step 3 uses the spatiotemporal Transformer module output, which contains multimodal fusion environmental information (geometric structure, semantic rules, spatiotemporal dynamics), directly as the conditional input to the CVAE model, achieving the mapping from "environmental perception features to driving trajectory." After flattening, this information is input into the CVAE encoder, providing complete scene context constraints for trajectory generation.

[0113] encoder with As input, a multi-layer neural network learns the mapping relationship between scene features and the distribution of latent variables, and outputs the mean of the latent variables. and standard deviation This process compresses high-dimensional environmental features into low-dimensional latent variable probability distribution parameters, making the latent variables... The sampling can reflect scene specificity (e.g., the latent variable distribution of a low-curvature trajectory corresponds to a straight-line scene, while the distribution of a high-curvature trajectory corresponds to a curved-line scene), which is represented in the encoding stage as:

[0114] ;

[0115] in The mean vector representing the distribution of latent variables reflects the central tendency of a reasonable driving trajectory in the current scenario; The standard deviation vector represents the distribution of latent variables and controls trajectory diversity; This represents a latent variable sampled from a Gaussian distribution, whose randomness is the core of generating diverse trajectories; Indicated by For the mean, The covariance is a Gaussian distribution, and the variables are latent variables. The sampling distribution; Represents the identity matrix, used to define the covariance structure of the Gaussian distribution; Indicates encoder; This represents the motion trend characteristics output by the spatiotemporal Transformer.

[0116] From Gaussian distribution Mid-sampled latent variables Its randomness is determined by the standard deviation vector of the latent variable distribution. control( The larger the value, the stronger the sampling diversity. This step is the core of achieving trajectory diversity: the same... (The same scenario) can be achieved through different Generate differentiated trajectories (such as left turns, straight-ahead maneuvers, and deceleration / yielding maneuvers at an intersection), and This ensures that the trajectory distribution revolves around the scene and the intended focus.

[0117] During the decoding phase, the decoder uses... As input, generate candidate trajectories , is represented as:

[0118] ;

[0119] in This represents the candidate trajectory output by the decoder, which includes parameters such as position, curvature, and velocity in the time series. The decoder network representing CVAE will contain latent variables. Mapped to specific candidate trajectories;

[0120] The decoder learns latent variables The mapping with trajectory parameters (position, curvature, velocity) transforms the randomness of latent variables into a specific trajectory shape, and because of the latent variables... by Constraints, generated To adapt to the current scenario, the total loss of CVAE is expressed as:

[0121] ;

[0122] ;

[0123] ;

[0124] in This represents the total loss function of CVAE, balancing the rationality and diversity of trajectories. Represents the reconstruction loss, measuring the generated trajectory. With real driving trajectory Differences; This represents the actual driving trajectory (the compliant trajectory in the GroundTruth label). This represents the number of trajectory points, used to normalize the reconstruction loss; The weighting coefficients of the KL loss are used to balance "trajectory rationality" and "diversity". This represents the KL divergence loss, which constrains the difference between the latent variable distribution and the standard Gaussian distribution to avoid trajectory pattern collapse. Represents the logarithmic symbol. The summation symbol is used.

[0125] In the total loss of CVAE, reconstruction loss (measure With the actual trajectory (Differences) ensure that the generated trajectory conforms to human driving habits; KL divergence loss (The deviation between the latent variable distribution and the standard Gaussian distribution is constrained by controlling the mean.) and the standard deviation vector of the latent variable distribution The value of [value] is chosen to avoid trajectory pattern collapse (e.g., generating only a single type of trajectory). Both work together to achieve the trajectory generation goal of "diversity (dominated by KL loss) + rationality (dominated by reconstruction loss)," where the weights [are crucial]. For balancing: In complex scenarios (such as intersections), the weighting coefficient of the KL loss can be increased. To enhance diversity, the weighting coefficients for KL loss can be reduced in simple scenarios (such as high-speed straight roads). To ensure trajectory stability.

[0126] Step 4.2, Collision Risk Assessment;

[0127] By combining the motion vectors of the previous three frame point cloud sequences, Kalman filtering is used to correct the positional deviation of the BEV features in the current frame point cloud, ensuring the trajectory continuity of the dynamic target. This processing method complements the spatiotemporal Transformer module of the visual BEV features in step 3.3, providing high-confidence trajectory information for collision risk assessment of dynamic obstacles.

[0128] A probabilistic occupancy field is constructed based on a Gaussian mixture model (GMM) to quantify the collision risk between each candidate trajectory and dynamic obstacles, and to screen low-risk trajectories.

[0129] ;

[0130] in Indicates the position in BEV space The probability that a location is occupied by a dynamic obstacle, with a value ranging from 0 to 1. The larger the value, the higher the risk of an obstacle at that location. It is used to quantify the probability of a collision between a candidate trajectory and an obstacle. Indicates the first The contribution of each obstacle to the overall occupancy probability distribution; Indicates the first The covariance matrix corresponding to each obstacle The determinant is used to quantify the range of uncertainty in the location of obstacles (the larger the determinant value, the more dispersed the distribution of obstacle locations). Indicates the first The mean position vector of each obstacle, i.e. the center position of the obstacle (obtained by spatiotemporal feature tracking in step 3); Represents the covariance matrix The inverse matrix is ​​used to calculate the exponential term of the Gaussian distribution and quantize the position. Center of the obstacle The degree of deviation; Indicates the summation symbol; Indicates the number of dynamic obstacles in the environment. This represents the natural exponential function, used to describe the process of exponential growth or decay, as well as some distributions associated with probability density functions.

[0131] Step 4.3, rule-guided optimization;

[0132] Generate a rule mask based on the semantic text to be processed. , rule mask The generation is based on the semantic text to be processed. (Including key traffic rule-related entities and spatial locations such as "red light" and "pedestrian crossing"), the semantic text to be processed The regulatory constraints in the text are converted into quantifiable mask values ​​(ranging from 0 to 1). For example, when processing semantic text... When a "red light" is detected, the network generates a high mask value (approaching 1) to strengthen deceleration constraints; when no special rules are in place, it generates a low mask value (approaching 0), achieving dynamic matching between the strength of rule constraints and the semantic scene. Based on the generated rule mask, the trajectory parameters of low-risk trajectories are dynamically adjusted.

[0133] ;

[0134] ;

[0135] in This represents the curvature after dynamic adjustment under the influence of the rule mask. For the original curvature, For safety curvature, For the original speed, The speed is dynamically adjusted under the rule mask. Represents the rule mask.

[0136] Original curvature The curvature parameter is the candidate trajectory generated by the conditional variational autoencoder in step 4.1. It reflects the initial curvature of the trajectory (e.g., the curvature of the straight trajectory is close to 0, and the curvature of the turning trajectory is positive or negative, depending on the turning direction). Its value is automatically generated by the CVAE model based on the input fusion spatiotemporal features and is directly related to the original driving intention of the candidate trajectory (e.g., left turn, right turn, avoidance, etc.), without the need for manual preset.

[0137] Safety curvature It is a curvature threshold preset based on traffic regulations, road safety standards, and scene characteristics, used to ensure trajectory safety in rule-constrained scenarios (such as encountering pedestrians or turning at intersections). Its value is determined through offline calibration and scene binding.

[0138] For example, when driving straight through an unobstructed section of road, the safety curvature Set to a minimum value close to 0 (to ensure a smooth trajectory); when encountering pedestrian crossing scenarios, the safety curvature is... Set it to a fixed value less than the original turning curvature (e.g., within 0.02 rad / m for urban roads); for intersection turning scenarios, refer to road design specifications (e.g., the curvature value corresponding to the minimum turning radius of urban main roads).

[0139] In practical applications, safety curvature Through the semantic text to be processed The scene tags (such as "pedestrian crossing" and "T-junction") dynamically call the preset parameter table to achieve accurate matching with the scene.

[0140] Step 5: Map the optimized trajectory into control commands to achieve end-to-end autonomous driving control;

[0141] The optimized trajectory is mapped to steering wheel angle and accelerator / brake commands, with the goal of minimizing L2 displacement error (compared to human driving trajectory) and achieving zero collisions, thus realizing end-to-end autonomous driving control.

[0142] Control commands are generated using Model Predictive Control (MPC), with the objective function being:

[0143] ;

[0144] Constraints:

[0145] ;

[0146] in: Indicates the control input Minimization optimization is performed, that is, finding the optimal control command that minimizes the objective function; Indicates the first Control input at any given time refers to the specific control commands given to the vehicle. The summation symbol is used to represent the summation symbol. The predicted steps for MPC cover the planning control cycle; Indicates the first The actual state of the vehicle at any given moment, including the vehicle's position (coordinates), speed, heading angle, and other state quantities that reflect the vehicle's motion; Indicates the first The reference trajectory state at time t, i.e., the optimized trajectory output in step 4 at time t. The desired state at any given time is used as the control objective; The square of the Euclidean norm is used to represent quantization error. Indicates the first The deviation between the actual state of the vehicle and the state of the reference trajectory at any given time is defined, and the goal is to minimize this deviation in order to track and optimize the trajectory. Indicates the first The input size is constantly controlled to constrain the intensity of control commands (such as avoiding sudden steering wheel turns or sudden braking). This represents the weighting coefficient, used to balance trajectory tracking accuracy and control smoothness. Increasing the weighting coefficient indicates a significant improvement. It can make control commands smoother (reducing sudden acceleration / deceleration) and decrease... Priority can be given to ensuring the accuracy of trajectory tracking; Represents the vehicle kinematic model, describing the first The vehicle status at time t is determined by the first state of time and control input Joint decision; , These represent the minimum and maximum values ​​of the control input, respectively, which are physical constraints to ensure that the control commands are within the range that the vehicle hardware can execute.

[0147] An end-to-end autonomous driving system based on multimodal attention fusion, used to execute the aforementioned end-to-end autonomous driving method based on multimodal attention fusion, includes:

[0148] The data acquisition and visual semantic feature generation module is used to acquire high-definition images of the road ahead and driving intention data in real time and generate initial semantic text, and to calibrate the initial semantic text to form semantic text to be processed.

[0149] The BEV feature generation and spatiotemporal fusion module is used to perform feature processing on environmental images collected by multi-view cameras and point cloud data collected by lidar to generate the first BEV feature. The CLIP model is used to perform cross-modal alignment of the semantic text to be processed and the first BEV feature, and to perform spatiotemporal feature modeling to output the second BEV feature that fuses spatiotemporal information.

[0150] The multimodal trajectory prediction and safety planning module is used to input the second BEV feature into the conditional variational autoencoder to generate candidate trajectories, construct a probability occupancy field based on a Gaussian mixture model, quantify the collision risk between each candidate trajectory and dynamic obstacles, filter out low-risk trajectories, generate a rule mask based on the semantic text to be processed, and guide the optimization of the low-risk trajectories through the rule mask.

[0151] The control command output module is used to map the optimized trajectory into control commands to achieve end-to-end autonomous driving control.

[0152] Here, the present invention provides an implementation example:

[0153] The intersection in District B of City A was selected as a typical test scenario. This scenario has the following characteristics: high traffic volume, with more than 2,000 vehicles passing through per hour during peak hours; frequent pedestrian crossings, with an average of 15 to 20 people crossing the street per minute; traffic signal control, including dedicated lanes for left turns, straight ahead, and right turns; and complex environmental factors, including building obstructions and weather changes.

[0154] Multi-view camera image data:

[0155] Front-facing camera: 1920×1080 pixels resolution, 30Hz frame rate, 120-degree field of view;

[0156] Front left camera: 1280×720 pixel resolution, 30Hz frame rate, 90-degree field of view;

[0157] Right front camera: 1280×720 pixels resolution, 30Hz frame rate, 90-degree field of view;

[0158] Left camera: 1280×720 pixels resolution, 30Hz frame rate, 90-degree field of view;

[0159] Right-side camera: 1280×720 pixels resolution, 30Hz frame rate, 90-degree field of view;

[0160] Rearview camera: 1280×720 pixels resolution, 30Hz frame rate, 90-degree field of view;

[0161] Left rear camera: 1280×720 pixels resolution, 30Hz frame rate, 90-degree field of view;

[0162] Right rear camera: 1280×720 pixel resolution, 30Hz frame rate, 90-degree field of view.

[0163] LiDAR point cloud data: 64-line LiDAR, scanning frequency 10Hz; ranging range 0.5-200 meters, angular resolution 0.1 degrees; point cloud density, approximately 1000 points per square meter at close range (<50 meters), and approximately 100 points per square meter at long range (>100 meters).

[0164] Vehicle CAN bus data: Vehicle CAN bus data includes key parameters such as steering angle (range -540 degrees to +540 degrees, accuracy 0.1 degrees), vehicle speed (0-200km / h, accuracy 0.1km / h), acceleration (-10 to +10m / s², accuracy 0.01m / s²), brake pedal position (0%~100%, accuracy 1%), and accelerator pedal position (0%~100%, accuracy 1%).

[0165] The sensor configuration for this system is as follows:

[0166] A Hikvision DS-2CD2T47G1-L forward-facing camera is installed in the center of the vehicle's windshield. It has a resolution of 1920×1080, a frame rate of 30 frames per second, a field of view of 120 degrees, outputs RGB24 format images, and has a sampling frequency of 30Hz.

[0167] Surround view cameras are installed at the four corners and front and rear of the vehicle body. They are also Hikvision DS-2CD2T47G1-L models, with a resolution of 1280×720, a frame rate of 30 frames per second, a field of view of 90 degrees, output RGB24 format images, and a sampling frequency of 30Hz, so as to achieve all-round perception of the vehicle's surrounding environment.

[0168] The roof is equipped with a Velodyne VLP-64 lidar, which has 64-line scanning capability, a scanning frequency of 10Hz, an angular resolution of 0.1 degrees, and outputs point cloud data in N×4 format (including x, y, z coordinates and reflection intensity) with a sampling frequency of 10Hz.

[0169] The vehicle is also equipped with a high-precision GPS / IMU integrated navigation system of the NovAtel SPAN-CPT model, which is installed at the rear of the roof. It can achieve centimeter-level positioning accuracy and 0.1-degree attitude measurement, output NMEA and IMU format data, and has a sampling frequency of 100Hz.

[0170] In addition, the vehicle's original CAN bus is used to collect key parameters such as the vehicle's steering angle, speed, and acceleration. The data format is CAN2.0B, and the sampling frequency is 100Hz, ensuring real-time acquisition of vehicle dynamic information.

[0171] The data acquisition specifications are shown in Table 1:

[0172] Table 1: Data Acquisition Specifications

[0173]

[0174] The model parameter configuration is shown in Table 2:

[0175] Table 2: Model Parameter Configuration Table

[0176]

[0177] The performance metrics are compared in Table 3:

[0178] Table 3: Performance Indicator Comparison Table

[0179]

[0180] The system acquired a 1920×1080×3 RGB image (intersection scene) from the forward-facing camera, eight 1280×720×3 surround view images (covering 360 degrees), a point cloud of approximately 150,000 points (including vehicles, pedestrians, and buildings), and CAN bus data (steering angle 15.2 degrees, vehicle speed 25.3 km / h, acceleration 0.8 m / s²). Subsequently, it detected a red light signal ahead, a pedestrian preparing to cross the street on the left, and the point cloud showed a stationary vehicle 15 meters ahead that was decelerating. Finally, the system generated a multimodal fusion data packet containing information such as timestamps, sensor IDs, data quality scores, and environmental complexity assessments.

[0181] The system first acquires a front-view image, which reads, "A red light is on 20 meters ahead, there are 3 pedestrians waiting to cross the street at the crosswalk on the left, and vehicles are queuing in the right lane." The BLIP-2 model then analyzes the image, initially generating the semantic text, "Traffic light ahead, pedestrians on the left, vehicles on the right." Next, the MLP calibration network, combined with CAN bus data (such as vehicle speed 25.3 km / h, vehicle decelerating), calibrates the semantic text, outputting a more precise description: "A red light is on 20 meters ahead, 3 pedestrians are waiting at the crosswalk on the left, 2 vehicles are queuing in the right lane, current speed 25.3 km / h, decelerating." The final calibrated semantic text not only includes spatial location information and quantity statistics but also provides accurate semantic constraints for subsequent trajectory planning.

[0182] The input data for the BEV feature generation and spatiotemporal fusion example includes image feature maps from eight perspectives, point cloud feature volumes, and calibrated semantic text features. The system first uses a visual encoder to extract image features from each perspective and projects them into the BEV space to generate a 200×200×64 feature map. At the same time, the point cloud data is projected through the lidar extrinsic matrix to generate point cloud BEV features of the same resolution. The semantic text is processed by a CLIP encoder to obtain 512-dimensional features, which are then reduced to 77-dimensional through a linear layer and fused with the visual BEV features. Subsequently, a spatiotemporal Transformer processes the third BEV features of the current frame and the previous three frames, and finally outputs a 200×200×64 BEV feature that integrates geometric structure, semantic information, and spatiotemporal dynamic information, providing a unified environmental representation for subsequent trajectory prediction.

[0183] By integrating spatiotemporal BEV features with environmental information such as red lights, pedestrians, and queuing vehicles, the system encodes the BEV features into a latent variable distribution using a CVAE encoder and samples them to generate 10 candidate trajectories. Then, it uses a GMM to construct a probabilistic occupancy field to assess the collision risk of each candidate trajectory with pedestrians and vehicles. At the same time, the rule masking engine combines semantic text to generate traffic rule constraints and dynamically adjusts the trajectory parameters. Finally, it selects the optimal trajectory that stops and waits at a red light, yields to pedestrians crossing the street, and maintains a safe distance, outputting position, velocity, and curvature parameters with 50 time steps.

[0184] Using the optimized reference trajectory and the current vehicle state as input, the MPC controller aims to track the trajectory, comprehensively considers vehicle dynamics constraints, solves the optimal control sequence, and achieves a steering wheel angle of -2.3 degrees, accelerator pedal 0%, and brake pedal 15% output by minimizing the trajectory tracking error and the smoothness of the control input, ultimately achieving smooth deceleration and stopping.

[0185] This method demonstrates superior performance in perception accuracy, trajectory planning, and system performance: the target detection accuracy reaches 78.5%, an improvement of 8.6 percentage points over the baseline; the semantic segmentation average intersection-over-union ratio is 82.3%, an improvement of 9.1 percentage points; and the BEV feature quality score is 94.2%, maintaining high-precision perception even in complex scenarios. The average displacement error (ADE) of the trajectory is reduced to 0.45 meters, and the final displacement error is reduced to 0.89 meters, representing reductions of 33.8% and 33.6% respectively over the baseline. Simultaneously, the trajectory smoothness score reaches 91.5%, the safety score is 94.2%, and the regulatory compliance rate is 96.8%. The system's end-to-end inference speed is 25 milliseconds, meeting real-time requirements; memory usage is controlled within 4GB; and system latency is less than 100 milliseconds, enabling stable operation on in-vehicle computing platforms and ensuring driving safety.

[0186] Comparative experiments on the nuScenes dataset show that our method outperforms existing methods on all evaluation metrics, improving object detection mAP by 8.6%, reducing trajectory planning ADE by 33.8%, and improving inference speed by 28.6%. The statistical significance test p-value is <0.001, proving that the performance improvement is statistically significant.

[0187] In simple scenarios (such as straight urban main roads, no intersections, and low traffic volume), the system achieves a target detection accuracy of 85.2%, a trajectory tracking error of 0.32 meters, and a response time of only 18 milliseconds. It can also successfully handle vehicles suddenly changing lanes, initiating avoidance 0.8 seconds in advance. In moderately complex scenarios such as T-junctions, with traffic light control and frequent pedestrian crossings, the system achieves a target detection accuracy of 79.8%, a trajectory tracking error of 0.48 meters, and a response time of 22 milliseconds. It can correctly handle traffic light malfunctions and select safe routes based on semantic understanding. The system employs a full-passage strategy. In highly complex scenarios such as multi-lane, high-traffic, and pedestrian-dense roundabouts, the target detection accuracy is 76.3%, the trajectory tracking error is 0.52 meters, and the response time is 28 milliseconds. It can dynamically adjust the trajectory planning strategy to prioritize the passage of emergency vehicles. Furthermore, in extreme weather conditions (such as heavy rain), the system's perception accuracy remains above 70%. When some cameras fail, it can automatically adjust the fusion strategy to maintain basic functions. When GPS signals are lost, it relies on visual SLAM to maintain positioning accuracy, demonstrating good robustness and adaptability.

[0188] This method has been field-tested in autonomous driving test areas in cities such as A and C, with a cumulative test mileage exceeding 100,000 kilometers. In practical applications, the system demonstrates good stability and reliability, capable of handling various complex traffic scenarios, and provides strong support for the commercial application of autonomous driving technology.

[0189] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. An end-to-end autonomous driving method based on multimodal attention fusion, characterized in that, include: Real-time acquisition of high-definition images of the road ahead and driving intention data to generate initial semantic text; semantic calibration of the initial semantic text through an MLP network to suppress the illusion problem of visual language model; output of unprocessed semantic text containing key semantic entities and their spatial locations. The semantic calibration process is as follows: ； in This represents the calibrated text semantic feature vector; It is the ReLU activation function; and These represent the weight matrices of the first fully connected layer and the second fully connected layer, respectively. Represents the feature vector of the input image; and These represent the first-layer bias vector and the second-layer bias vector, respectively. The first BEV feature is generated by performing feature processing on environmental images acquired by multi-view cameras and point cloud data acquired by LiDAR. The CLIP model is used to perform cross-modal alignment between the semantic text to be processed and the first BEV feature. Through a learnable residual modulation module, the dimensionality-reduced text feature and the first BEV feature are weighted and fused at the feature channel level to achieve explicit alignment of semantic and geometric features. Spatiotemporal feature modeling is performed to output the second BEV feature that fuses spatiotemporal information. The step of using the CLIP model to perform cross-modal alignment between the semantic text to be processed and the first BEV feature includes: using the text encoder of the CLIP model to encode the semantic text to be processed to generate a high-dimensional text feature vector; reducing the dimensionality of the text feature to a preset dimension through a linear layer; freezing the CLIP visual encoder and retaining only the text encoder parameters; and mapping the dimensionality-reduced text feature to the BEV feature space. The second BEV feature is input into a conditional variational autoencoder to generate candidate trajectories. A probability occupancy field is constructed based on a Gaussian mixture model. The collision risk of each candidate trajectory with dynamic obstacles is quantified, and low-risk trajectories are selected. A rule mask is generated based on the semantic text to be processed, and the low-risk trajectory is optimized by guiding the optimization of the low-risk trajectory through the rule mask. The trajectory parameters of the low-risk trajectory are dynamically adjusted. The candidate trajectory generation method utilizes a conditional variational autoencoder to generate multiple sets of candidate trajectories, and uses KL divergence loss to constrain trajectory diversity and cover driving intentions. The optimized trajectory is mapped into control commands to achieve end-to-end autonomous driving control.

2. The end-to-end autonomous driving method based on multimodal attention fusion according to claim 1, characterized in that, Real-time acquisition of high-definition images of the road ahead and driving intention data to generate initial semantic text; calibration of the initial semantic text to form semantic text to be processed includes: High-definition images of the road ahead and driving intention data are collected via a forward-facing camera and a CAN bus, respectively. The GroundTruth tag information is recorded synchronously during the data collection process. The high-resolution image of the road ahead, driving intention data, and GroundTruth labels are input into the BLIP-2 model to generate initial semantic text.

3. The end-to-end autonomous driving method based on multimodal attention fusion according to claim 1, characterized in that, The text features are reduced to a preset dimension using a linear layer: ； in This represents a high-dimensional feature vector generated after the calibrated semantic text is encoded by the CLIP text encoder. This represents the preset-dimensional text feature vector output after dimensionality reduction; This represents the linear projection weight matrix, with dimension 1. ; This represents the linear projection bias vector, with a preset dimension. This represents the original feature dimension output by the CLIP text encoder.

4. The end-to-end autonomous driving method based on multimodal attention fusion according to claim 1, characterized in that, The process of weighted fusion of the dimensionality-reduced text features and the first BEV feature at the feature channel level is represented as follows: ； in Indicates the third BEV characteristic, Represents the residual weight matrix. Scaling factor The residual weight matrix is... This represents the preset-dimensional text feature vector output after dimensionality reduction. Represents the Hadamard product. Presentation layer normalization operation.

5. The end-to-end autonomous driving method based on multimodal attention fusion according to claim 1, characterized in that, The spatiotemporal feature modeling includes: transforming the historical third BEV features of the previous 3 frames to the current coordinate system through a relative transformation matrix to eliminate spatial displacement deviation between different frames; inputting the aligned historical third BEV features of the previous 3 frames and the current third BEV features into a 6-layer spatiotemporal Transformer, with each layer employing an 8-head self-attention mechanism to capture spatiotemporal dependencies layer by layer, and outputting a second BEV feature that integrates spatiotemporal information.

6. The end-to-end autonomous driving method based on multimodal attention fusion according to claim 1, characterized in that, The step of guiding the optimization of the low-risk trajectory through rule masks includes generating a rule mask based on the semantic text to be processed, converting the regulatory constraint information in the semantic text to be processed into quantifiable mask values ​​through a neural network, and dynamically adjusting the trajectory parameters of the low-risk trajectory based on the generated rule mask. ； ； in This represents the curvature after dynamic adjustment under the influence of the rule mask. For the original curvature, For safety curvature, For the original speed, The speed is dynamically adjusted under the rule mask. Represents the rule mask.

7. An end-to-end autonomous driving system based on multimodal attention fusion, characterized in that, An end-to-end autonomous driving method based on multimodal attention fusion as described in any one of claims 1-6, comprising: The data acquisition and visual semantic feature generation module is used to acquire high-definition images of the road ahead and driving intention data in real time and generate initial semantic text, and to calibrate the initial semantic text to form semantic text to be processed. The BEV feature generation and spatiotemporal fusion module is used to perform feature processing on environmental images collected by multi-view cameras and point cloud data collected by lidar to generate the first BEV feature. The CLIP model is used to perform cross-modal alignment of the semantic text to be processed and the first BEV feature, and to perform spatiotemporal feature modeling to output the second BEV feature that fuses spatiotemporal information. The multimodal trajectory prediction and safety planning module is used to input the second BEV feature into the conditional variational autoencoder to generate candidate trajectories, construct a probability occupancy field based on a Gaussian mixture model, quantify the collision risk between each candidate trajectory and dynamic obstacles, filter out low-risk trajectories, generate a rule mask based on the semantic text to be processed, and guide the optimization of the low-risk trajectories through the rule mask. The control command output module is used to map the optimized trajectory into control commands to achieve end-to-end autonomous driving control.

Images