Multi-modal semantic-action alignment method and device for end-to-end automatic driving

Abstract

The invention provides a multi-modal semantic-action alignment method and equipment for end-to-end automatic driving, and belongs to the technical field of automatic driving, and the method comprises the steps: carrying out semantic reasoning through a large language model based on various sensor data and language instruction information of a vehicle, and generating semantic information containing a driving intention; inputting the semantic information into a semantic-action alignment module, and converting the semantic information into corresponding driving action representation through a learned consistency mapping relation from a semantic space to an action space; and generating an executable control track of the vehicle according to the driving action representation. According to the method, the problem of insufficient semantic and action space alignment is solved, information distortion and precision limitation caused by post-processing depending on rules are avoided, the high-level driving intention can be generated based on complex multi-mode information (such as navigation instructions and traffic environments), it is ensured that the final execution action is highly consistent with the intention, and the driving intention is more accurate. And the decision-making rationality of the system in a complex scene is enhanced.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving method and system that combines multimodal perception, semantic reasoning, and vehicle control trajectory generation using a large language model. Background Technology

[0002] Current end-to-end autonomous driving solutions, especially those based on large vision-language-action models, face significant challenges in translating high-level semantic decisions into precise and safe vehicle control commands. First, a modal gap exists between the semantic and action spaces. Existing technologies lack differentiable mapping mechanisms to convert language tags into low-dimensional continuous action vectors, leading to rule-based post-processing of inference results, resulting in semantic information distortion and limited planning accuracy. Second, generative planners (such as diffusion models) struggle to embed semantic constraints (such as traffic rules) provided by large language models at the action generation source, often causing trajectories to deviate from intent or violate safety boundaries. Finally, existing solutions lack an interpretable planning process when outputting trajectory sequences, making it impossible to trace back the semantic logic corresponding to the driving intent, hindering safety verification and scenario review. Therefore, a new method capable of achieving accurate, reliable, and interpretable semantic-action alignment is urgently needed. Summary of the Invention

[0003] In view of the technical defects and drawbacks existing in the prior art, embodiments of the present invention provide a multimodal semantic-action alignment method and device for end-to-end autonomous driving that overcomes or at least partially solves the above problems, the specific solution of which is as follows;

[0004] As a first aspect of the present invention, a multimodal semantic-action alignment method for end-to-end autonomous driving is provided, the method comprising:

[0005] Based on various sensor data and language command information from the vehicle, semantic reasoning is performed through a large language model to generate semantic information containing driving intentions.

[0006] The semantic information is input into the semantic-action alignment module, which uses the consistent mapping relationship between the semantic space and the action space learned by the module to convert the semantic information into the corresponding driving action representation.

[0007] Based on the driving actions, an executable control trajectory for the vehicle is generated.

[0008] Furthermore, the method also includes: processing various sensor data and voice command information of the vehicle, specifically including:

[0009] The system acquires visual image data from a camera, 3D point cloud data from a lidar system, and language command information; the language command information includes navigation commands from the vehicle navigation system and / or natural language commands from the user.

[0010] The visual image data is processed using a pre-trained image feature extraction model to obtain image feature vectors; and the three-dimensional point cloud data is processed using a pre-trained point cloud feature extraction model to obtain point cloud feature vectors.

[0011] The image feature vector and point cloud feature vector are respectively input into a projection network and projected onto a semantic embedding space shared with the language instruction features.

[0012] The projected image feature vectors and point cloud feature vectors are fused to form a unified multimodal environment representation.

[0013] Furthermore, the semantic reasoning through a large-scale language model specifically includes:

[0014] A preset prompt word template is constructed, the multimodal environment representation is embedded into the template as context information, and combined with the language instruction information to form a complete prompt message;

[0015] The complete prompt information is input into a large language model to drive the model to perform semantic reasoning.

[0016] From the output of the large language model, structured semantic reasoning results are parsed out; the semantic reasoning results include at least:

[0017] High-level driving intentions;

[0018] Candidate driving strategies corresponding to the aforementioned high-level driving intentions;

[0019] Semantic embedding vectors used for subsequent action mapping.

[0020] Furthermore, the semantic information is converted into a driving action representation, which is achieved by the large language model in conjunction with the semantic-action alignment module through the following steps:

[0021] Generative latent alignment: The semantic embedding vector is input into a generative model, which maps and generates a low-dimensional latent action representation.

[0022] Action representation learning: Through a trainable mapping network, physical driving action parameters are mapped to the lexical embedding space of the large language model to obtain corresponding action tags;

[0023] Context-adaptive reasoning: The potential action representation, the action tag, and the multimodal context information of the current scene are input into the large language model; the large language model is driven to reason based on the thought chain mechanism, first generating intermediate reasoning steps, and then outputting a sequence of driving actions composed of the action tags that is adapted to the current scene as the driving action representation.

[0024] Furthermore, the generative latent alignment is achieved by training a variational autoencoder, the training process of which includes:

[0025] The semantic embedding vector output by the large language model is input into the encoder of the variational autoencoder to obtain a low-dimensional latent vector; then this low-dimensional latent vector is input into the decoder of the variational autoencoder to reconstruct the trajectory parameters, which include path point coordinates, velocity and acceleration.

[0026] The semantic embedding vector is used as the anchor sample, the latent vector obtained from the encoding of the real trajectory parameters is used as the positive sample, and the latent vectors of other samples in the same training batch are randomly selected as the negative samples.

[0027] Calculate a mixture loss function, which includes:

[0028] Reconstruction loss: Used to measure the difference between the reconstructed trajectory parameters and the true trajectory parameters;

[0029] Contrast loss: used to bring the anchor sample closer to the positive sample in the latent space, while pushing the anchor sample further away from the negative sample;

[0030] Based on the hybrid loss, the network parameters of the variational autoencoder and decoder are updated using the backpropagation algorithm.

[0031] Furthermore, the action representation learning is achieved through the following steps:

[0032] A trainable multilayer perceptron network is provided, which takes physical driving action parameters as input, maps them to the label embedding space of the large language model, and outputs the corresponding action labels.

[0033] Action tags are added to the vocabulary of the large language model to expand its vocabulary;

[0034] Supervised fine-tuning of a large language model with an expanded vocabulary is performed using real driving data sequences; wherein, the sensor and language commands in the data sequences are used as model inputs, and the action label sequence obtained by mapping real driving action parameters is used as supervision labels;

[0035] During fine-tuning, the goal is to minimize the cross-entropy loss between the model's predicted labels and the supervised labels, thereby optimizing the parameters of the entire large language model so that the model can learn to directly output the correct action label sequence.

[0036] Furthermore, the context-adaptive reasoning is achieved by performing the following steps through a large language model:

[0037] The multimodal environment representation of the current scene, language instruction information, and potential action representation obtained by generative latent alignment are all input as prompt information into a large language model;

[0038] Add thought chain guidance instructions to the prompt information to drive the large language model to perform step-by-step reasoning;

[0039] In response to the guidance, the large language model first generates at least one intermediate reasoning step, which describes the cognition of the current scene, risk assessment, and driving strategy priority in the form of natural language text.

[0040] Based on the intermediate semantic decisions already generated, the large language model continues to reason and outputs a sequence of action tags defined by action representation learning, which represents the final driving action command.

[0041] Further, based on the driving action representation, an executable control trajectory for the vehicle is generated, including the following steps:

[0042] The driving action representation output by the semantic-action alignment module is input into a trajectory decoder to decode and obtain an initial trajectory parameter sequence, which includes path point coordinates, velocity and acceleration information;

[0043] The initial trajectory parameter sequence is input into a predefined vehicle dynamics model to verify kinematic feasibility, and trajectory segments that do not meet the physical motion limits of the vehicle are smoothed and corrected.

[0044] For the processed trajectory, the smoothness of the trajectory is optimized with the goal of reducing the abrupt changes in acceleration and jerk.

[0045] The optimized trajectory parameter sequence is converted into control commands that the vehicle actuators can directly execute, generating the final executable control trajectory.

[0046] Furthermore, the end-to-end model involved in the method is optimized through a two-stage training strategy, specifically including:

[0047] Phase One: Monitoring and Fine-tuning

[0048] Prepare a large-scale real-world driving dataset, which includes sensor data, voice command information, and corresponding real vehicle control trajectories.

[0049] Using the sensor data and language command information as model inputs and the real vehicle control trajectory as supervision labels, supervised training is performed on an end-to-end model containing a large language model and a semantic-action alignment module.

[0050] By minimizing the loss function between the model-predicted control trajectory and the actual control trajectory, the model parameters are updated, and an initial semantic-action mapping relationship is established.

[0051] The second phase involves fine-tuning the reinforcement learning process.

[0052] The model trained in the first stage is used as the initial strategy and placed in a simulated driving environment;

[0053] In the simulation environment, the model outputs a control trajectory based on the current scene state and executes that trajectory;

[0054] Based on the results of the model action execution, a multi-objective reward value is calculated, which comprehensively evaluates the safety, driving efficiency and ride comfort of the action;

[0055] A grouped relative strategy optimization algorithm is adopted to update the model strategy based on the reward value in order to maximize the long-term cumulative reward.

[0056] By sequentially executing the first and second training steps, the model ultimately acquires collaboratively optimized semantic reasoning and trajectory planning capabilities.

[0057] As a second aspect of the present invention, an electronic device is provided, comprising:

[0058] One or more processors;

[0059] Memory, used to store one or more programs;

[0060] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods described above.

[0061] The present invention has the following beneficial effects:

[0062] This invention constructs an end-to-end, semantically driven decision-making system. Specifically, the method uses a large language model (LLM) for semantic reasoning and a dedicated semantic-action alignment module for mapping, directly generating driving action representations corresponding to driving intentions. This fundamentally solves the problem of insufficient alignment between semantic and action spaces, avoiding information distortion and accuracy limitations caused by rule-based post-processing. Through deep semantic understanding using LLM, the system can generate high-level driving intentions based on complex multimodal information (such as navigation instructions and traffic environment) and ensure that the final executed action is highly consistent with the intention, enhancing the system's decision-making rationality in complex scenarios and is key to achieving high-level autonomous driving. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating a multimodal semantic-action alignment method for end-to-end autonomous driving provided by an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram illustrating the process of processing various sensor data and voice command information of a vehicle according to an embodiment of the present invention.

[0065] Figure 3 A flowchart illustrating semantic reasoning using a large language model, provided as an embodiment of the present invention.

[0066] Figure 4 This is a schematic diagram of the overall architecture of a multimodal semantic-action alignment method for end-to-end autonomous driving provided by an embodiment of the present invention;

[0067] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0069] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0070] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0071] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0072] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0073] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0074] To address at least one of the technical problems existing in the aforementioned related technologies, this invention provides a multimodal semantic-action alignment method for end-to-end autonomous driving. Figure 1 This is a flowchart illustrating a multimodal semantic-action alignment method for end-to-end autonomous driving provided by an embodiment of the present invention. The method includes:

[0075] S1. Based on various sensor data and language command information of the vehicle, semantic reasoning is performed through a large language model to generate semantic information containing driving intentions;

[0076] S2. Input the semantic information into the semantic-action alignment module, and convert the semantic information into the corresponding driving action representation through the consistent mapping relationship from the semantic space to the action space learned by the module;

[0077] S3. Based on the driving action representation, generate an executable control trajectory for the vehicle.

[0078] This invention constructs an end-to-end, semantically driven decision-making system. Specifically, the method uses a large language model (LLM) for semantic reasoning and a dedicated semantic-action alignment module for mapping, directly generating driving action representations corresponding to driving intentions. This fundamentally solves the problem of insufficient alignment between semantic and action spaces, avoiding information distortion and accuracy limitations caused by rule-based post-processing. Through deep semantic understanding using LLM, the system can generate high-level driving intentions based on complex multimodal information (such as navigation instructions and traffic environment) and ensure that the final executed action is highly consistent with the intention, enhancing the system's decision-making rationality in complex scenarios and is key to achieving high-level autonomous driving.

[0079] See Figure 2 As shown, in some embodiments, the method further includes: processing various sensor data and voice command information of the vehicle, specifically including:

[0080] S21. Data Acquisition: Acquire visual image data from the camera, 3D point cloud data from the lidar, and language command information; the language command information includes navigation commands from the vehicle navigation system and / or natural language commands from the user;

[0081] S22. Visual feature extraction: The visual image data is processed using a pre-trained image feature extraction model to obtain an image feature vector; and the three-dimensional point cloud data is processed using a pre-trained point cloud feature extraction model to obtain a point cloud feature vector.

[0082] S23. Feature Projection and Alignment: The image feature vector and point cloud feature vector are respectively input into a projection network and projected onto a semantic embedding space shared with the language instruction features.

[0083] S24. Forming an environmental representation: The projected image feature vector and point cloud feature vector are fused to form a unified multimodal environmental representation.

[0084] In the above embodiments, by fusing visual images and 3D point cloud data and extracting their features separately, a unified multimodal environment representation is formed. This fused perception provides richer and more accurate environmental information than a single modality, helping LLM to understand the scene context more comprehensively. By projecting visual features onto a semantic embedding space shared with language instructions, visual and linguistic information can interact and fuse at the same semantic level. This enables subsequent high-quality, multimodal context-based semantic reasoning in LLM, directly contributing to the improvement of the final planning accuracy.

[0085] See Figure 3As shown, in some embodiments, the semantic reasoning using a large language model includes the following steps:

[0086] S31. Constructing prompt information: Constructing a preset prompt word template, embedding the multimodal environment representation as context information into the template, and combining it with the language instruction information to form complete prompt information;

[0087] S32. Perform model reasoning: Input the complete prompt information into a large language model to drive the model to perform semantic reasoning;

[0088] S33. Parse the structured output: Parse the structured semantic reasoning results from the output of the large-scale language model; the semantic reasoning results include at least:

[0089] High-level driving intentions;

[0090] Candidate driving strategies corresponding to the aforementioned high-level driving intentions;

[0091] Semantic embedding vectors used for subsequent action mapping.

[0092] The above embodiments specifically define how to use LLM for semantic reasoning, focusing on improving the interpretability and logic of the decision-making process. Specifically, the LLM output includes not only high-level driving intentions but also candidate strategies and semantic embedding vectors for action mapping. This structured output makes the system's decision logic clear and traceable, effectively solving the problem of "lack of interpretability in the action space," facilitating safety verification and scenario review. By constructing prompt word templates, environmental representations and language instructions are combined and input into the LLM to drive the model's reasoning. This method leverages the powerful common-sense and logical reasoning capabilities of LLM to handle complex scenarios requiring deep understanding, such as "slowing down to avoid pedestrians" or "yielding at intersections according to rules," thereby enhancing the system's generalization ability.

[0093] In some embodiments, the conversion of semantic information into driving action representations is achieved by the large language model in conjunction with the semantic-action alignment module through the following steps:

[0094] S41, Generative Latent Alignment: The semantic embedding vector is input into a generative model, which maps and generates a low-dimensional latent action representation.

[0095] S42. Action representation learning: Through a trainable mapping network, physical driving action parameters are mapped to the lexical embedding space of the large language model to obtain corresponding action tags.

[0096] S43. Context-adaptive reasoning: The potential action representation, the action tag, and the multimodal context information of the current scene are input into the large language model; the large language model is driven to reason based on the thought chain mechanism, first generating intermediate reasoning steps, and then outputting a driving action sequence composed of the action tag that is adapted to the current scene as the driving action representation.

[0097] In the above embodiments, through the close collaboration of three sub-steps—generative latent alignment, action representation learning, and context-adaptive reasoning—a complete and trainable conversion channel from high-level semantic embedding vectors to low-level driving action sequences is established, enabling the semantic reasoning results of LLM to directly and smoothly influence the final action generation and avoiding information distortion.

[0098] The solution emphasizes context-adaptive reasoning through the Chain of Thought (CoT) mechanism of a large language model. The model first generates intermediate reasoning steps (such as risk perception and strategy prioritization) before outputting the final action sequence. This step-by-step reasoning process significantly enhances the logicality of decision-making and its adaptability to complex and dynamic scenarios, enabling the system not only to "take actions" but also to "explain why these actions were taken," thereby improving the safety of planning.

[0099] In some embodiments, the generative latent alignment is achieved by training a variational autoencoder, the training process of which includes the following steps:

[0100] S51. Forward computation of the model: The semantic embedding vector output by the large language model is input into the encoder of the variational autoencoder to obtain a low-dimensional latent vector; then this low-dimensional latent vector is input into the decoder of the variational autoencoder to reconstruct the trajectory parameters, which include path point coordinates, velocity and acceleration.

[0101] S52. Contrastive learning sample construction: using the semantic embedding vector as the anchor sample, using the latent vector obtained from the encoding of the real trajectory parameters as the positive sample, and randomly selecting the latent vectors of other samples in the same training batch as the negative sample.

[0102] S53. Mixed Loss Calculation: Calculate a mixed loss function, which includes:

[0103] Reconstruction loss: Used to measure the difference between the reconstructed trajectory parameters and the true trajectory parameters in S51;

[0104] Contrast loss: used to bring the anchor sample closer to the positive sample in the latent space, while pushing the anchor sample further away from the negative sample;

[0105] S54. Model parameter update: Based on the hybrid loss, update the network parameters of the variational autoencoder and decoder through the backpropagation algorithm.

[0106] In the above embodiments, by introducing a contrastive learning loss, matching semantic-trajectory pairs are explicitly brought closer together in the latent space, while mismatched pairs are pushed away. This training mechanism strongly constrains the generative model, ensuring that its output trajectory parameters (such as waypoints and speeds) accurately reflect the driving intent represented by the input semantic embedding (such as "stop at a red light"), effectively preventing the generation of invalid or dangerous trajectories that deviate from the semantics.

[0107] Hybrid loss functions (combining reconstruction loss and contrastive loss) provide multi-dimensional optimization objectives for model training. Reconstruction loss ensures that the generated trajectories are numerically close to the true trajectories, while contrastive loss guarantees the reasonableness of the underlying spatial structure. Compared to a single loss function, this training approach converges to a better solution faster and improves the model's robustness.

[0108] In some embodiments, the action representation learning is achieved through the following steps:

[0109] S61. Establish action label mapping relationship: Provide a trainable multilayer perceptron network, take physical driving action parameters as input, map to the label embedding space of the large language model, and output the corresponding action label;

[0110] S62. Expand the model vocabulary: Add the action tags defined in S61 to the vocabulary of the large language model to expand its vocabulary.

[0111] S63, Supervised Fine-tuning Model: Supervised fine-tuning of a large language model with an expanded vocabulary is performed using real driving data sequences; wherein, the sensors and language commands in the data sequences are used as model inputs, and the action label sequence obtained after mapping the real driving action parameters through S61 is used as supervision labels.

[0112] S64. Model Optimization: During fine-tuning, the parameters of the entire large language model are optimized with the goal of minimizing the cross-entropy loss between the model's predicted labels and the supervision labels, so that the model can learn to directly output the correct action label sequence.

[0113] The above embodiments map physical driving action parameters to specific tokens in the LLM vocabulary, enabling the LLM to directly understand and output driving action sequences just like understanding and generating natural language. This design allows the powerful LLM reasoning capabilities to be directly applied to the generation of control commands, achieving the unification of semantic reasoning and action generation within the same model and improving computational efficiency.

[0114] Supervised fine-tuning of the expanded vocabulary LLM using real driving data and optimization with cross-entropy loss ensures that the action-labeled sequences output by the model are highly aligned with real driving behavior. This method not only learns the statistical regularities of actions but also implicitly incorporates vehicle dynamics constraints, making the generated action sequences not only semantically reasonable but also physically feasible, thereby improving the accuracy and safety of planning.

[0115] In some embodiments, the context-adaptive reasoning is implemented by performing the following steps using a large language model:

[0116] S71, Inference Initialization: The multimodal environment representation of the current scene, language instruction information, and potential action representation obtained by generative latent alignment are input as prompt information into the large language model;

[0117] S72, Thought Chain Guidance: Add thought chain guidance instructions to the prompt information to drive the large language model to perform step-by-step reasoning;

[0118] S73. Generate intermediate semantic decisions: In response to the guidance, the large language model first generates at least one intermediate reasoning step, which describes the cognition of the current scene, risk judgment, and driving strategy priority in the form of natural language text.

[0119] S74. Generate the final action sequence: Based on the generated intermediate semantic decisions, the large language model continues to reason and outputs a sequence of action tags defined by action representation learning, which represents the final driving action command.

[0120] The above embodiments explicitly require that large language models (LLMs) first generate intermediate reasoning steps described in natural language text (such as scene cognition, risk assessment, etc.) before outputting the final action sequence. This explicit reasoning based on thought chain (CoT) transforms the model's "black box" decision-making process into a readable logical chain, greatly facilitating developers in security verification, debugging, and scenario review, and directly solving the problem of "lack of interpretability of action space" in the background technology.

[0121] Through a step-by-step reasoning mechanism, the model can more carefully handle complex or rare scenarios such as intersections and pedestrians suddenly entering the road. By performing semantic analysis and judgment first, and then generating actions, it effectively avoids hasty or erroneous decisions that may result from direct end-to-end mapping, thereby reducing the risk of accidents and improving the overall safety of the system.

[0122] In some embodiments, generating an executable control trajectory for the vehicle based on the driving action representation includes the following steps:

[0123] S81. Trajectory Decoding: The driving action representation output by the semantic-action alignment module is input into a trajectory decoder to decode and obtain an initial trajectory parameter sequence, which includes path point coordinates, velocity and acceleration information.

[0124] S82, Dynamics Constraint Verification and Smoothing: Input the initial trajectory parameter sequence into a predefined vehicle dynamics model to verify kinematic feasibility, and perform smoothing interpolation and correction on trajectory segments that do not meet the vehicle's physical motion limits.

[0125] S83, Comfort Optimization: For the trajectory processed by S82, the smoothness of the trajectory is optimized with the goal of reducing the abrupt changes in acceleration and jerk.

[0126] S84. Generate control commands: Convert the optimized trajectory parameter sequence into control commands that the vehicle actuators can directly execute, and generate the final executable control trajectory.

[0127] In the above embodiments, by introducing a predefined vehicle dynamics model for kinematic feasibility verification and smoothing correction, it is ensured that the decoded trajectory parameters (such as path points and speeds) strictly conform to the vehicle's physical motion limits (such as maximum turning radius and maximum acceleration), thus preventing the generation of dangerous trajectories that cannot be executed or may cause the vehicle to lose control from the bottom layer.

[0128] In particular, comfort optimization is proposed with the goal of reducing abrupt changes in acceleration and jerk. Through trajectory smoothing processing, the bumps caused by rapid acceleration, deceleration and sharp turns can be effectively reduced, significantly improving passenger comfort.

[0129] In some embodiments, the end-to-end model involved in the method is optimized using a two-stage training strategy, which is executed sequentially according to the following steps:

[0130] Phase One: Monitoring and Fine-tuning

[0131] S91: Prepare a large-scale real-world driving dataset, which includes sensor data, voice command information, and corresponding real vehicle control trajectories.

[0132] S92: Using the sensor data and language command information as model inputs and the real vehicle control trajectory as supervision labels, supervised training is performed on the end-to-end model containing a large language model and a semantic-action alignment module.

[0133] S93: By minimizing the loss function between the model-predicted control trajectory and the actual control trajectory, the model parameters are updated, and the initial semantic-action mapping relationship is established;

[0134] The second phase involves fine-tuning the reinforcement learning process.

[0135] S94: The model trained in the first stage is used as the initial strategy and placed in the simulated driving environment;

[0136] S95: In the simulation environment, the model outputs a control trajectory based on the current scene state and executes the trajectory;

[0137] S96: Based on the results of the model action execution, calculate a multi-objective reward value, which comprehensively evaluates the safety, driving efficiency and ride comfort of the action;

[0138] S97: The grouped relative strategy optimization algorithm is adopted to update the model strategy according to the reward value in order to maximize the long-term cumulative reward;

[0139] By sequentially executing the first and second training steps, the model ultimately acquires collaboratively optimized semantic reasoning and trajectory planning capabilities.

[0140] In the above embodiments, the first stage of large-scale supervised fine-tuning (SFT) uses real driving data to enable the model to quickly learn and establish an initial mapping relationship from multimodal perception to control trajectory. This provides the model with a high-performance starting point and ensures the basic rationality and safety of its behavior.

[0141] The second stage of reinforcement learning (RL) fine-tuning, particularly employing the Grouped Relative Policy Optimization (GRPO) algorithm, allows the model to optimize its long-term decision-making capabilities through trial and error in a safe simulated environment. Optimization based on a multi-objective reward function (evaluating safety, efficiency, and comfort) drives the model to learn better trade-offs in complex tasks, significantly enhancing its generalization and robustness in handling various unknown scenarios. This two-stage strategy combines the advantages of imitation learning (rapid learning) and reinforcement learning (refinement), and is key to achieving high-performance autonomous driving systems.

[0142] Example: A complete implementation of a multimodal semantic-action alignment method for end-to-end autonomous driving

[0143] Figure 4 This is a schematic diagram of the overall architecture of a multimodal semantic-action alignment method for end-to-end autonomous driving provided by an embodiment of the present invention, as shown below. Figure 4 As shown, the overall architecture of the method includes a multimodal perception module, a semantic reasoning module, a semantic-action alignment module, and a trajectory decoding module. The method is executed in the following order:

[0144] (1) Multimodal perception and feature extraction

[0145] Data Acquisition: The vehicle's sensor system begins to operate. Cameras acquire visual image data of the current environment, and LiDAR acquires 3D point cloud data of surrounding objects. Simultaneously, the system receives voice commands, which may originate from navigation commands generated by the vehicle's navigation system (such as "Turn right onto the auxiliary road 300 meters ahead") and / or natural language commands issued by the user (such as "Please find the nearest charging station").

[0146] Visual feature extraction: The visual image data is processed using a pre-trained convolutional neural network (such as ResNet) model to extract high-dimensional image feature vectors. Simultaneously, the 3D point cloud data is processed using a pre-trained point cloud processing model (such as PointNet++) to extract point cloud feature vectors with geometric information.

[0147] Feature projection and alignment: The extracted image feature vectors and point cloud feature vectors are input into a shared projection network (e.g., a multilayer perceptron) to project them into a semantic embedding space shared with the language instruction features. This aims to bridge the gap between the visual and linguistic modalities.

[0148] Forming an environmental representation: The projected image feature vector and point cloud feature vector are fused (e.g., by splicing or weighted summation) to form a unified multimodal environmental representation rich in multimodal information, which serves as contextual information for subsequent semantic reasoning.

[0149] (2) Semantic reasoning based on large-scale language models

[0150] Constructing prompt information: The system presets a prompt word template, for example: "You are an autonomous driving system. Based on the following scenario information: [multimodal environment representation]. Please generate driving decisions according to the instructions: [language instruction information]." The multimodal environment representation obtained in step 1 is embedded as context information into the template to form a complete prompt message.

[0151] Execution Model Inference: The complete prompt information is input into a large language model (such as a model with a GPT-4 architecture). This drives the LLM to perform semantic reasoning based on its internal knowledge and logical reasoning capabilities.

[0152] Parse the structured output: Extract the structured semantic reasoning results from the LLM output. These results should include at least:

[0153] High-level driving intentions: such as "slow down at the intersection and prepare to yield to cross traffic".

[0154] Candidate driving strategies: such as "Strategy 1: Decelerate to a stop; Strategy 2: Glide slowly through".

[0155] Semantic embedding vector: A high-dimensional vector extracted from the LLM hidden layer that encodes the semantic essence of this inference and is used for subsequent action mapping.

[0156] (3) Semantic-action alignment generation

[0157] Generative latent alignment: The semantic embedding vector obtained in step (2) is input into a pre-trained variational autoencoder. The encoder of the VAE compresses the semantic vector into a low-dimensional latent vector; subsequently, the decoder of the VAE reconstructs the initial trajectory parameters, including the path point coordinate sequence, velocity, and acceleration values, based on this latent vector. The VAE is trained using a hybrid loss function that includes reconstruction loss and contrast loss to ensure that the latent vector it generates accurately reflects the semantic intent.

[0158] Action Representation Learning: A trainable multilayer perceptron (LLM) network is provided to map physical driving action parameters (such as steering angle, throttle opening, and brake pressure) to the vocabulary embedding space of a large language model, forming specific action tokens. For example, "turn left 30 degrees" is mapped to a unique token. These action tokens are added to the LLM's original vocabulary to form an expanded vocabulary. Subsequently, the LLM is fine-tuned under supervision using a large amount of real driving data, enabling it to learn to output the correct sequence of action tokens in a given scenario.

[0159] Context-adaptive reasoning: The multimodal environment representation from step (2), language instruction information, and latent action representation obtained from generative latent alignment are input as prompts into the finely tuned large language model. Thought chain guidance instructions (such as "Please reason step by step and output the action") are added to the prompts. The LLM will first generate intermediate reasoning steps, for example: "1. Recognition: There is a pedestrian crossing the road ahead. 2. Judgment: The pedestrian has the highest right-of-way and must yield. 3. Decision: Adopt a slowdown and stop strategy." Based on this intermediate decision, the LLM finally outputs a sequence of action tags, such as [Slow down][Maintain steering wheel angle][Stop], which is the final driving action representation.

[0160] (4) Trajectory generation and optimized execution

[0161] Trajectory decoding: The driving action representation (i.e., action marker sequence) output in step (3) is input into a trajectory decoder (which can be regarded as the inverse process of VAE decoder or an independent neural network) to decode and obtain the initial trajectory parameter sequence, which includes detailed path point coordinates, speed and acceleration information.

[0162] Dynamics constraint verification and smoothing: The initial trajectory parameter sequence is input into a predefined vehicle dynamics model to verify kinematic feasibility. For trajectory segments that do not meet the vehicle's physical motion limits (such as excessive curvature or excessive acceleration), smoothing interpolation and correction are performed.

[0163] Comfort optimization: The trajectory is smoothed to improve ride comfort by reducing abrupt changes in acceleration and jerk.

[0164] Generate control commands: Convert the optimized trajectory parameter sequence into control commands that can be directly executed by vehicle actuators (such as drive-by-wire, steering-by-wire, and brake-by-wire systems), generate the final executable control trajectory, and realize the vehicle's autonomous driving behavior.

[0165] (5) Model training strategy

[0166] The first stage, supervised fine-tuning, involves preparing a large-scale real-world driving dataset. Sensor data and language command information are used as model inputs, and real vehicle control trajectories are used as supervision labels. The end-to-end model, which includes LLM and semantic-action alignment modules, is trained in a supervised manner to minimize the difference between the predicted trajectory and the real trajectory and establish an initial semantic-action mapping relationship.

[0167] The second stage involves reinforcement learning fine-tuning: the model trained in the first stage is used as the initial policy and placed in a high-fidelity simulated driving environment. The model outputs and executes control trajectories based on the scenario state, and then the system calculates a reward value that comprehensively evaluates safety, efficiency, and comfort. A grouped relative policy optimization algorithm is used to update the model policy based on the reward value to maximize long-term cumulative rewards, enabling the model to ultimately acquire collaboratively optimized semantic reasoning and trajectory planning capabilities.

[0168] Through the above steps, this embodiment fully realizes the end-to-end autonomous driving decision-making process from multimodal perception to final control command generation, fully demonstrating the technical advantages of the method described in this invention.

[0169] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement an asynchronous tracking method for a vehicle remote control link as described in any of the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0170] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0171] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0172] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0173] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps in any of the asynchronous tracking methods for a vehicle remote control link described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0174] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described asynchronous tracking method for a vehicle remote control link.

[0175] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0176] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0177] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0178] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0179] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0180] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0182] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0184] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multi-modal semantic-action alignment method for end-to-end autonomous driving, characterized in that, The method comprises: Based on the vehicle's various sensor data and language instruction information, semantic reasoning is performed through a large language model to generate semantic information containing driving intent; The semantic information is input into a semantic-action alignment module, and through the consistency mapping relationship between the semantic space and the action space learned by the module, the semantic information is converted into corresponding driving action representation; According to the driving action representation, an executable control trajectory of the vehicle is generated.

2. The method of claim 1, wherein, The method further comprises processing the vehicle's various sensor data and language instruction information, specifically including: Obtaining visual image data from a camera, three-dimensional point cloud data from a lidar, and language instruction information; the language instruction information includes navigation instructions from the vehicle's navigation system and / or natural language instructions from the user; Using a pre-trained image feature extraction model to process the visual image data to obtain an image feature vector, and using a pre-trained point cloud feature extraction model to process the three-dimensional point cloud data to obtain a point cloud feature vector; The image feature vector and the point cloud feature vector are input into a projection network respectively, and are projected into a semantic embedding space shared with language instruction features; The projected image feature vector and point cloud feature vector are fused to form a unified multi-modal environment representation.

3. The method of claim 2, wherein, The semantic reasoning through a large language model specifically includes: A preset prompt template is constructed, the multi-modal environment representation is embedded into the template as context information, and is combined with the language instruction information to form complete prompt information; The complete prompt information is input into a large language model to drive the model to perform semantic reasoning; From the output of the large language model, a structured semantic reasoning result is parsed; the semantic reasoning result at least includes: High-level driving intent; Candidate driving strategies corresponding to the high-level driving intent; Semantic embedding vectors for subsequent action mapping.

4. The method of claim 3, wherein, The semantic information is converted into driving action representation, which is realized by the large language model in cooperation with the semantic-action alignment module through the following steps: Generative latent alignment: input the semantic embedding vector into a generative model, and map a low-dimensional latent action representation by the generative model; Action representation learning: through a trainable mapping network, map physical driving action parameters to the vocabulary embedding space of the large language model to obtain corresponding action labels; Context-adaptive reasoning: input the latent action representation, the action labels, and the multi-modal context information of the current scene into the large language model; drive the large language model to perform reasoning based on the think chain mechanism, first generate intermediate reasoning steps, and then output a driving action sequence composed of the action labels and adapted to the current scene as the driving action representation.

5. The method of claim 4, wherein, The generative latent alignment is realized by training a variational autoencoder, and the training process includes: The semantic embedding vector output by the large language model is input into the encoder of the variational autoencoder to obtain a low-dimensional latent vector; the low-dimensional latent vector is input into the decoder of the variational autoencoder to reconstruct the trajectory parameters, including path point coordinates, speed and acceleration; The semantic embedding vector is used as an anchor point sample, the latent vector obtained by encoding the real trajectory parameters is used as a positive sample, and the latent vectors of other samples in the same training batch are randomly selected as negative samples; A hybrid loss function is calculated, including: reconstruction loss: used to measure the difference between the reconstructed trajectory parameters and the real trajectory parameters; contrastive loss: used to reduce the distance between the anchor point sample and the positive sample in the latent space, while increasing the distance between the anchor point sample and the negative sample; Based on the hybrid loss, the network parameters of the variational autoencoder encoder and decoder are updated through the back propagation algorithm.

6. The method of claim 4, wherein, The action representation learning is achieved by the following steps: A trainable multi-layer perceptron network is provided, which maps the physical driving action parameters to the token embedding space of the large language model as input, and outputs the corresponding action tokens; The action tokens are added to the vocabulary of the large language model to expand its vocabulary; The large language model with an expanded vocabulary is supervised and fine-tuned using real driving data sequences; wherein the sensors and language instructions in the data sequence are used as model input, and the action token sequence obtained by mapping the real driving action parameters is used as a supervision label; During the fine-tuning process, the parameters of the entire large language model are optimized to minimize the cross-entropy loss between the model predicted tokens and the supervision labels, so that the model learns to directly output the correct action token sequence.

7. The method of claim 4, wherein, The context-adaptive reasoning is achieved by the large language model by performing the following steps: The multi-modal environment representation, language instruction information and latent action representation obtained by generative latent alignment of the current scene are collectively input into the large language model as prompt information; The thinking chain guide instruction is added to the prompt information to drive the large language model to perform step-by-step reasoning; The large language model responds to the guide and first generates at least one intermediate reasoning step, which describes the cognition, risk judgment and driving strategy priority of the current scene in natural language text form; Based on the generated intermediate semantic decision, the large language model continues to reason and output a sequence of action tokens defined by the action representation learning, which represents the final driving action instruction.

8. The method of claim 1, wherein, According to the driving action representation, an executable control trajectory of the vehicle is generated, including the following steps: The driving action representation output by the semantic-action alignment module is input into a trajectory decoder to decode an initial trajectory parameter sequence, which includes path point coordinates, speed and acceleration information; The initial trajectory parameter sequence is input into a pre-defined vehicle dynamics model for kinematic feasibility checking, and the trajectory segments that do not meet the physical motion limits of the vehicle are smoothed, interpolated and corrected. The processed trajectory is optimized in terms of reducing the sudden change of acceleration and jerk; The optimized trajectory parameter sequence is converted into control instructions that can be directly executed by the vehicle actuators to generate the final executable control trajectory.

9. The method of claim 1, wherein, The end-to-end model involved in the method is optimized through a two-stage training strategy, specifically including: First stage, supervised fine-tuning: Prepare a large-scale real driving dataset, which contains sensor data, language instruction information and corresponding real vehicle control trajectory; Use the sensor data and language instruction information as model input, and the real vehicle control trajectory as supervised label to supervise the training of the end-to-end model containing large language model and semantic-action alignment module; By minimizing the loss function between the predicted control trajectory and the real control trajectory, the model parameters are updated to establish an initial semantic-action mapping relationship; Second stage, reinforcement learning fine-tuning: Use the model trained in the first stage as the initial strategy and place it in a simulated driving environment; In the simulation environment, the model outputs a control trajectory according to the current scene state and executes the trajectory; Based on the results of the model action execution, a multi-objective reward value is calculated, which evaluates the safety, driving efficiency and ride comfort of the action; Use the grouping relative strategy optimization algorithm to update the model strategy according to the reward value to maximize the long-term cumulative reward; By sequentially executing the training steps of the first and second stages, the model finally obtains the collaborative optimization of semantic reasoning and trajectory planning capabilities.

10. An electronic device, comprising: It includes: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-9.

Images