End-to-end visual action model training method, trajectory planning method and application thereof

Abstract

The invention discloses a training method of an end-to-end visual motion model, a trajectory planning method and an application thereof to meet the requirements of high-level automatic driving application, and the training method comprises the steps: inputting a sample image at a T moment into a first model, and obtaining a perception task query and an image Token corresponding to an image perception result at the T moment; based on the perception task Query at the T moment and a second model, obtaining a predicted trajectory of the target object and a planned trajectory of the target vehicle at the T moment; based on the predicted trajectory of the target object at the T moment, the planned trajectory of the target vehicle and a third model, obtaining a predicted trajectory of the target vehicle at the T + 1 moment; based on the predicted trajectory of the target vehicle at the T + 1 moment, the planned trajectory of the target vehicle at the T moment, the image Token and a fourth model, obtaining a predicted image at the T + 1 moment; and executing back propagation of the model gradient based on the predicted image at the T + 1 moment so as to update parameters of the model and construct an end-to-end visual action model.

Description

Training methods, trajectory planning methods and applications of end-to-end visual action models Technical Field

[0001] This application belongs to the field of autonomous driving technology, specifically relating to an end-to-end visual motion model training method and apparatus, a trajectory planning method and apparatus, and an unmanned vehicle and storage medium applying the above methods. Background Technology

[0002] The classic autonomous driving system (AD) software stack is a modular system, mainly comprising modules such as perception, localization, prediction, planning and control, and mapping. These modules are relatively independent, and communication between them is defined by a set of standard interfaces. The development of one module generally does not affect other modules, facilitating collaboration, independent development, and testing within large teams, thus improving development efficiency.

[0003] However, because the modules are connected in series through interfaces, the errors of the upstream modules (such as perception) will be transmitted to the downstream and may be amplified step by step, forming "error accumulation". Furthermore, the rule-based modules are difficult to cover all complex and especially extreme driving scenarios (corner cases), making the overall system optimization more difficult and unable to meet the requirements of higher levels of autonomous driving.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a training method for an end-to-end visual action model, which is used to address the problem of how to meet the requirements of higher levels of autonomous driving.

[0006] To achieve the above objectives, this application provides a method for training an end-to-end visual action model, the method comprising:

[0007] Input the sample image at time T into the first model to obtain the perception task query and image token corresponding to the image perception result at time T;

[0008] Based on the perception task query and the second model at time T, the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T are obtained.

[0009] Based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model, the predicted trajectory of the target vehicle at time T+1 is obtained.

[0010] Based on the predicted trajectory of the target vehicle at time T+1, the planned trajectory of the target vehicle at time T, the image token, and the fourth model, the predicted image at time T+1 is obtained.

[0011] Backpropagation of model gradients is performed based on the predicted image at time T+1 to update the parameters of the first model, the second model, the third model, and the fourth model.

[0012] An end-to-end visual action model is constructed based on the first, second, third, and fourth models that have been trained.

[0013] In one embodiment, the first model includes a feature extraction network and an image sensing head;

[0014] Input the sample image at time T into the first model to obtain the perception task query and image token corresponding to the image perception result at time T, specifically including:

[0015] Based on the feature extraction network, image features of the sample image at time T are extracted and converted into bird's-eye view BEV features;

[0016] By fusing the BEV features from time T and historical time points, a fused BEV feature is obtained.

[0017] Based on the fused BEV features and image sensing head, the perception task query and image token corresponding to the image perception result at time T are obtained.

[0018] In one embodiment, the image sensing head includes at least one of a target detection head, a lane line recognition head, and an occupancy grid prediction head; and / or,

[0019] The sample images at time T include images of the target vehicle from at least two perspectives.

[0020] In one embodiment, the second model includes a dynamic target encoder, a local map encoder, a static target encoder, a global navigation perceptron, and a first diffusion model;

[0021] Based on the perception task query and the second model at time T, the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T are obtained, specifically including:

[0022] Based on the perception task Query at time T and the dynamic target encoder, local map encoder and static target encoder, at least one of the dynamic target token, local map token and static target token is obtained;

[0023] Based on the dynamic target token, local map token and static target token, information fusion is performed to obtain the scene fusion token;

[0024] Based on the global navigation instructions and the global navigation sensor, a global navigation Token is obtained;

[0025] Based on the scenario fusion token, global navigation token, and first diffusion model, the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T are obtained.

[0026] In one embodiment, the second model further includes a Transformer network; fusing the dynamic target token, local map token, and static target token to obtain a scene fusion token specifically includes: calculating the attention weight of any token among the dynamic target token, local map token, and static target token relative to tokens of the same type based on the Transformer network, thereby obtaining a first fused dynamic target token, a first fused local map token, and a first fused static target token; calculating the attention weight of a portion of at least one of the first fused dynamic target token, the first fused local map token, and the first fused static target token relative to tokens of different types based on the Transformer network, thereby obtaining at least one of a second fused dynamic target token, a second fused local map token, and a second fused static target token.

[0027] In one embodiment, the method further includes: obtaining the predicted trajectory of the target object at time T+1 based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model; and obtaining at least one of the following based on the predicted trajectory of the target vehicle at time T+1, the predicted trajectory of the target object, and the third model: a score of the probability of the target vehicle experiencing an abnormal traffic event at time T+1, a score of the target object complying with preset traffic rules, and a score of the target vehicle completing the global navigation route.

[0028] In one embodiment, when an abnormal traffic event occurs to the target vehicle at time T+1, the method further includes: stopping the training of the end-to-end visual motion model for the current round.

[0029] In one embodiment, the fourth model includes an encoder and a second diffusion model;

[0030] Based on the predicted trajectory of the target vehicle at time T+1, the planned trajectory of the target vehicle at time T, the image token, and the fourth model, the predicted image at time T+1 is obtained, specifically including:

[0031] Based on the planned trajectory of the target vehicle at time T and the encoder, the state code of the target vehicle at time T is obtained;

[0032] The state code of the target vehicle at time T is used as the position code and fused with the image token to obtain the image fusion token;

[0033] Using the predicted trajectory of the target vehicle at time T+1 as a condition, the predicted image at time T+1 is obtained based on the image fusion token and the second diffusion model.

[0034] In one embodiment, before performing end-to-end model training, the method further includes: performing training on the first model; and / or performing joint training on the first model and the second model; and / or performing joint training on the first model and the fourth model.

[0035] In one embodiment, the end-to-end visual action model further includes a reward model and a cost model; after performing end-to-end model training, the method further includes: applying safety constraints to the planned trajectory of the target vehicle output by the second model based on the cost model, and training the second model with the goal of maximizing the reward score of the planned trajectory of the target vehicle output by the reward model.

[0036] This application also provides a trajectory planning method, the method comprising acquiring an environmental image of a target vehicle and processing it with a target model to obtain a planned trajectory of the target vehicle, wherein the target model includes a second model in an end-to-end visual action model trained as described above.

[0037] In one embodiment, the target model includes a first model in the end-to-end visual motion model trained by the method described above; the method includes: calling the first model to process the environmental image of the target vehicle to obtain obstacle information in the environment of the target vehicle; and performing collision detection on the planned trajectory based on the obstacle information.

[0038] This application also provides a training device for an end-to-end visual action model, comprising:

[0039] The first calling module is used to input the sample image at time T into the first model and obtain the perception task query and image token corresponding to the image perception result at time T.

[0040] The second calling module is used to obtain the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the perception task Query and the second model at time T.

[0041] The third calling module is used to obtain the predicted trajectory of the target vehicle at time T+1 based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model.

[0042] The fourth calling module is used to obtain the predicted image at time T+1 based on the predicted trajectory of the target vehicle at time T, the planned trajectory of the target vehicle at time T, the image token, and the fourth model.

[0043] The parameter update module is used to perform backpropagation of the model gradient based on the predicted image at time T+1 to update the parameters of the first model, the second model, the third model and the fourth model.

[0044] The model building module is used to build an end-to-end visual action model based on the first, second, third, and fourth models that have been trained.

[0045] This application also provides a trajectory planning device, including an image acquisition module for acquiring environmental images of a target vehicle, and a planning module for processing a target model to obtain the planned trajectory of the target vehicle, wherein the target model includes a second model in the end-to-end visual action model trained as described above.

[0046] This application also provides an unmanned vehicle, including:

[0047] At least one processor; and

[0048] The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the end-to-end visual motion model training method or trajectory planning method as described above.

[0049] This application also provides a machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the end-to-end visual motion model training method or trajectory planning method as described above.

[0050] Compared with existing technologies, the end-to-end visual action model training method of this application, based on the architecture of the first model, the second model, the third model and the fourth model, inputs the sample image at time T, and the above models cooperate to predict the image at time T+1. In this way, the backpropagation of the model gradient can be performed based on the loss between the predicted image and the real image, thereby updating the parameters of each model. Based on this model architecture, end-to-end closed-loop simulation training can be realized, reducing the error accumulation in the independent module serial architecture, so that the trained model can meet the requirements of high-level autonomous driving.

[0051] On another front, a method is proposed to decompose the global attention of the traditional Transformer into "same-type token attention" + "different-type token attention", which can greatly reduce the computational burden and is more suitable for in-vehicle platform applications.

[0052] In another aspect, a closed-loop simulation training strategy is proposed, which considers the impact of the vehicle's behavior on other traffic participants during simulation training, scores the predicted target trajectory and evaluates the quality of the predicted target vehicle trajectory, and stops the current round of training when an abnormal traffic event occurs, so as to ensure the adaptability of the trained model to the actual production environment of the vehicle.

[0053] Additional aspects and advantages of this application will be set forth in part in the following description of embodiments, and will be obvious from the description or may be learned by practice of this application. Attached Figure Description

[0054] Figure 1 is a typical autonomous driving software stack architecture diagram;

[0055] Figure 2 is an architecture diagram of an end-to-end visual motion model according to an embodiment of this application;

[0056] Figure 3 is an application scenario diagram of an end-to-end visual motion modeling method according to an embodiment of this application;

[0057] Figure 4 is an architecture diagram of an unmanned vehicle with an end-to-end visual motion model configured according to an embodiment of this application;

[0058] Figure 5 is a flowchart of an end-to-end visual motion model training method according to an embodiment of this application;

[0059] Figure 6 is an architecture diagram of the first model in an end-to-end visual motion model according to an embodiment of the present application;

[0060] Figure 7 is an architecture diagram of the second model in an end-to-end visual motion model according to an embodiment of this application;

[0061] Figure 8 is an architecture diagram of the third model in an end-to-end visual motion model according to an embodiment of this application;

[0062] Figure 9 is an architecture diagram of the fourth model in an end-to-end visual motion model according to an embodiment of this application;

[0063] Figure 10 is a schematic diagram of the principle framework of an end-to-end visual action model training method according to an embodiment of the present application;

[0064] Figure 11 is a block diagram of a vehicle end-to-end visual motion model training device according to an embodiment of this application;

[0065] Figure 12 is a block diagram of a trajectory planning device according to an embodiment of the present application;

[0066] Figure 13 is a hardware structure diagram of an unmanned vehicle according to an embodiment of this application. Detailed Implementation

[0067] The present application will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0068] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0069] Before introducing the embodiments of this application, the basic technologies and some technical terms involved in the embodiments of this application will be explained illustratively:

[0070] Autonomous driving systems: Systems that enable different levels of autonomous driving functions in vehicles, such as driver assistance systems (L2), high-speed autonomous driving systems requiring human supervision (L3), and highly / fully autonomous driving systems (L4 / L5).

[0071] Trajectory planning refers to the process of calculating a smooth trajectory for a vehicle given its initial state (including starting position, velocity, and acceleration), target state (including target position, velocity, and acceleration), obstacle locations, and dynamic and comfort requirements, and guiding the vehicle along this trajectory to the target state. Trajectory planning consists of two parts: path planning and velocity planning. Path planning calculates a smooth path from the starting position to the target position, while velocity planning calculates the velocity at each point along this path, forming a velocity curve.

[0072] World models (WMs) typically aim to learn spatiotemporal patterns from past data, thereby predicting future frames based on one or more past frames, regardless of the causal effects of intervention. One of the ultimate goals of world models is to create dream environments that allow machine agents to raise hypothetical questions within these created environments, simulating alternative futures not encountered in the real world.

[0073] Diffusion model: A generative model based on iterative denoising. It trains a deep network to denoise noisy data by simulating the forward denoising process of data diffusion. This allows randomly sampled noise to be transformed into samples of the target data distribution through tens to hundreds of denoising steps during the testing phase. The training objective of the diffusion model can also be viewed as an estimation of the fractional gradient.

[0074] Existing autonomous driving systems rely on the collaboration of various relatively independent modules to achieve vehicle path planning and automatic control. The following section, with reference to Figure 1, introduces a typical autonomous driving software stack architecture.

[0075] The perception module acts as the system's "eyes," responsible for identifying and understanding the vehicle's surroundings by processing raw sensor data. Its core tasks include target detection, classification, tracking, and semantic segmentation. The sensor input and data fusion module receives data from various sensors, such as cameras, LiDAR, and millimeter-wave radar. Through multi-sensor fusion algorithms, it integrates data from different sources into a unified coordinate system, forming a more comprehensive and reliable perception of the environment. For example, LiDAR provides precise 3D distance information, while cameras can identify colors, textures, and traffic signs; the two complement each other. The perception module outputs structured information, including the position, size, and speed of dynamic obstacles such as vehicles, pedestrians, and cyclists, as well as static environmental elements such as lane lines, traffic lights, and drivable areas. This information provides the foundation for all subsequent modules to understand the world.

[0076] The positioning module is responsible for answering the fundamental question "Where am I?" by determining the vehicle's position and attitude (orientation) in real-time with centimeter-level precision on a high-definition map. Since single positioning technologies often have limitations, this module typically employs a multi-sensor fusion approach, such as Global Navigation Satellite System (GNSS), particularly when combined with Real-Time Kinematic (RTK) technology. This provides absolute position, but signals are easily lost in tunnels or urban canyons. Inertial Measurement Unit (IMU) infers position by measuring acceleration and angular velocity, offering high short-term accuracy but suffering from cumulative errors. Matching lidar point clouds or camera visual features with a pre-made high-definition map (HD Map) can correct for accumulated errors, achieving high-precision positioning. This module ultimately outputs the vehicle's precise pose (X, Y, Z coordinates, heading angle, pitch angle, etc.) in the high-definition map coordinate system, providing the foundation for path planning and control.

[0077] The prediction module acts as a "prophet," predicting the possible trajectories of obstacles within the next few seconds based on obstacle information output by the perception module. The module's input primarily includes the perceived obstacle's historical trajectory, the vehicle's status, and prior information such as lane topology and traffic rules provided by high-precision maps. The output consists of multiple probabilistic trajectories for the obstacle. For example, it predicts whether the vehicle ahead will maintain its lane or prepare to change lanes. The accuracy of the prediction module's results is often measured using metrics such as Average Distance Error (ADE), Final Displacement Error (FDE), and False Negative Rate (MR).

[0078] The planning and control module acts as the system's "decision brain" and "cerebellum," responsible for formulating safe, comfortable, and efficient driving strategies and translating them into specific vehicle control commands. Planning typically involves three layers: Route Planning, which generates a global path based on the destination, similar to a route provided by navigation software; Behavioral Decision, which determines the vehicle's instantaneous behavior based on the current traffic scenario (such as intersections and following traffic), including acceleration, deceleration, lane changing, or yielding; and Motion Planning, which generates a local, smooth, collision-free trajectory, including waypoints and speed planning. The control module receives the planned trajectory and uses algorithms (such as PID control and model predictive control) to calculate specific commands such as steering wheel angle, throttle, and brake opening, driving the vehicle to precisely follow the trajectory.

[0079] High-precision maps are the "ultimate memory" of autonomous driving systems, far surpassing ordinary navigation maps. The map module contains lane-level, centimeter-level precision information, such as the precise geometry of lanes (width, curvature), lane line types (solid / dashed), traffic signs, traffic light locations, and even rich semantic and attribute information like road slope and curvature. This information provides vehicles with beyond-line-of-sight perception capabilities and enables them to accurately understand road rules.

[0080] With the development of autonomous driving technology, unmanned logistics is reaching a critical inflection point, moving from the pilot exploration stage to the stage of large-scale commercial deployment, which also brings about higher-level demands for autonomous driving. How to better handle the coupling between system modules, reduce information loss, and improve the system's ability to handle application uncertainties are directions that the field of autonomous driving continues to explore.

[0081] As shown in Figure 2, in order to solve the above problems, this application proposes an end-to-end visual motion model architecture, which can perform end-to-end closed-loop simulation training and perform efficient, accurate and customizable trajectory planning based on the trained model.

[0082] Overall, this end-to-end visual action model comprises a first model, a second model, a third model, and a fourth model. The first model is a pre-fusion model. During the training phase, given camera image data at time T, the first model processes this image data and outputs the perception result at time T, along with the corresponding perception task query and image token. The second model is an imitation learning model that outputs the predicted trajectory of the target object and the planned trajectory of the vehicle based on the perception task query output by the first model. The third model is a closed-loop simulation model that evaluates the quality and score of the planned trajectory of the vehicle output by the second model and performs a forward simulation to update the vehicle's state (i.e., update to obtain the predicted trajectory of the vehicle at time T+1). The fourth model is a world model that outputs the predicted camera image data at time T+1 based on the updated vehicle state and the image token output by the first model. This end-to-end visual action model can backpropagate the model gradient stepwise to the first model based on the camera image data at time T+1, thereby updating the model's parameters.

[0083] During training, the first, second, third, and fourth models can be trained in stages. For example, the first model can be trained first, then the first and second models can be trained together, then the first and fourth models can be trained together, and finally, the first, second, third, and fourth models can be trained in a closed end-to-end manner. Alternatively, the first, second, third, and fourth models can be pre-trained using other combinations, or a closed end-to-end training can be performed directly on the first, second, third, and fourth models.

[0084] Furthermore, to adapt to different application scenarios or preferences, the reward function can be adaptively adjusted during the model training phase by using different weights. For example, for scenarios requiring a more conservative autonomous driving strategy, penalties such as safe distance, rapid acceleration / deceleration, deviation from traffic rules, and comfort (acceleration) can be assigned higher weights, while travel time and following distance can be assigned lower weights. Conversely, for scenarios requiring a more aggressive autonomous driving strategy, travel time (encouraging efficiency), average speed, and close but safe following distance can be assigned higher weights, while conditions such as smoothness and comfort can be assigned lower weights. The tuning of these different driving strategy planning styles can also be completed in the post-training phase of the model. For example, based on an end-to-end visual-action model that has already been trained with general settings, a second model can be post-trained to meet the above requirements. These specific training methods will be described in detail in the embodiments below.

[0085] Figure 3 is a schematic diagram of an optional system architecture involved in an embodiment of this application. As shown in Figure 3, at time T, the autonomous vehicle collects image data of the environment through visual sensors (such as cameras). This image data is directly or indirectly uploaded to the server by the autonomous vehicle or an intermediate medium. The server plans the trajectory of the autonomous vehicle at time T using a first model and a second model, and obtains the predicted trajectory of traffic participants in the current environment at time T. The server then uses a third model, the planned trajectory of the autonomous vehicle at time T, and the predicted trajectory of traffic participants in the current environment to determine the predicted trajectory of traffic participants in the current environment at time T+1 and updates the predicted trajectory of the autonomous vehicle at time T+1. The server also uses a fourth model, the predicted trajectory of the autonomous vehicle at time T+1, the planned trajectory of the autonomous vehicle at time T, and the image data to determine the image data that the autonomous vehicle may observe at time T+1, i.e., the predicted image at time T+1. Similarly, the autonomous vehicle continuously collects image data of the environment through visual sensors, such as the image data at time T+1. In this way, the gradient of each model can be calculated by the difference between the real image of the physical world and the predicted image. In some scenarios, the autonomous vehicle transmits closed-loop image data back in real time, which can be used by the server to continuously train models based on real road scenarios and perform trajectory planning for the autonomous vehicle.

[0086] Referring again to Figure 3, a specific scenario applying the end-to-end visual action model training method and trajectory planning method of this application may include a server and terminal devices. The server can be a backend server corresponding to the client, or a server specifically used for category recognition; this application does not impose specific limitations. Specifically, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices may include in-vehicle terminals and user terminals. In-vehicle terminals may include vehicle computers or on-board units (OBUs). In-vehicle terminals may also be applications (APPs) on the terminal, APPs on smart rearview mirrors, APPs or mini-programs on mobile phones, etc., without limitation. User equipment (UE) can be either wireless or wired. Wireless terminal equipment can refer to a device with wireless transceiver capabilities. User equipment can be a mobile phone, tablet, computer with wireless transceiver capabilities, virtual reality (VR) user equipment, augmented reality (AR) user equipment, intelligent voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., without limitation.

[0087] The terminal communicates with the server via a network. The data storage system can store the data the server needs to process; it can exist independently or be integrated into the server. Training and trajectory planning for the end-to-end visual motion model can be completed on the server or the terminal. For example, the server can perform training of the end-to-end visual motion model. The trained model (which can be compressed, distilled, and lightweighted) is deployed on the in-vehicle terminal. The server can continuously train the model and periodically update the model deployed on the terminal. During autonomous driving, the driverless car directly acquires environmental images and calls the model on the in-vehicle terminal for inference to obtain the planned trajectory. Alternatively, the end-to-end visual motion model can be trained and stored by the server. During autonomous driving, the driverless car uploads the acquired environmental images to the server in real time, and the server calls the model for inference to obtain the planned trajectory.

[0088] It should be noted that, regardless of the type of server / terminal device, the end-to-end visual motion model and vehicle trajectory planning method provided in this application embodiment can be adapted to the autonomous driving system of unmanned vehicles, including autonomous driving systems at levels L2, L3, L4 and above.

[0089] Referring to Figure 4, a functional framework diagram of an unmanned vehicle applicable to embodiments of this application is shown. As shown in Figure 4, the unmanned vehicle may include a perception system, a display device, and a computing platform. For example, the perception system may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou system, or another positioning system. As another example, the perception system 120 may also include one or more of an inertial measurement unit (IMU) and a camera device.

[0090] Some or all of the functions of an autonomous vehicle can be controlled by a computing platform. This platform can include more than one processor, which can be a circuit capable of reading and executing instructions, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU), or digital signal processor (DSP). The processor can also implement certain functions through the logical relationships of hardware circuits. These logical relationships can be fixed or reconfigurable. For example, the processor can be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as a type of ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), or deep learning processing unit (DPU). In addition, the computing platform may also include memory for storing instructions, and some or all of the processors in the processor can call the instructions in memory to perform the corresponding functions.

[0091] The display device can be a traditional in-vehicle display screen or a projection display screen (such as a head-up display, HUD). An in-vehicle display screen is a physical display screen and an important component of the in-vehicle infotainment system. Autonomous vehicles can have multiple displays, such as digital instrument cluster displays and central control screens. In some possible implementations, one or more of these in-vehicle displays can be human-machine interfaces (HMIs), for example, the central control screen can be an HMI. HUDs include, for example, combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. Of course, in some applications of autonomous vehicles, the display device may be unnecessary, especially in scenarios where human operation is not required.

[0092] Referring specifically to Figure 5, an embodiment of the training method for the end-to-end visual action model of this application is first introduced. In this embodiment, the method includes:

[0093] S11. Input the sample image at time T into the first model to obtain the perception task query and image token corresponding to the image perception result at time T.

[0094] The process of converting an image into a token involves structuring the image and transforming it into a series of continuous vectors that can represent local visual information. Referring to Figure 6, the image token can be obtained through feature extraction via the feature extraction network in the first model. The feature extraction network can be a network model with a Residual Network (ResNet), ResNeXt, or a VisionTransformer (ViT) as its backbone network. This embodiment does not limit the backbone network used in the feature extraction model.

[0095] In one embodiment, to meet the requirements of low power consumption and high performance in onboard chip applications for autonomous vehicles, Neural Architecture Search (NAS) technology can be used to search for suitable feature extraction network structures. The core idea of ​​NAS is to automatically explore different network structures using algorithms to discover the most effective model for a specific task. In a specific embodiment, a suitable search space can be predefined, containing various possible network architectures. The NAS algorithm can employ methods such as reinforcement learning, evolutionary algorithms, or gradient descent to search within this space and iteratively optimize by evaluating the performance of different architectures. Performance evaluation can be based on feedback from a validation set, allowing the algorithm to select the architecture with better performance. Furthermore, during the search process, the NAS algorithm can continuously generate new network architectures and perform training and evaluation. Based on these evaluation results, the search strategy is learned and improved to more efficiently explore architectures with better performance. As the search iterates, the algorithm gradually focuses on network structures with high-performance metrics that can meet the needs of onboard chip applications. Experimental results show that this method can be used to search for feature extraction network structures with performance comparable to the classic ResNet-50 and inference time comparable to ResNet-34.

[0096] In the specific process of obtaining image tokens, a "block embedding" method can be used. For example, firstly, the input image (e.g., 224*224*3, i.e., height, width, and RGB color channels) is uniformly divided into N square image blocks, each of which is 16*16 in size, for a total of (224 / 16) * (224 / 16) = 196 image blocks; then, each image block is flattened into a vector of length 16 * 16 * 3 = 768, and this vector is linearly projected onto a set dimension D. This projection process can be regarded as each image block learning its corresponding embedding representation; finally, the D-dimensional vector obtained by projection is an image token. Learnable positional codes can also be added to the obtained N image tokens to store the position information of the corresponding image token in the original image. Alternatively, a convolutional approach can be used, treating the above "blocking + projection" process as a convolutional layer operation where the kernel size and stride are equal to the image block size. This convolutional layer directly maps the image from (H, W, C) to a feature map of (N, D), and after slight deformation, the image token sequence can be obtained. This application does not restrict this process.

[0097] The perception task query can be randomly initialized, for example, 100, before model training. After the sample image is processed by the first model, multiple image features are extracted. Each perception task query typically queries all locations of these image features. During this process, an attention mechanism can be used to collect the most relevant features from the global image information. For example, a particular query might focus on features near the wheels of an autonomous vehicle in the center of the sample image. Thus, each perception task query gradually collects information through interaction with image features and outputs the corresponding prediction result, for example, through the image sensing head described later. It is understood that in the embodiments of this application, the perception task query obtained after processing by the first model is the query after interacting with the image features, rather than the initial query without model processing.

[0098] In this embodiment, the first model includes a feature extraction network and an image sensing head. Step S11 specifically includes: extracting image features of the sample image at time T based on the feature extraction network and converting them into bird's-eye view BEV features; fusing the BEV features from time T and historical times to obtain fused BEV features; and obtaining the perception task query and image token corresponding to the image perception result at time T based on the fused BEV features and the image sensing head.

[0099] BEV feature transformation can be achieved through various spatial transformation schemes, such as LSS, BEVFormer, Dense AT-attention, and IPM. Specifically: In the LSS scheme, a discrete depth probability distribution (e.g., from 1 meter to 100 meters, divided into several intervals) is predicted for each pixel in the image feature map. The features of each pixel are "painted" onto all possible depth intervals according to its depth distribution, forming a 3D feature volume in 3D space. Then, this 3D feature volume is pooled along the height axis and projected onto the BEV plane to obtain BEV features. In the BEVFormer scheme, a set of learnable 2D grid-like queries (BEV Queries) is defined on the BEV plane. Each BEV Querie corresponds to a specific location (grid) on the BEV plane and is projected onto the feature map of all panoramic images through 3D spatial location and camera parameters. Deformable attention interactions are performed on the image regions within its field of view to collect features. The BEV Queries also interact with BEV features from historical moments to fuse temporal information, ultimately outputting BEV features containing spatiotemporal context. In the Dense AT-attention scheme, a dense 2D mesh is predefined on the BEV plane. Each mesh point, based on a preset height (e.g., vehicle height 0), is back-projected onto the pixel coordinates of all surrounding images using camera intrinsic and extrinsic parameters. For each BEV point, features are sampled from all the image feature maps it has projected onto using bilinear interpolation. Then, these multi-view features are fused using an MLP or a simple attention mechanism to obtain the features of that BEV point. The IPM scheme is based on the assumption that the world (ground) is a perfect plane and that the relative pose of the camera capturing the sample image and the ground is known. By utilizing inverse perspective transformation in perspective geometry, a one-to-one mapping relationship is established between points on the BEV plane and corresponding points on the image plane, thereby directly resampling pixels from the image feature map or the original image to the corresponding positions on the BEV plane to generate the BEV feature map.

[0100] In one embodiment, optimizations such as operator merging and int8 quantization can be performed based on the LSS scheme to meet the application requirements of automotive chip platforms. Exemplarily, a convolutional layer, a batch normalization layer, and a ReLU activation function can be merged into a single "large convolution" operation, or a fully connected layer plus a biased linear layer can be merged. Alternatively, in operations such as view transformation and voxel pooling, continuous deformation (Reshape), transpose, and index gathering operations can be merged or optimized to reduce the generation of intermediate tensors. int8 quantization can be post-training quantization, for example, after model training, using a calibration dataset to statistically analyze the distribution (range) of activation values ​​in each layer of the network to determine a quantization parameter; or, for example, during the fine-tuning phase of model training, quantization effects can be simulated in forward propagation (by adding "pseudo-quantization" nodes) to allow the model weights to "adapt" to this low-precision representation during training.

[0101] In this embodiment, the sample images at time T may include images of the target vehicle from at least two perspectives (multi-view images). The BEV features can be based on the BEV space centered on the autonomous vehicle body. By fusing the BEV features from time T and historical times, temporal stability can be enhanced. For example, if the autonomous vehicle is equipped with 11 cameras, it will acquire data from all 11 cameras simultaneously. After passing through the feature extraction network, the corresponding multi-scale image features are extracted. After converting to BEV space, the BEV characteristics at time T are obtained. Compare the BEV characteristics at time T with the BEV characteristics at the previous k times. The fusion process is performed (e.g., by aligning the BEV features from historical moments with the BEV features at moment T using location odometers) to obtain fused BEV features. .

[0102] In one embodiment, when fusing BEV features from different time points, a TAM (Temporal Attention Module) can be used. For example, for k+1 feature maps to be fused, a spatiotemporal attention weight map is learned to dynamically determine how much information to extract from each time point and each spatial location. It can be understood that, compared to simple average pooling or max pooling methods, this approach, by considering spatiotemporal relationships, allows the model to distinguish the importance of different time points while maintaining spatial structure and independently calculating temporal weights for each spatial location. This scheme is end-to-end trainable and can be optimized together with the complete end-to-end visual-action model provided in this application.

[0103] Understandably, for the first model, the feature map of the sample image after passing through the feature extraction network includes a key and a value. The key can be understood as the "label" of the image features, and the value can be understood as the "content" of the image features. The image perception head can calculate the similarity or relevance between the perception task Query and the key based on their matching, thereby obtaining attention weights. The weighted value corresponding to these weights is used for the final perception output of the sample image. Furthermore, the perception task Query and image token corresponding to the image perception result can be determined.

[0104] The type of sensing head can be adaptively selected in different application scenarios. For example, there are sensing heads for large obstacles such as large motor vehicles, sensing heads for small, shape-variable, and unpredictable vulnerable road users (such as pedestrians and bicycles), sensing heads for road structures such as lane lines, road edges, and drivable areas, and sensing heads for traffic facilities such as traffic signs and traffic lights. Each type of sensing head can be designed specifically for a particular purpose. For example, a large obstacle sensing head can use a standard 2D / 3D bounding box sensing head, while a road structure sensing head can output pixel-level masks for road structure instances or classify grid points on the BEV feature map (e.g., whether they belong to lane lines), etc., which will not be elaborated on in this application.

[0105] In this embodiment, the image perception head includes at least one of a target detection head, a lane line recognition head, and an occupancy grid prediction head. In the application scenario of autonomous vehicles, the target detection head is used to detect other vehicles, pedestrians, and other dynamic and static traffic participants around the vehicle, providing key environmental perception information for the vehicle's decision-making and planning, and ensuring the basic safe operation of the vehicle; the lane line recognition head can identify lane lines, roadside positions, and the topological structure between lanes around the vehicle, solving the problem of low freshness of traditional high-precision maps by constructing local maps in real time; the occupancy grid prediction head is used to detect objects outside the ODD (Operational Design Domain), such as trucks carrying trees, construction areas on the road, and randomly piled sand.

[0106] S12. Based on the perception task Query and the second model at time T, obtain the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T.

[0107] Referring to Figure 7, the perception task Query retains rich information and uncertainty in the original features of the sample image. Compared to using the perception task Value, this implementation uses the perception task Query as input to the second model, enabling the second model to make "soft decisions" based on more complete features and generate more reasonable planned trajectories. Specifically, the perception task Query may implicitly contain the first model's focus information and prior assumptions about the sample image. Passing it to the second model is equivalent to transmitting the first model's metacognitive judgment of the scene in the image. The model thus possesses stronger cognitive coherence. The trajectory planning and prediction of the second model are not passive responses to the perception results, but actively continue the reasoning logic of perception, which also eliminates information transmission errors to some extent. In addition, in the end-to-end training framework, the gradient of the perception task Query is directly related to the original task loss. The second model's prediction and planning based on the perception task Query enables the gradient to propagate back more effectively. Conversely, if the perception task Value is used, gradient information is more likely to decay during propagation.

[0108] In the implementation scheme of the second model proposed in this embodiment, the architecture of the second model adopts an architecture based on multimodal token representation and fusion. It encodes different categories of scene information through an encoder designed for it, performs deep interaction modeling based on the Transformer network, and finally generates robust prediction and planning results through a diffusion model.

[0109] Specifically, in this embodiment, the second model includes a dynamic target encoder, a local map encoder, a static target encoder, a global navigation perception machine, and a first diffusion model. Step S12 specifically includes obtaining at least one of the following based on the perception task Query at time T and the dynamic target encoder, local map encoder, and static target encoder: dynamic target token, local map token, and static target token; fusing information based on the dynamic target token, local map token, and static target token to obtain a scene fusion token; obtaining a global navigation token based on global navigation instructions and the global navigation perception machine; and obtaining the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the scene fusion token, global navigation token, and the first diffusion model.

[0110] Dynamic target encoders are primarily used to encode the historical state information of dynamic targets, such as traffic participants like vehicles, pedestrians, and bicycles. This historical state information can include the dynamic target's position, velocity, and heading angle. For example, VectorNet can be used as an encoder. VectorNet abstracts all dynamic targets in a scene as polylines, each composed of a series of continuous vector segments (Vectors). Each vector segment (e.g., from point p1 to point p2) is represented as a node, with attributes such as starting coordinates and vector displacement. Then, all nodes belonging to the same polyline are combined into a subgraph, and the local structural information within the polyline is aggregated, outputting a feature vector representing the entire polyline for each subgraph. Finally, all feature vectors of the polylines are used as nodes to construct a fully connected global graph. For example, an attention mechanism can be used to allow all polyline nodes to interact, enabling the model to learn the global dependencies between scene elements, completing the encoding of the dynamic target and obtaining a dynamic target token. VectorNet's encoding scheme avoids the information loss associated with traditional rendering methods and significantly reduces computational resource overhead.

[0111] Static target encoders are primarily used to encode static targets, such as cones and fences in a scene. The encoded information can include the type and location of the static target. Regarding encoder selection, VectorNet can be used similarly, or MLPs (Multilayer Perceptrons) can be directly employed. MLPs arrange the vector representations of static targets in a fixed order to form a long vector, which is then input into an MLP formed by stacking multiple fully connected layers and activation functions (such as ReLU). Through nonlinear transformation, the mapping from the original coordinate sequence to a low-dimensional semantic feature vector is learned, completing the encoding of the static target and obtaining a static target token. Compared to using VectorNet, the MLP encoding scheme has higher computational efficiency and lower memory footprint when processing static, structurally stable targets, making it more suitable for the needs of onboard chip applications in autonomous vehicles.

[0112] The local map encoder is used to encode the local map surrounding the autonomous vehicle, such as lane lines and curbs in high-precision map information. The encoder selection scheme can similarly employ VectorNet, MLPs, etc., which will not be elaborated upon here. The local map token obtained by the local map encoder can provide the model with accurate prior knowledge of the road structure.

[0113] It can be seen that dynamic target token, local map token, and static target token are the main information representations that may affect the autonomous vehicle's decision-making in the scenario. The scenario fusion token obtained by fusing the three can relatively completely represent the current environment of the autonomous vehicle, providing an important information foundation for the trajectory planning of the autonomous vehicle.

[0114] The global navigation perception unit is used to encode the global navigation commands of the autonomous vehicle, such as the final destination and navigation waypoints. The encoder selection scheme can similarly employ VectorNet, MLPs, etc., which will not be elaborated upon here. The global navigation token obtained by the global navigation perception unit can be further integrated into the current scene understanding, enabling the first diffusion model to perceive scene environment information and global navigation intent based on the scene fusion token and the global navigation token, thereby obtaining the predicted trajectory of target objects and the planned trajectory of the target vehicle in the scene.

[0115] It should be noted that when the first diffusion model performs trajectory prediction of the target object and trajectory planning of the target vehicle based on the scene fusion token and the global navigation token, the scene fusion token and the global navigation token can be fused again in advance, or they can be directly input into the first diffusion model for processing. This application does not impose any restrictions on this.

[0116] In the implementation scheme of the second model proposed in this embodiment, the second model adopts an improved Transformer network as the backbone network for the aforementioned information fusion. The improvement aims to effectively control the computational complexity of the Transformer's self-attention mechanism, which increases sharply with the number of tokens, while maintaining strong modeling capabilities. The core idea of ​​this scheme is: based on the autonomous vehicle application scenario, the global attention of the Transformer is decomposed into "internal attention within the same type of token" and "attention between different types of tokens," thereby significantly reducing the computational burden while ensuring sufficient information interaction among dynamic targets, local maps, and static targets.

[0117] Specifically, in this embodiment, firstly, attention weights of any token among the dynamic target token, local map token, and static target token relative to tokens of the same class are calculated based on the Transformer network, thereby obtaining the first fused dynamic target token, the first fused local map token, and the first fused static target token; then, attention weights of at least one of the first fused dynamic target token, the first fused local map token, and the first fused static target token relative to tokens of different classes are calculated based on the Transformer network, thereby obtaining at least one of the second fused dynamic target token, the second fused local map token, and the second fused static target token.

[0118] For example, suppose the dynamic target tokens include: {Token 1a, Token 2a, …, Token Na}, the local map tokens include: {Token 1b, Token 2b, …, Token Nb}, and the static target tokens include: {Token 1c, Token 2c, …, Token Nc}. Then, for the dynamic target tokens, each of {Token 1a, Token 2a, …, Token Na} is a token of the same class relative to the others, while each of {Token 1b, Token 2b, …, Token Nb} and {Token 1c, Token 2c, …, Token Nc} is a token of a different class.

[0119] The above solutions take into account that targets of the same type may have similar dynamic models or behavioral patterns, and the attention weight calculation of tokens of the same type can learn these shared patterns more efficiently; while between dissimilar objects, they may need to follow clear traffic rules or have physical constraints, and do not require completely dense attention connections.

[0120] In one embodiment, for each of the three types of tokens, a subset of tokens can be selected, and the attention of these selected subsets relative to other selected heterogeneous tokens can be calculated. Alternatively, for each selected subset of tokens, the attention of the selected subsets relative to one other heterogeneous token category can be calculated. When selecting a subset of tokens from each type, methods such as sliding windowing or random selection can be used, and this application does not impose any restrictions on this method.

[0121] Taking a single-layer attention interaction as an example, we can illustrate the effect of the above scheme on reducing computational burden. The second model first calculates the attention of tokens of the same class for dynamic target tokens, local map tokens, and static target tokens, with a computational cost of 3*N*N. Then, it selects k (k < N) tokens from each of the three classes and calculates their attention relative to the other selected tokens of different classes, with a computational cost of 6*k*k. It can be seen that the smaller the value of k, the more significant the computational saving compared to the global attention computational cost (9*N*N).

[0122] Furthermore, it should be noted that the architecture of the second model shown in Figure 7 is illustrated using an example containing one layer of "same-type token attention + different-type token attention," which is not a limitation on the architecture of the second model. In practical applications, it can be, for example, stacked multiple layers of attention (e.g., 6 to 12 layers), and the attention of some of these layers can be set as same-type token attention, different-type token attention, or a combination of both. This application does not impose any restrictions on this.

[0123] S13. Based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model, obtain the predicted trajectory of the target vehicle at time T+1.

[0124] In this embodiment, one of the functions of the third model is to predict the trajectory of the target vehicle at time T+1 based on the predicted trajectory of the target object at time T and the planned trajectory of the target vehicle. The predicted trajectory of the target vehicle at time T+1 will be used in the fourth model described below to predict the camera data of the target vehicle at time T+1. Here, the interval between time T+1 and time T can also be set according to different application scenarios of the autonomous vehicle, such as 50ms, 100ms, 200ms, etc., and this application does not limit it in this way.

[0125] In one embodiment, another function of the third model is to construct a closed-loop simulation system. Specifically, the traditional autonomous driving simulation system only simulates the behavior of the vehicle itself, assuming that the vehicle's behavior does not affect other traffic participants in the scene. This open-loop simulation strategy may lead to a significant discrepancy between the offline simulation results and the actual effects of the vehicle-side production environment. The closed-loop simulation system constructed based on the third model can predict the trajectories of these traffic participants at time T+1 using environmental information at time T, such as the trajectory information of traffic participants (target objects) around the vehicle, local maps, and global navigation information. In this way, it is equivalent to obtaining the environmental information at time T+1 (i.e., the trajectories of the traffic participants at time T+1 and the trajectory of the target vehicle at time T+1), thereby enabling the third model to perform further evaluations based on this information. This part will be specifically described in the embodiments below.

[0126] Referring to Figure 8, the functionality of the third model depends on the second model. In the actual architecture, the third model can include an independent module similar to the second model, or it can be built directly based on the second model. This application does not impose any restrictions on this.

[0127] In this embodiment, the training method further includes obtaining the predicted trajectory of the target object at time T+1 based on the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T, as well as the third model; and obtaining at least one of the following based on the predicted trajectory of the target vehicle at time T+1, the predicted trajectory of the target object, and the third model: a score of the probability of the target vehicle experiencing an abnormal traffic event at time T+1, a score of the target object complying with preset traffic rules, and a score of the target vehicle completing the global navigation route.

[0128] In different embodiments, the third model can perform the above scoring steps based on a suitable scheme.

[0129] For example, to score the probability of abnormal traffic events, the trajectories of the target vehicle and the target object at time T+1 can be projected onto a unified spatiotemporal coordinate system, and the positional overlap can be checked; or, the lateral displacement change of the target object at time T+1 can be identified to determine whether the safe distance is insufficient after cutting in (i.e., whether it is a malicious cut-in); or, the acceleration and jerk of the target object can be calculated to determine whether they exceed the danger threshold.

[0130] For example, to score compliance with preset traffic rules, one could query the lane line type corresponding to the trajectory point of the target object at time T+1 and detect whether the target object crossed lanes within the solid line section; or, one could calculate the lateral distance between the target object and the lane center line and count the proportion of deviations from the lane center line exceeding a set threshold (e.g., 25%); or, one could detect the trajectory speed of the target object and compare it with the road segment speed limit information provided by the map to determine whether the speed limit was exceeded or the proportion of exceeding the speed limit.

[0131] For example, to score the completion of a global navigation route, the offset distance between the target vehicle's trajectory point and the reference path at time T+1 can be calculated, and the root mean square error can be calculated; or, from an efficiency perspective, the ratio of the straight-line distance from the trajectory's starting point to the actual driving distance corresponding to the trajectory can be calculated.

[0132] In one embodiment, if an abnormal traffic event occurs to the target vehicle at time T+1, the method further includes stopping the training of the end-to-end visual motion model for this round. The abnormal traffic event can be predefined or determined by the model's scoring of traffic events. For example, if the target vehicle collides with other road users in the environment, or if the target vehicle drives into a prohibited area outside the map, these events can be recorded as failures. If no abnormal traffic event occurs to the target vehicle at time T+1, the closed-loop training process of the entire end-to-end visual motion model continues.

[0133] S14. Based on the predicted trajectory of the target vehicle at time T+1, the planned trajectory of the target vehicle at time T, the image token, and the fourth model, obtain the predicted image at time T+1.

[0134] S15. Perform backpropagation of the model gradient based on the predicted image at time T+1 to update the parameters of the first model, the second model, the third model, and the fourth model.

[0135] S16. Construct an end-to-end visual action model based on the first, second, third, and fourth models that have been trained.

[0136] Overall, in this process, the fourth model can use the state of the target vehicle at time T+1 as a condition, fusing camera data from time T and the target vehicle's state to predict the image at time T+1. Based on the loss between the predicted image and the real image (observed image) at time T+1, the gradient can be backpropagated to update the parameters of each base model in the end-to-end visual action model. In short, the fourth model learns rich spatiotemporal representations through historical camera data and the target vehicle's state, thereby generating highly realistic test scenarios during closed-loop simulation. Simultaneously, within the overall architecture of the end-to-end visual action model, it improves the accuracy and robustness of tasks such as target object trajectory prediction and behavior analysis.

[0137] Referring to Figure 9, specifically in this embodiment, the fourth model includes an encoder and a second diffusion model. Step S14 includes obtaining the state code of the target vehicle at time T based on the planned trajectory of the target vehicle at time T and the encoder; fusing the state code of the target vehicle at time T as a position code with the image token to obtain an image fusion token; and obtaining the predicted image at time T+1 based on the predicted trajectory of the target vehicle at time T+1, using the image fusion token and the second diffusion model.

[0138] The state encoding of the target vehicle can be implemented, for example, using MLPs, which will not be elaborated here. In one embodiment, the target vehicle's planned trajectory and image token at time T may not be at a single time, but rather the planned trajectories and image tokens corresponding to sample images of the target vehicle at k past time points. For example, the sample images at time T include... The target vehicle trajectory (including status information) includes The image tokens generated from these sample images at several time points are fused with the state codes generated from the corresponding target vehicle trajectories to obtain image fusion tokens.

[0139] In the architecture of the fourth model, the fourth model can also include a built-in encoder (Tokenizer) for image encoding, allowing it to be invoked independently in certain applications. Furthermore, when the encoder encodes images at multiple time points, it can achieve cross-time feature space alignment through weight sharing, laying the foundation for temporal fusion. Similarly, the encoder here can be a network model with a visual transformer or CNN as its backbone; this application does not impose any restrictions on this.

[0140] As can be seen, the training process of the fourth model has significant advantages in efficiency and scalability, as it does not rely on additional manually labeled data. In autonomous vehicle applications, training can be completed entirely based on closed-loop data transmitted back from the vehicle in real time, and can be accomplished, for example, through a self-supervised learning mechanism. This approach not only significantly reduces the cost and time of data labeling but also enables the model to continuously learn and evolve autonomously from real road scenarios, possessing strong online adaptability. In specific embodiments, for example, sample images transmitted back by autonomous vehicles can be used to design tasks such as "predicting the next frame" and "filling in occluded blocks," allowing the model to learn to extract stable, useful, and causally related features from the scene. Another example is having the model attempt to predict how the environmental state will change after a given action, thus achieving dynamic learning of the environment.

[0141] Referring to Figure 10, the training methods for end-to-end visual action models according to various embodiments of this application have been described above. In some embodiments, before performing end-to-end model training, individual models or several models may be pre-trained jointly. For example, before performing end-to-end model training, training of a first model, joint training of the first and second models, joint training of the first and fourth models, etc., may be performed first. These pre-training of individual models or multiple models may have a predetermined order, for example, training of the first model first, then joint training of the first and second models, and finally joint training of the first and fourth models. It should be understood that the selection and order of these pre-training modules are merely illustrative and not limiting.

[0142] Pre-training allows a single model or a combination of models to possess basic capabilities before end-to-end training. For example, if the first model can reliably perceive sample images, it can output reliable perception task queries and image tokens during end-to-end training. This ensures that the gradients are calculated based on relatively reasonable intermediate results during subsequent backpropagation of the model's gradients, making the model optimization process smoother and faster.

[0143] Referring again to Figure 10, in some embodiments, the end-to-end visual action model further includes a reward model and a cost model. After performing end-to-end model training, post-training can be performed on the second model, enabling it to plan trajectories that satisfy specific preferences. Specifically, the planned trajectory of the target vehicle output by the second model can be subject to safety constraints based on the cost model, and the second model can be trained with the goal of maximizing the reward score of the reward model for the planned trajectory of the target vehicle output by the second model.

[0144] As an example, the preference for target vehicle trajectory planning can be reflected in the score of the reward model, while the constraints of some objective safety rules can be reflected in the score of the cost model. In this way, through the design of the reward model and the cost model, the subjective preferences and objective safety rules of autonomous driving are unified in a rigorous optimization problem, improving the trajectory planning capability of the second model and its adaptability in different application scenarios.

[0145] For example, the reward model's score can guide whether a preference is to maintain a larger following distance (conservative strategy) or to closely follow the car in front to improve driving efficiency (aggressive strategy). The cost model's constraints include the distance to obstacles needing to be greater than a set threshold, and the trajectory not being able to cross solid lines to change lanes. Thus, when the second model plans a trajectory, the cost model acts as a "safety filter," calculating the total cost of the trajectory. If the total cost exceeds the set expected total cost, it is directly rejected or a higher penalty is imposed. Among the trajectories that satisfy the cost model's constraints, the reward model selects the trajectory with the highest reward score or guides the second model to continue optimizing in the direction of maximizing the reward score.

[0146] The following describes an embodiment of the trajectory planning method provided in this application. In this embodiment, the method includes acquiring an environmental image of a target vehicle and processing it using a target model to obtain a planned trajectory for the target vehicle. The target model includes a second model in the end-to-end visual-action model trained in the above embodiments.

[0147] Specifically, the architecture of the second model has been exemplarily described in the foregoing embodiments. During trajectory planning, the dynamic target encoder, local map encoder, and static target encoder of the second model can directly encode the environmental image of the target vehicle to obtain dynamic target tokens, local map tokens, and static target tokens. These different types of tokens can selectively interact through attention mechanisms such as "internal attention within similar tokens" and "inter-attention between dissimilar tokens," thereby further fusing them into a scene fusion token. The global navigation perceptron of the second model can encode the global navigation commands of the target vehicle, and the first diffusion model of the second model can predict the trajectories of each target object in the current scene based on the scene fusion token and the global navigation token, while simultaneously completing the trajectory planning of the target vehicle.

[0148] In one embodiment, the target model called during trajectory planning may also include the first model in the end-to-end visual action model trained in the above embodiments. The first model is called to process the environmental image of the target vehicle, obtain obstacle information in the target vehicle environment, and perform collision detection on the planned trajectory based on the obstacle information.

[0149] Specifically, the architecture of the first model has been exemplarily described in the foregoing embodiments. The feature extraction network and image sensing head of the first model can work together to detect traffic participants of specific targets and some objects outside the detected ODD range from the environmental image of the target vehicle. These can be used as obstacle information to perform collision detection with the trajectory planned by the second model, thereby providing a "backup" for the planned trajectory output by the second model.

[0150] Referring to Figure 11, an embodiment of the training device for the end-to-end visual action model of this application is introduced. In this embodiment, the training device for the end-to-end visual action model includes a first calling module 211, a second calling module 212, a third calling module 213, a fourth calling module 214, a parameter update module 215, and a model building module 216.

[0151] The first calling module 211 is used to input the sample image at time T into the first model to obtain the perception task query and image token corresponding to the image perception result at time T; the second calling module 212 is used to obtain the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the perception task query and the second model at time T; the third calling module 213 is used to obtain the predicted trajectory of the target vehicle at time T+1 based on the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T, as well as the third model; the fourth calling module 214 is used to obtain the predicted image at time T+1 based on the predicted trajectory of the target vehicle at time T+1, the planned trajectory of the target vehicle at time T, the image token, and the fourth model; the parameter update module 215 is used to perform backpropagation of the model gradient based on the predicted image at time T+1 to update the parameters of the first model, the second model, the third model, and the fourth model; the model construction module 216 is used to construct an end-to-end visual action model based on the trained first model, the second model, the third model, and the fourth model.

[0152] In one embodiment, the first model includes a feature extraction network and an image sensing head; the first calling module 211 is specifically used to: extract image features of the sample image at time T based on the feature extraction network, and convert them into bird's-eye view BEV features; fuse the BEV features at time T and historical time to obtain fused BEV features; and obtain the perception task query and image token corresponding to the image perception result at time T based on the fused BEV features and the image sensing head.

[0153] In one embodiment, the image sensing head includes at least one of a target detection head, a lane line recognition head, and an occupancy grid prediction head.

[0154] In one embodiment, the sample images at time T include images of the target vehicle from at least two perspectives.

[0155] In one embodiment, the second model includes a dynamic target encoder, a local map encoder, a static target encoder, a global navigation sensor, and a first diffusion model; the second invocation module 212 is specifically used for: obtaining at least one of a dynamic target token, a local map token, and a static target token based on the perception task Query at time T and the dynamic target encoder, the local map encoder, and the static target encoder; performing information fusion based on the dynamic target token, the local map token, and the static target token to obtain a scene fusion token; obtaining a global navigation token based on the global navigation command and the global navigation sensor; and obtaining the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the scene fusion token, the global navigation token, and the first diffusion model.

[0156] In one embodiment, the second model further includes a Transformer network; the second invocation module 212 is specifically configured to: calculate the attention weight of any token among the dynamic target token, local map token, and static target token relative to tokens of the same type based on the Transformer network, thereby obtaining a first fused dynamic target token, a first fused local map token, and a first fused static target token; and calculate the attention weight of a portion of at least one of the first fused dynamic target token, the first fused local map token, and the first fused static target token relative to tokens of different types based on the Transformer network, thereby obtaining at least one of the second fused dynamic target token, the second fused local map token, and the second fused static target token.

[0157] In one embodiment, the third calling module 213 is further configured to: obtain the predicted trajectory of the target object at time T+1 based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model; and obtain at least one of the following based on the predicted trajectory of the target vehicle at time T+1, the predicted trajectory of the target object, and the third model: a score of the probability of the target vehicle experiencing an abnormal traffic event at time T+1, a score of the target object complying with preset traffic rules, and a score of the target vehicle completing the global navigation route.

[0158] In one embodiment, when an abnormal traffic event occurs to the target vehicle at time T+1, the third calling module 213 is further used to: stop the training of the end-to-end visual motion model in this round.

[0159] In one embodiment, the fourth model includes an encoder and a second diffusion model; the fourth calling module 214 is specifically used to: obtain the state code of the target vehicle at time T based on the planned trajectory of the target vehicle at time T and the encoder; fuse the state code of the target vehicle at time T as a position code with the image token to obtain an image fusion token; and obtain the predicted image at time T+1 based on the predicted trajectory of the target vehicle at time T+1, the image fusion token, and the second diffusion model.

[0160] In one embodiment, the end-to-end visual motion module further includes a pre-training module 217 and / or a post-training module 218.

[0161] In one embodiment, before performing end-to-end model training, the pre-training module 217 is also used to perform training on the first model.

[0162] In one embodiment, before performing end-to-end model training, the pre-training module 217 is also used to perform joint training of the first model and the second model.

[0163] In one embodiment, before performing end-to-end model training, the pre-training module 217 is also used to perform joint training of the first model and the fourth model.

[0164] In one embodiment, the end-to-end visual action model further includes a reward model and a cost model; after performing end-to-end model training, the post-training module 218 is further configured to: apply safety constraints to the planned trajectory of the target vehicle output by the second model based on the cost model, and train the second model with the goal of maximizing the reward score of the planned trajectory of the target vehicle output by the reward model.

[0165] Referring to Figure 12, an embodiment of the trajectory planning device of this application is introduced. In this embodiment, the trajectory planning device includes an image acquisition module 221 and a planning module 222. The image acquisition module 221 is used to acquire environmental images of the target vehicle, and the planning module 222 calls the target model to process them to obtain the planned trajectory of the target vehicle. The target model includes the second model in the end-to-end visual action model trained by the method described in the above embodiment of this application.

[0166] Referring to Figures 1 to 10 above, a training method and a trajectory planning method for an end-to-end visual motion model according to embodiments of this specification have been described. The details mentioned in the above description of the method embodiments also apply to the training apparatus and trajectory planning apparatus for the end-to-end visual motion model according to embodiments of this specification. The training apparatus and trajectory planning apparatus for the end-to-end visual motion model described above can be implemented in hardware, software, or a combination of hardware and software.

[0167] Figure 13 illustrates a hardware structure diagram of an unmanned vehicle according to an embodiment of this specification. As shown in Figure 13, the unmanned vehicle 30 may include at least one processor 31, a memory 32 (e.g., non-volatile memory), a memory 33, and a communication interface 34, and the at least one processor 31, memory 32, memory 33, and communication interface 34 are connected together via an internal bus 35. At least one processor 31 executes at least one computer-readable instruction stored or encoded in the memory 32.

[0168] It should be understood that the computer-executable instructions stored in memory 32, when executed, cause at least one processor 31 to perform the various operations and functions described above in conjunction with Figures 1 to 10 in the various embodiments of this specification.

[0169] In the embodiments of this specification, the unmanned vehicle 30 can be configured with a functional terminal to carry the above-mentioned hardware structure. The terminal may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.

[0170] According to one embodiment, a program product, such as a machine-readable medium, is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the various operations and functions described above in conjunction with Figures 1-10 in the various embodiments of this specification. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and causing the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0171] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0172] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0173] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this specification should be defined by the appended claims.

[0174] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure. That is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or they may be jointly implemented by certain components in multiple independent devices.

[0175] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0176] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0177] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles applicable herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A training method for an end-to-end visual action model, characterized in that, The method includes: inputting a sample image at time T into a first model to obtain a perception task query and an image token corresponding to the image perception result at time T; obtaining the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the perception task query and the second model at time T; obtaining the predicted trajectory of the target vehicle at time T+1 based on the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T, and a third model; obtaining the predicted image at time T+1 based on the predicted trajectory of the target vehicle at time T+1, the planned trajectory of the target vehicle at time T, the image token, and a fourth model; performing backpropagation of model gradients based on the predicted image at time T+1 to update the parameters of the first model, the second model, the third model, and the fourth model; and constructing an end-to-end visual action model based on the trained first model, the second model, the third model, and the fourth model.

2. The training method for an end-to-end visual action model according to claim 1, characterized in that, The first model includes a feature extraction network and an image sensing head. The sample image at time T is input into the first model to obtain the perception task query and image token corresponding to the image perception result at time T. Specifically, this includes: extracting image features from the sample image at time T based on the feature extraction network and converting them into bird's-eye view BEV features; fusing the BEV features from time T and historical times to obtain fused BEV features; and obtaining the perception task query and image token corresponding to the image perception result at time T based on the fused BEV features and the image sensing head.

3. The training method for an end-to-end visual action model according to claim 2, characterized in that, The image sensing head includes at least one of a target detection head, a lane line recognition head, and an occupancy grid prediction head; and / or, the sample image at time T includes images of the target vehicle from at least two perspectives.

4. The training method for an end-to-end visual action model according to claim 1, characterized in that, The second model includes a dynamic target encoder, a local map encoder, a static target encoder, a global navigation perception machine, and a first diffusion model. Based on the perception task query at time T and the second model, the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T are obtained. Specifically, this includes: obtaining at least one of a dynamic target token, a local map token, and a static target token based on the perception task query at time T and the dynamic target encoder, the local map encoder, and the static target encoder; performing information fusion based on the dynamic target token, the local map token, and the static target token to obtain a scene fusion token; obtaining a global navigation token based on global navigation commands and the global navigation perception machine; and obtaining the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the scene fusion token, the global navigation token, and the first diffusion model.

5. The training method for an end-to-end visual action model according to claim 4, characterized in that, The second model also includes a Transformer network; it fuses the dynamic target token, local map token, and static target token to obtain a scene fusion token, specifically including: calculating the attention weight of any token among the dynamic target token, local map token, and static target token relative to tokens of the same type based on the Transformer network, thereby obtaining a first fused dynamic target token, a first fused local map token, and a first fused static target token; and calculating the attention weight of a portion of at least one of the first fused dynamic target token, the first fused local map token, and the first fused static target token relative to tokens of different types based on the Transformer network, thereby obtaining at least one of a second fused dynamic target token, a second fused local map token, and a second fused static target token.

6. The training method for an end-to-end visual action model according to claim 1, characterized in that, The method further includes: obtaining the predicted trajectory of the target object at time T+1 based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model; and obtaining at least one of the following based on the predicted trajectory of the target vehicle at time T+1, the predicted trajectory of the target object, and the third model: a score of the probability of the target vehicle experiencing an abnormal traffic event at time T+1, a score of the target object complying with preset traffic rules, and a score of the target vehicle completing the global navigation route.

7. The training method for an end-to-end visual action model according to claim 6, characterized in that, When an abnormal traffic event occurs to the target vehicle at time T+1, the method further includes: stopping the training of the end-to-end visual motion model for the current round.

8. The training method for an end-to-end visual action model according to claim 1, characterized in that, The fourth model includes an encoder and a second diffusion model. Based on the predicted trajectory of the target vehicle at time T+1, the planned trajectory of the target vehicle at time T, the image token, and the fourth model, a predicted image at time T+1 is obtained. Specifically, this includes: obtaining the state code of the target vehicle at time T based on the planned trajectory of the target vehicle at time T and the encoder; fusing the state code of the target vehicle at time T as a position code with the image token to obtain an image fusion token; and obtaining the predicted image at time T+1 based on the image fusion token and the second diffusion model, using the predicted trajectory of the target vehicle at time T+1 as a condition.

9. The training method for an end-to-end visual action model according to any one of claims 1 to 8, characterized in that, Before performing end-to-end model training, the method further includes: performing training on the first model; and / or performing joint training on the first model and the second model; and / or performing joint training on the first model and the fourth model; and / or, the end-to-end visual action model further includes a reward model and a cost model; after performing end-to-end model training, the method further includes: applying safety constraints to the planned trajectory of the target vehicle output by the second model based on the cost model, and training the second model with the objective of maximizing the reward score of the planned trajectory of the target vehicle output by the reward model.

10. A trajectory planning method, characterized in that, The method includes: acquiring an environmental image of a target vehicle and processing it using a target model to obtain a planned trajectory of the target vehicle, wherein the target model includes a second model in an end-to-end visual motion model trained by the method of any one of claims 1 to 9.

11. The trajectory planning method according to claim 10, characterized in that, The target model includes a first model in the end-to-end visual motion model trained by the method of any one of claims 1 to 9; the method includes: calling the first model to process the environmental image of the target vehicle to obtain obstacle information in the environment of the target vehicle; and performing collision detection on the planned trajectory based on the obstacle information.

12. A training device for an end-to-end visual-action model, characterized in that, include: The first calling module is used to input the sample image at time T into the first model to obtain the perception task query and image token corresponding to the image perception result at time T; the second calling module is used to obtain the predicted trajectory of the target object and the planned trajectory of the target vehicle at time T based on the perception task query at time T and the second model. The third calling module is used to obtain the predicted trajectory of the target vehicle at time T+1 based on the predicted trajectory of the target object at time T, the planned trajectory of the target vehicle, and the third model. The fourth calling module is used to obtain the predicted image at time T+1 based on the predicted trajectory of the target vehicle at time T, the planned trajectory of the target vehicle at time T, the image token, and the fourth model. The parameter update module is used to perform backpropagation of the model gradient based on the predicted image at time T+1 to update the parameters of the first model, the second model, the third model and the fourth model. The model building module is used to build an end-to-end visual action model based on the first, second, third, and fourth models that have been trained.

13. A trajectory planning device, characterized in that, include: An image acquisition module is used to acquire environmental images of the target vehicle and process them by calling a target model through a planning module to obtain the planned trajectory of the target vehicle. The target model includes a second model in the end-to-end visual action model trained by the method of any one of claims 1-9.

14. An unmanned vehicle, characterized in that, include: At least one processor; And a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the end-to-end visual motion model training method as claimed in any one of claims 1 to 9 or the trajectory planning method as claimed in any one of claims 10 to 11.

15. A machine-readable storage medium storing executable instructions, characterized in that, When the instruction is executed, it causes the machine to perform the training method for an end-to-end visual motion model as described in any one of claims 1 to 9 or the trajectory planning method as described in any one of claims 10 to 11.

Images