Cognitive-driven VLA world model automatic driving system with mixed expert and truncated diffusion

Abstract

The invention relates to the technical field of artificial intelligence and automatic driving, and discloses a mixed expert and cut-off diffusion cognitive driven VLA world model automatic driving system, which comprises a unified autoregression Transform backbone, a cognitive driven causal attention module and a mixed expert feed-forward network are included in the layer of the unified autoregression Transform backbone, the backbone is used for jointly generating a future world token and an instant action token, an anchor point token is generated, then a truncation diffusion sampling module for executing deep truncation denoising sampling is initialized through the anchor point token, and an instant action token is finally generated. Through a causal reasoning mechanism of cognitive driving, the system is upgraded from perception driving to cognitive driving, and the decision-making problem of automatic driving in a complex interaction scene is solved.

Description

Technical Field

[0001] This invention relates to a hybrid expert and truncated diffusion cognitive-driven VLA world model autonomous driving system, belonging to the field of artificial intelligence and autonomous driving technology. Background Technology

[0002] Currently, computer systems based on specific computational models, especially large-scale models using autoregressive generation, have become the mainstream technical approach for processing high-dimensional heterogeneous information flows. Through unified sequence modeling capabilities, they have demonstrated effectiveness in various complex cognitive tasks. However, when such computational models are applied to dynamic environments requiring real-time, continuous planning, their computational architecture reveals limitations. On the one hand, to achieve high-precision representation of the world state and complex logical reasoning, the scale of model parameters continues to grow, causing delays in the reasoning process due to its densely activated computational methods, which cannot meet the real-time requirements of high-frequency decision-making. On the other hand, models simplified in pursuit of real-time performance are limited in their representational capabilities, making it difficult to construct internal world models that can be used for long-term extrapolation, thus limiting their planning capabilities.

[0003] Industry attempts to address this issue have introduced new problems. For example, while multi-step iterative models can simulate the multimodal uncertainty of decision-making, their multi-round denoising process results in excessive computational overhead. Forcibly compressing the number of iterations often leads to a decrease in generation quality and a loss of diversity. Meanwhile, technical solutions attempt to decouple complex driving tasks by utilizing the reasoning capabilities of Large Language Models (LLMs), trying to manage the decision-making process through modular approaches such as perception, memory, planning, and reflection. For instance, Chinese invention patent CN119203388A discloses a method and system for autonomous driving motion planning driven by a large language model. Although this solution introduces cognitive and reflection modules, it is essentially still a highly serial modular architecture. It relies on large language models for high-level policy generation and separates the understanding of the world state from the final motion planning through experience memory retrieval. This architecture not only has bottlenecks in information transmission and fusion between modules, but also the retrieval and multi-step reasoning process is difficult to meet the high-frequency real-time decision-making requirements of autonomous driving. This makes it impossible for the predictions of the world model to be fully utilized by the action model, and the two fail to form a deep coupling at the computational representation level.

[0004] Therefore, the technical problem to be solved by this invention is how to provide a computational model architecture that can balance model capacity and computational efficiency, decision diversity and real-time generation, and the coupling of world prediction and action generation. Summary of the Invention

[0005] This invention provides a cognitively driven VLA world model autonomous driving system that combines experts and truncated diffusion. Its main purpose is to solve the problem that it is difficult to balance model capacity and computational efficiency, decision diversity and real-time generation, and world prediction and action generation in the existing computational model architecture.

[0006] To achieve the above objectives, this invention provides a hybrid expert and truncated diffusion cognitive-driven VLA world model autonomous driving system. The system includes a unified token representation space, a world state encoder, an autoregressive Transformer backbone, and a truncated diffusion sampling module. A unified token representation space is used to map visual observations, language commands, action sequences, and the world state of a 3D Gaussian field consisting of a set of 3D Gaussian units to a common vocabulary. World State Encoder is used to compress 3D Gaussian field world state into future world tokens; An autoregressive Transformer backbone receives the input sequence and autoregressively generates an output sequence containing a future world token and an immediate action token. The Transformer layer of the autoregressive Transformer backbone includes: a cognitively driven causal attention module, which embeds a causal graph prior for the driving domain generated by encoding a causal model of the driving domain structure through a graph neural network; and a hybrid expert feedforward network, which includes multiple expert networks and a routing network. The routing network dynamically selects the top-4 expert networks for sparse activation. When generating the immediate action token, the autoregressive Transformer backbone generates an anchor token. The truncated diffusion sampling module initializes the Gaussian distribution by the distribution parameters mapped to the anchor token, and performs a fixed 10-step denoising sampling process starting from the initialized Gaussian distribution to generate an instant action token.

[0007] Preferably, the future world token is a sequence of 128 discrete tokens used to represent each future time step through Gaussian VQ-VAE compression, and the instant action token is a sequence of 8 discrete tokens representing macroscopic intentions, mesoscopic trajectories and microscopic adjustments respectively through three-level hierarchical quantization encoding.

[0008] Preferably, the multiple expert networks of the hybrid expert feedforward network are divided into four functionally specialized expert groups. The four functionally specialized expert groups include: a group of perception experts for processing visual representations of different driving environments; a group of reasoning experts for performing causal inference and planning; a group of modeling experts for predicting the geometric and semantic evolution of the world state; and a group of decision experts for generating specific driving maneuvers.

[0009] Preferably, the system also supports do-calculus intervention operations; the intervention operations simulate forced intervention and support counterfactual reasoning by modifying the attention mask of the cognitively driven causal attention module during inference to mask the causal path pointing to the intervention variable in the causal graph prior in the driving domain.

[0010] Preferably, the system also employs a load-balanced auxiliary loss during training. To ensure that multiple expert networks are used in a balanced manner; load balancing assists in loss. The calculation method is as follows: Where N is the total number of expert networks, The actual frequency at which expert j is selected by the routing network in the batch. Predict the probability of selecting expert j for the routing network.

[0011] Preferred 3D Gaussian elements Composition, in which For location, Let covariance matrix be the variance matrix. For color, For opacity, For motion speed; the system also includes a Gaussian VQ-VAE decoder to decode and reconstruct the future world tokens generated by the autoregressive Transformer backbone into 3D Gaussian field world states.

[0012] Preferably, the 3D Gaussian elements are divided into static Gaussian elements and dynamic Gaussian elements. The static Gaussian elements undergo rigidity transformation with the movement of the vehicle, while the dynamic Gaussian elements are based on the motion speed predicted by the system. Perform a location update.

[0013] Preferably, during inference, the system generates multiple candidate action-world joint sequences by changing the random seed of the truncated diffusion sampling module; in the multiple imagined worlds corresponding to the multiple candidate action-world joint sequences reconstructed from the future world token, the system expands each candidate sequence in multiple steps, and selects the optimal sequence to execute based on a comprehensive scoring function that includes security and efficiency indicators.

[0014] Preferably, the system adopts a four-stage progressive training paradigm, which includes: Stage 1, self-supervised world modeling, used to learn the world state encoder and world state prediction; Stage 2, supervised action alignment, used to align the actions generated by the system with expert behavior; Stage 3, thought chain distillation, used to train the system to generate interpretable reasoning chain tokens; and Stage 4, reinforcement learning optimization, used to optimize the system's decision success rate in a simulated environment.

[0015] Preferably, the autoregressive Transformer backbone jointly generates future world tokens and immediate action tokens in the same autoregressive generation process, so as to achieve deep coupling between action generation and world state evolution.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention constructs a unified discrete token representation space, mapping heterogeneous information such as visual occupancy, world state, and action sequences to a common computational vocabulary. This unification at the representation level enables a single autoregressive Transformer architecture to jointly model world evolution and action generation, avoiding the information loss and error accumulation inherent in traditional modular computing systems when transmitting information between different representation spaces.

[0017] 2. This invention integrates a hybrid expert architecture with a causal attention mechanism. Specialized reasoning and decision-making expert groups, guided by causal graph priors, enable the computational model to go beyond surface data relationships when processing information and instead use embedded causal structures for inference. This approach ensures that the system's output decision is based on a prediction of the logic of things rather than a simple imitation of historical data, thereby improving the computational effectiveness of the model in unseen or complex interactive situations.

[0018] 3. By using the anchored Gaussian truncation diffusion method, the inherent computational delay problem in generating multimodal uncertain outputs of the computational model is solved. The model quickly generates a high-probability anchor distribution through autoregression and performs a limited number of denoising sampling steps only at this time-series cutoff point. This mechanism combines the fast inference capability of the autoregressive model with the fine sampling capability of the diffusion model. Without sacrificing the diversity of outputs, the computation time of the generation process is compressed to a controllable low level, making it suitable for computational tasks with high real-time requirements. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the collaborative architecture of autoregressive backbone and truncated diffusion sampling in this invention; Figure 2 This is a schematic diagram of the comprehensive scoring of the multi-hypothesis prospective evaluation of the present invention; Figure 3This is a schematic diagram of the training process and vehicle-mounted deployment architecture of the model of this invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. However, it should be understood that the detailed description and specific embodiments of the present invention are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0021] The hybrid expert and truncated diffusion cognitive-driven VLA world model autonomous driving system disclosed in this invention is an end-to-end system architecture based on a specific computational model. In its engineering implementation, the system mainly includes a unified token representation space, a world state encoder, an autoregressive Transformer backbone, and a truncated diffusion sampling module. In the system's data flow, external multimodal inputs are mapped to the unified token representation space via their respective encoders. The autoregressive Transformer backbone uses these tokens as context to autoregressively and jointly generate future world tokens representing future world evolution and immediate actions representing current decisions. The final generation of the instantaneous action token is achieved by first outputting an anchor token from the backbone, then using this anchor token to initialize the truncated diffusion sampling module, and finally performing fast refinement sampling. In the specific computational task of autonomous driving, traditional modular systems suffer from loss and error accumulation when transmitting information between modules due to the heterogeneous intermediate representations used in perception, prediction, and planning. To address this, this system constructs a unified token representation space to map all heterogeneous information flows to a common computational vocabulary V, enabling a single computational model to process them uniformly. Specifically, for visual observation, the system acquires a 6-view RGB image sequence. Using a 4D spatiotemporal occupancy encoder, it is converted into an occupancy grid that represents the environmental occupancy in space and time (taking T=16 time steps, X=Y=200 meters spatial range, Z=16 height layers as an example). The continuous occupancy grid is then mapped to the codebook through a vector quantization (VQ) process, compressed into discrete visual representations. For language instructions, they are encoded into a sequence of language tokens using a standard word segmentation tool. For vehicle states, a vector containing 16 continuous values ​​such as speed and orientation is encoded and quantized into four discrete states. For the action sequence, this system compresses its future trajectory into 8 discrete instantaneous actions using a three-level hierarchical quantization encoder. The encoder is designed to divide the eight tokens into three levels, corresponding to macroscopic intent, mesoscopic trajectory, and microscopic adjustment. For the representation of the world state, this system employs a set of 3D Gaussian primitives. A 3D Gaussian field is constructed, and a specialized world-state encoder, namely Gaussian VQ-VAE, is used to process this Gaussian field. Compressed into a discrete future world The future world token is a sequence of 128 discrete tokens used to represent each future time step, thus all modalities are converted into a token sequence on the common vocabulary V.

[0022] After achieving representational unification, the core computation of the system is performed by an autoregressive Transformer backbone. This backbone addresses the problem of separation between world prediction and action planning in existing models, failing to form a cognitive loop. The backbone receives the input sequence... Visuals that include history ,language Current status And prompts for controlling the generation mode Based on the input sequence, the backbone jointly and autoregressively generates a unified output sequence. The output sequence also contains an interpretable inference chain. Current pending actions And the future world H time steps ahead. This computational method, which jointly generates actions and world states within the same sequence, enables the generation of actions... Able to evolve the future world This is a direct condition, and vice versa, thus achieving deep coupling and closed-loop optimization of the two at the representation level of the computational model. Furthermore, the internal structure of this autoregressive Transformer backbone is designed to simultaneously address the dual challenges of computational efficiency and cognitive depth in autonomous driving tasks. Firstly, to address the inference efficiency bottleneck caused by the expansion of large model parameters, the Transformer layer of this backbone employs a hybrid expert feedforward network (MoE-FFN), which contains multiple expert networks—32 independent in a specific implementation. And a routing network; during computation, the routing network calculates the routing score based on the hidden state h of the current token. The top-K highest-scoring expert networks in this scheme (K=4) are dynamically selected for sparse activation, and their outputs are... By relying solely on the weighted contributions of these four experts, the system achieves improved computational efficiency by requiring only 8.75 bytes of computation per inference while maintaining a total parameter capacity of 70 bytes. Secondly, to ensure balanced expert usage and prevent overloading of a few experts, a load balancing auxiliary loss is introduced during training. Where N is the total number of experts, e.g., N=32. It is the actual frequency with which expert j is selected in the batch. The routing network predicts the probability of selecting that expert, and this loss function is used to penalize... and The imbalance among them prompts the routing network to distribute the computational load evenly; thirdly, in order to perform functional specialization, these N=32 expert networks are divided into 4 functional specialization expert groups, namely a group of perception experts (E1-E8) for processing visual representations of different driving environments, a group of reasoning experts (E9-E16) for performing causal inference and planning, a group of modeling experts (E17-E24) for predicting the geometric and semantic evolution of the world state, and a group of decision experts (E25-E32) for generating specific driving maneuvers.

[0023] Meanwhile, to enable the computational model to acquire cognitive abilities beyond the surface-level associations of data, the Transformer layer of the autoregressive Transformer backbone also includes a cognitively driven causal attention module. Standard attention mechanisms can only capture correlations, but are easily misled by spurious correlations in complex interactive scenarios. Therefore, this system constructs a structural causal model (SCM) for the driving domain. This model defines the causal relationships between variables such as the driver's actions, other vehicles' behaviors, and road conditions, and encodes them into a causal graph prior matrix for the driving domain through a graph neural network (GNN). During attention calculation, this prior matrix C is embedded into the standard attention formula, i.e. Through parameters The prior matrix C enhances the attention weights between token pairs with genuine causal relationships while suppressing spurious correlations. Furthermore, the system supports do-calculus interventions for counterfactual reasoning. During inference, when an intervention needs to be queried... At that time, the system modifies the attention mask. To achieve graph surgery, all causal paths pointing to the intervened variable X are masked in the causal graph prior, thereby simulating the effect of forced intervention within the computational model. This allows the system to evaluate the possible consequences of different actions taken at the time. For autonomous driving decision-making, simple deterministic action outputs cannot model the multimodal characteristics of real-world driving decisions, such as the choice between turning left or waiting at an intersection. Furthermore, the multi-step iterations of the standard diffusion model (50-1000 steps) cannot meet the real-time requirements above 10Hz. To address this computational latency bottleneck, this system employs a collaborative approach between a truncated diffusion sampling module and an autoregressive backbone. The execution flow is as follows: In the first stage, when an immediate action token needs to be generated, the autoregressive Transformer backbone quickly generates the anchor point with the highest probability through its efficient autoregressive inference. In the second stage, the anchor token is mapped to a set of Gaussian distribution parameters through codebook lookup. These parameters determine the initial Gaussian distribution for the truncated diffusion sampling module; in the third stage, the sampling process does not begin with pure noise, but rather with this anchored distribution. Initially, the number of denoising iterations is truncated to a fixed, small value, i.e. Step; the system performs these 10 steps of denoising sampling. In a specific implementation, DDIM sampling is used. Each step of the denoising calculation is conditioned on the hidden state h of the Transformer backbone, and is performed through a lightweight denoising network. Predict noise and update it according to the formula. and After 10 iterations, a refined real-time action token is generated. This mechanism combines the fast localization capability of autoregression with the fine sampling capability of diffusion model, compressing the inference latency to within real-time requirements (e.g., 50ms) while maintaining the multimodal generation capability.

[0024] The computational model of this system also requires a high-fidelity, real-time interactive representation of the internal world to support planning; therefore, the system uses a 3D Gaussian sputtering field to represent the world state and generates its token sequence through an autoregressive Transformer backbone. When it is necessary to explicitly represent the future world, the system also includes a Gaussian VQ-VAE decoder, which decodes and reconstructs the future world token generated by the backbone into a 3D Gaussian field world state. The basic elements constituting this Gaussian field Includes location Covariance matrix ,color Opacity And the key speed of movement To achieve dynamic evolution, these Gaussian units are divided into two categories: static and dynamic; static Gaussian units... For example, the location of roads and buildings is updated based on vehicle movement. Perform rigid transformation ; while dynamic Gaussian elements Other vehicles, however, rely on the system's prediction of their own speed. Perform location update The 3D Gaussian field supports differentiable high-speed rendering of over 150 FPS, enabling the system to perform low-cost trial and error in an imagined future world. To ensure the effective activation of the functionally specialized expert group in the hybrid expert feedforward network and the stable establishment of the mapping relationship between the anchor tokens generated by the autoregressive Transformer backbone and the Gaussian distribution initialized by the truncated diffusion sampling module, in the initial stage of the four-stage progressive training, i.e., when the self-supervised world modeling in stage one and the supervised action alignment in stage two, the system, in addition to using the load balancing auxiliary loss, employs the aforementioned load balancing auxiliary loss. In addition, two auxiliary training procedures are introduced: one is task-guided routing loss. In the early stages of training, it forcibly directs the routing score g of input tokens for specific task types, such as world modeling task tokens, to a pre-defined expert group, such as a modeling expert group. The weight coefficients gradually decay with the number of training steps until they are removed, and subsequent operations rely on the routing network based on... The first is to perform autonomous dynamic routing with the main task loss; the second is to use an anchored distribution alignment procedure, which utilizes an independent action encoder to process expert actions. Encoding as target distribution parameters And introduce anchoring alignment loss To minimize the anchor points generated by the backbone The mapped initial distribution parameters With that The KL divergence between them, and the diffusion denoising loss. Simultaneous optimization of denoising network This enables it to obtain from that Starting from the distribution, reconstruction is performed within a fixed 10-step denoising sampling process. .

[0025] To utilize this differentiable high-fidelity world model, this system employs a multi-hypothesis look-ahead evaluation mechanism during inference. First, the system generates multiple candidate action-world joint sequences, such as K=16, by changing the random seed of the truncated diffusion sampling module. Second, the system evaluates the future world of each candidate sequence... In the corresponding imagined world that has been decoded and reconstructed, each candidate sequence undergoes multiple steps, such as H=5 steps of rollout; finally, the system, based on a security-related... ,efficiency Comfort and adherence to rules Comprehensive scoring function for indicators such as The K=16 complete trajectories derived through H steps are evaluated, and the trajectory with the highest score is selected. This mechanism enables the system to evaluate a total of K×H (e.g., 16×5=80) future possibilities before executing physical actions, improving decision-making robustness in complex and edge scenarios. To ensure effective training of this complex system, a four-stage progressive training paradigm is adopted. Stage 1, self-supervised world modeling, utilizes a large number of unlabeled driving videos to train the world state encoder Gaussian VQ-VAE and its world state prediction capabilities through self-supervised tasks such as future frame generation and mask occupancy prediction. Stage 2, supervised action alignment, uses labeled expert driving clips and methods such as behavior cloning (BC) to align the system's generated instantaneous actions. Aligning expert behavior; Phase 3, thought chain distillation, using manually labeled reasoning process data, through sequence-to-sequence supervised learning, trains the system to generate interpretable reasoning chain tokens. Phase four involves reinforcement learning optimization. In high-fidelity simulation environments such as CARLA, algorithms such as Proximal Policy Optimization (PPO) are used, with security and traffic efficiency as reward signals, to optimize the system's decision success rate in complex interactive scenarios end-to-end.

[0026] To achieve real-time partitioning of 3D Gaussian Gaussian bits into static and dynamic Gaussian bits, the World State Encoder (Gaussian VQ-VAE) incorporates the following into its coding structure, except for the prediction position... Covariance In addition to parameters, it is also trained to output a dynamic attribute scalar for each Gaussian element i. Its value range is between 0 and 1; The training supervision signal comes from the self-supervised world modeling in stage one. Specifically, the system calculates the actual observed displacement of each primitive i between consecutive time steps t and t+1. And simultaneously calculate the vehicle's motion Resulting in rigid transformation displacement When the residual between the two Greater than a preset motion threshold When the unit is in meters per frame, the target of the primitive i at time t. It is set to 1, and vice versa; during inference, the system relies on the encoder output. With a fixed classification threshold >0.5 divides all primitives, and the dynamic Gaussian primitive Gdyn(j) is based on the prediction speed. Position updates are performed, while the static Gaussian unit Gstat performs rigid transformations; during training, the system employs a joint multi-task objective function. ,in The aforementioned MoE load balancing loss; the other five loss targets include: That is, used for supervising the inference chain Autoregressive language modeling loss; That is, used for aligning expert actions Action prediction loss (such as L2 loss and KL divergence); This refers to the world state reconstruction loss, which in turn includes rendering loss. (Compare predicted rendered images) With real images (differences) and token reconstruction loss (Supervising the World) (Generation) That is, geometric consistency loss, which is calculated using the Chamfer distance. To measure and predict Gaussian fields With real-world scenarios The differences in their geometric positions; and That is, diffusion denoising loss, which takes the form of Used to train the denoising network in the truncated diffusion sampling module. .

[0027] Example 1: At an unprotected left-turn intersection during peak urban traffic hours, the autonomous driving system of this invention faces a typical complex decision-making scenario. The main lane has a green light, but there is dense oncoming traffic, and pedestrians on the right-hand crosswalk are obscured by A-pillars or vehicles in front. This scenario presents a double challenge to the computational model: firstly, decisions must be made within an extremely short time, such as 100ms, to cope with oncoming high-speed traffic, which imposes strict constraints on computational latency; secondly, the decision space is highly multimodal, requiring the system to simultaneously evaluate multiple possibilities, including waiting in place, slowly creeping to obtain a better view, and quickly passing through gaps in oncoming traffic, as well as the complex interactive consequences they generate. After receiving this high-dimensional heterogeneous input, the system encodes the 6-view image sequence, vehicle dynamics, and intersection topology provided by the high-precision map into a unified token sequence through its unified token representation space. This sequence is input into the autoregressive Transformer backbone. Faced with this complex scenario, the routing network of the hybrid expert feedforward network (MoE-FFN) within the backbone is activated, dynamically allocating the computational load to the routing network. The expert group highly relevant to the task—specifically, experts in perception (for handling urban congestion), experts in reasoning (for causal deduction and risk assessment), and experts in modeling (for predicting dynamic interactions)—are activated with high weights (Top-4, for example). Other irrelevant experts, such as decision-makers for high-speed cruising, are sparsely activated. This concentrates the computational load of the current inference step on the key task without sacrificing the overall model capacity. Inside the backbone Transformer layer, the cognitive-driven causal attention module begins operation. This module utilizes its embedded causal graph priors from the driving domain to focus not only on the correlation of the data surface when calculating attention but also on strengthening the weights of key causal chains. The system pays close attention to the causal relationship between the speed of oncoming vehicles and whether they tend to decelerate when approaching an intersection, as well as the direction of pedestrian heads, such as whether pedestrians are looking at oncoming vehicles, and the causal relationship between this and their future trajectories. This allows the computational model to predict that oncoming vehicle A has no intention of decelerating and will force its way through, and that pedestrian B's attention is distracted and there is a risk of entering the lane.

[0028] Based on this cognitive judgment, the system enters the multi-hypothesis prospective evaluation stage. Instead of calculating a single optimal solution, the system generates K=16 candidate action-world joint sequences by changing the random seed of the truncated diffusion sampling module. These sequences semantically correspond to different decision branches: candidate A corresponds to immediately waiting in place, candidate B corresponds to slowly crawling 1 meter and then waiting, and candidate C corresponds to starting and turning left in the gap between vehicle A passing and vehicle D arriving. At this point, the system's core collaborative mechanism is triggered. The autoregressive Transformer backbone utilizes its efficient hybrid expert (MoE-FFN) architecture and causal attention module to quickly autoregressively generate H=5 future timestep future world tokens for each of these K=16 candidate branches. These token sequences are then decoded in parallel by the Gaussian VQ-VAE decoder into K=16 independent 3D Gaussian field imagined worlds. In the imagined world of candidate C, the system predicts the speed of vehicle D. The situation will change due to the intervention of the vehicle (candidate C), such as an emergency deceleration. In candidate B's imagined world, pedestrian B's... This indicates that it will enter the conflict zone of the vehicle (candidate B); the generation of this multimodal action-world sequence solves the real-time bottleneck of the traditional diffusion model; when generating each instantaneous action token, the system first uses the autoregressive Transformer backbone to quickly generate an anchor token, a high-probability neighborhood representing waiting, and the truncated diffusion sampling module only samples from the distribution mapped by this anchor token. Initially, a fixed 10-step denoising sampling process is performed to refine high-quality and diverse specific actions, such as waiting and keeping the steering wheel straight. This reduces the generation time of a single decision to less than 50ms, thus resolving the contradiction between decision diversity and real-time computation. After obtaining K=16 future trajectories that have been extrapolated through H=5 steps, the system, based on preset safety parameters... With efficiency The comprehensive scoring function is used for evaluation; candidate C, the forced insertion scheme, receives a very low score because it causes the collision time TTC of vehicle D to be below the safety threshold in the imagined world. The scoring process shows that candidate B, the creeping solution, receives a lower score due to the potential conflict with pedestrian B. Ultimately, candidate A, the solution of waiting in place for both vehicle A and pedestrian B to pass, receives the highest overall score. The system ultimately selects to execute the optimal sequence of candidate A, outputting a stable waiting action token, and the vehicle remains stationary.

[0029] Example 2: To objectively quantify the performance of the system of the present invention in terms of computational efficiency and decision robustness, especially in urban scenarios with high dynamics and high interaction complexity, this example constructs a standardized test environment based on the CARLA simulation platform. This platform was selected due to its high-fidelity physics engine and sensor simulation capabilities. The test scenario is set as an unprotected left-turn intersection in the Town05_Opt map, which reproduces the high-risk conditions of Example 1, including dense oncoming straight-through traffic and potential pedestrian obstruction. To evaluate the system's performance under different pressures, two traffic flow densities were set in the experiment. The test involved two traffic flow configurations: a medium-density oncoming traffic flow of 20 vehicles / minute and a high-density oncoming traffic flow of 45 vehicles / minute, accompanied by intermittent pedestrian crossings. Each configuration underwent 1000 independent closed-loop simulation tests to obtain statistical data. The core of this experiment was to compare three different computational model configurations to isolate and verify the role of the key technical features of the invention. The first group was the sample group of the invention, which fully deployed all modules as described in the specific implementation, including a hybrid expert feedforward network MoE-FFN with a total of 70B parameters, Top-4 sparse activation, a cognitively driven causal attention module, and an anchored Gaussian truncation diffusion sampling module. The first group was the control group A, which was a dense baseline model. This model adopted a dense Transformer architecture, and its total number of parameters (approximately 8.75B) was similar to the number of parameters activated in a single inference of the sample group of this invention. However, this control group did not have the MoE architecture and causal attention module. The third group was the control group B, which was a standard diffusion model. Except for replacing its truncated diffusion sampling module with a standard diffusion sampler that requires 100 iterations (T=100), the architecture of this model was consistent with that of the sample group of this invention. During the experiment, the system recorded the average single-frame inference latency (ms), average number of activated parameters (B), collision rate (%), and task success rate (%) for each simulation instance. Task success was defined as completing a left turn safely and without collision within 60 seconds. The experimental data are summarized in Table 1.

[0030] Table 1: Performance Comparison Test Table under Different Computation Model Configurations

[0031] Referring to Table 1, the experimental data present a clear performance difference; in terms of computational efficiency, the average inference latency of the present invention's sample group (approximately 48.9 ms) is on the same order of magnitude as that of control group A (approximately 45.5 ms), and both meet the real-time decision-making requirements of 10Hz (100 ms) for in-vehicle applications. However, control group B, due to its use of 100-step standard diffusion, has a latency of approximately 474.4 ms, far exceeding the real-time threshold, confirming the effectiveness of anchored Gaussian truncation diffusion (…). The invention demonstrates the role of maintaining extremely low latency. Secondly, in terms of decision quality, although the inference latency of the present invention sample and the control group A are similar, their task success rates in high-density scenarios differ by 90.3% and 58.4%, respectively. Furthermore, the collision rate of the present invention sample (1.9%) is significantly lower than that of the control group A (15.6%). This set of data shows that the present invention sample, through the sparse activation mechanism of MoE-FFN, effectively utilizes the world modeling and causal reasoning capabilities carried by its 70B total parameter capacity, further enhanced by the causal attention module, while the control group A, with the same amount of computation, is unable to handle this complex interaction scenario due to the limitation of model capacity.

[0032] Example 3: This example includes a comparative example 1. Comparative example 1 uses the same experimental platform, high-density test scenario, and computational model with a hybrid expert feedforward network MoE-FFN containing 70B total parameters, Top-4 sparse activation, and 10-step truncated diffusion as the sample group in Example 2. The only difference is that comparative example 1 removes the cognitively driven causal attention module and replaces it with the Transformer self-attention mechanism used in sequence modeling in this field. The computational model of comparative example 1 is placed in the same 1000 closed-loop simulation tests as in the high-density scenario of Example 2. In multiple tests, it was observed that when facing the unprotected left turn condition of Example 1, the model's self-attention mechanism only calculates based on the correlation at the data surface level. In a typical failure case, the model simultaneously observes two signals: an oncoming high-speed vehicle (vehicle A) and a pedestrian obstructed from the side (pedestrian B). Due to the lack of prior guidance from the causal graph, the model assigns high weight to the spurious correlation between the pedestrian's presence and the oncoming vehicle's deceleration, failing to fully utilize the speed of vehicle A itself. The observation data was not attenuated; based on this calculation, the model output an instantaneous action token for starting and turning left at the moment when the sample group of this invention selected to wait, which directly led to a side collision with vehicle A; after repeating the test 1000 times, its key performance indicators are shown in Table 2.

[0033] Table 2: Comparative Test Table of Cognitive-Driven Causal Attention Module Stripping

[0034] As shown in Table 2, the data indicates that, while keeping the number of computational parameters and inference latency almost unchanged (49.0ms vs 49.1ms), removing the cognitively driven causal attention module caused the system collision rate to increase from 1.9% to 12.8%, while the task success rate decreased from 90.3% to 65.2%. This result shows that the standard self-attention mechanism has limitations in handling such complex interaction scenarios, while the cognitively driven causal attention module is effective in solving such interactive decision-making problems.

[0035] Example 4: This example combines Figures 1 to 3 An explanation of autonomous driving systems using a cognitively driven VLA world model with hybrid experts and truncated diffusion, such as... Figure 1 As shown, the system receives multimodal inputs, including visual observations, language commands, and action sequences. These inputs are mapped to a common vocabulary via a unified token representation space. Simultaneously, the 3D Gaussian field world state is compressed through a world state encoder. The outputs of both are fed into an autoregressive Transformer backbone. The backbone contains a cognitively driven causal attention module for embedding causal graph priors and a hybrid expert feedforward network for sparse activation. When the backbone performs joint autoregressive generation of output sequences, it jointly generates future world tokens to predict future world state evolution and generates an anchor token. This anchor token initializes a truncated diffusion sampling module, which performs a fixed 10-step denoising sampling process to finally generate an instantaneous action token for real-time driving decisions.

[0036] like Figure 2 As shown, the horizontal axis of the graph lists 16 options, from candidate 1 to candidate 16, and the vertical axis represents the evaluation value, ranging from 0 to 100. The graph specifically presents three scoring curves: the solid line represents the safety score. The curve fluctuates significantly, peaking at candidates 4, 8, and 11, while dropping to a distinct trough at candidate 6. The efficiency score is represented by the dashed line. The curve also exhibits strong volatility, and its trend often contradicts the safety score; when the safety score of candidate 4 reaches its peak, the efficiency score... It is at a low point, while the efficiency score of candidate 13 is low. When it is high, the safety score is The score is relatively low. In stark contrast to the previous two curves, the comprehensive score, represented by a dotted line, shows high stability, remaining at a high level of around 80 points across all 16 candidates. This demonstrates the system's trade-off and balance between safety and efficiency. Ultimately, the system selects the optimal sequence based on this relatively stable and high comprehensive score.

[0037] like Figure 3As shown, the upper training flow demonstrates how the model is trained and deployed through Phase 1: Self-supervised world modeling, Phase 2: Supervised action alignment, Phase 3: Mind chain distillation, and Phase 4: Reinforcement learning optimization of the PPO algorithm in the CARLA simulator. The lower end-to-end system flow of the in-vehicle computing platform shows how, during actual operation, inputs such as visual observation of 6-view image sequences, language commands, and vehicle status are fed into the autoregressive Transformer backbone through a unified token representation space. This backbone utilizes its internal hybrid expert feedforward network and cognitively driven causal attention module to generate future world tokens through internal forward planning and to generate immediate action tokens through the anchor token initialization truncated diffusion sampling module, which are then ultimately executed by the vehicle actuators.

[0038] Example 5: The effectiveness of the cognitively driven causal attention module in the computational model of this invention depends on the embedded causal model of the driving domain. To ensure the accuracy and completeness of the model and to clarify the model construction process, this embodiment discloses a method for building, verifying, and maintaining the model. The reproducible procedure is executed offline before system deployment. Its first step is the definition of nodes and edges. The system's nodes are identified as key dynamic and static factors in the driving scenario, including 128 variables such as vehicle actions, other vehicle behaviors, pedestrian states, traffic lights, and road conditions like slipperiness and slope. The system's edges represent causal relationships, established through a two-step process: first, deterministic causal relationships are predefined based on physical laws such as vehicle dynamics and optics; second, a large-scale real-world driving log and accident database (up to 10 million kilometers) is used to mine causal chains hidden in statistical data through causal discovery algorithms, such as PC or FCI algorithms. These are then finalized by domain engineers, thus constructing an initial causal framework containing thousands of causal relationships. ;Should The prior is encoded into a graph neural network (GNN). To verify the validity of this prior, the system performs counterfactual verification in the CARLA simulator. Taking a standard following scenario as an example, the system performs an intervention query. The predicted collision probability (75% in this case) is compared with the actual ground collision frequency (72% in this case) caused by the delayed braking action in the simulator. The model's counterfactual prediction error is only considered valid if it falls below a preset threshold (5% in this case). Only then is it considered valid; at the same time, to ensure the long-term adaptability of the model, this... In a dynamic maintenance cycle, the system continuously extracts new interaction data from difficult scenarios or takeover events collected by the fleet, and periodically, which can be set to quarterly, re-verifies and incrementally updates the SCM.

[0039] Furthermore, the joint training objective function of the system of the present invention The convergence and final performance depend on six key hyperparameters, namely... to and The calibration of these weights is used to balance the relationship between the five main tasks and MoE load balancing. To clarify this calibration process, this embodiment adopts a phased parameter calibration procedure coupled with a four-stage progressive training paradigm. The first step of the procedure, in the first stage of training, namely the self-supervised world modeling stage, is to calibrate the weights related to actions, reasoning, and diffusion. The value is temporarily set to 0. At this point, the system focuses on optimizing the world model itself, using a grid search method to find the optimal value for world state reconstruction within an interval such as [0.1, 1.0]. With used for geometric consistency The optimization objective is to maximize the occupancy prediction IoU on the validation set while ensuring that the rendering quality PSNR is not lower than 32dB. This step yields a preliminary set of weights. For example; in the second step of the procedure, during the second phase of training, namely the supervised movement alignment phase, the calibrated... , And introduces methods for action prediction. With the use of diffusion denoising The optimization objective at this point is to minimize the trajectory error while maintaining the performance of the world model (i.e., PSNR no lower than 32dB) and preventing degradation. Through trade-offs, a set of weights is determined to... For example; in the third step of the procedure, during the third stage of training, namely the thought chain distillation stage, the preceding weights are fixed, and a set of weights for language modeling are introduced. This weight The calibration criteria are to ensure that the reasoning chain achieves a target value of human comprehensibility, which is 85% in this case, and to determine the minimum weight value to avoid interference with action decision-making. The fourth step of the procedure involves load balancing weights throughout all stages. It is always activated, and its calibration principle is to continuously monitor the routing percentage of all experts during training, and set a target that the coefficient of variation of expert load should be less than 0.2. The value, such as The calibration process is dynamically adjusted to meet this constraint, thus avoiding overload or underutilization of a few experts; this four-step calibration procedure can provide... Determine a set of reproducible weighting coefficients with engineering basis, such as... This provides a basis for the stability of the computational model training process and its final performance.

[0040] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A hybrid expert and truncated diffusion cognitive-driven VLA world model autonomous driving system, characterized in that, The system includes a unified token representation space, a world-state encoder, an autoregressive Transformer backbone, and a truncated diffusion sampling module. A unified token representation space is used to map visual observations, language commands, action sequences, and the world state of a 3D Gaussian field consisting of a set of 3D Gaussian primitives to a common vocabulary. World State Encoder is used to compress the 3D Gaussian field world state into a future world token; An autoregressive Transformer backbone is used to receive the input sequence and jointly autoregressively generate an output sequence containing the future world token and the immediate action token. The Transformer layer of the autoregressive Transformer backbone contains: a cognitively driven causal attention module, which embeds a causal graph prior of the driving domain generated by encoding a causal model of the driving domain structure through a graph neural network. It also includes a hybrid expert feedforward network, which contains multiple expert networks and a routing network. The routing network is used to dynamically select the top-4 expert networks from the multiple expert networks for sparse activation. The autoregressive Transformer backbone generates an anchor token when generating an instant action token. The truncated diffusion sampling module initializes the Gaussian distribution by the distribution parameters mapped to the anchor token, and performs a fixed 10-step denoising sampling process starting from the initialized Gaussian distribution to generate an instant action token.

2. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The future world token is a sequence of 128 discrete tokens used to represent each future time step through Gaussian VQ-VAE compression. The instant action token is a sequence of 8 discrete tokens used to represent macroscopic intentions, mesoscopic trajectories and microscopic adjustments through three-level hierarchical quantization encoding.

3. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The multiple expert networks of the hybrid expert feedforward network are divided into four functionally specialized expert groups. These four groups include: a group of perception experts for handling visual representations of different driving environments; a group of reasoning experts for performing causal inference and planning; a group of modeling experts for predicting the geometric and semantic evolution of the world state; and a group of decision experts for generating specific driving maneuvers.

4. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The system also supports do-calculus intervention operations; these operations simulate forced intervention and support counterfactual reasoning by modifying the attention mask of the cognitively driven causal attention module during inference to mask causal paths pointing to intervention variables in the causal graph prior in the driving domain.

5. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The system also employs a load balancer to assist loss during training. To ensure that multiple expert networks are used in a balanced manner; load balancing assists in loss. The calculation method is as follows: Where N is the total number of expert networks. The actual frequency at which expert j is selected by the routing network in the batch. Predict the probability of selecting expert j for the routing network.

6. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, 3D Gaussian Composition, in which For location, Let covariance matrix be the variance matrix. For color, For opacity, For motion speed; the system also includes a Gaussian VQ-VAE decoder for decoding and reconstructing the future world tokens generated by the autoregressive Transformer backbone into 3D Gaussian field world states.

7. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 6, characterized in that, 3D Gaussian elements are divided into static Gaussian elements and dynamic Gaussian elements. Static Gaussian elements undergo rigidity transformations as the vehicle moves, while dynamic Gaussian elements are based on the motion velocity predicted by the system. Perform a location update.

8. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, During inference, the system generates multiple candidate action-world joint sequences by changing the random seed of the truncated diffusion sampling module. In the multiple imagined worlds corresponding to the multiple candidate action-world joint sequences reconstructed from the future world token, the system expands each candidate sequence in multiple steps and selects the optimal sequence to execute based on a comprehensive scoring function that includes security and efficiency indicators.

9. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The system adopts a four-stage progressive training paradigm, which includes: Stage 1, self-supervised world modeling, used to learn the world state encoder and world state prediction; Stage 2, supervised action alignment, used to align the actions generated by the system with expert behavior; Stage 3, thought chain distillation, used to train the system to generate interpretable reasoning chain tokens; and Stage 4, reinforcement learning optimization, used to optimize the system's decision success rate in a simulated environment.

10. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The autoregressive Transformer backbone jointly generates future world tokens and instant action tokens within the same autoregressive generation process.

Images