Cognitive driven vla world model autonomous driving system mixing expert and truncated diffusion

Abstract

The application relates to the field of artificial intelligence and automatic driving technology and discloses a cognitive driving VLA world model automatic driving system of mixed experts and truncated diffusion, which comprises a unified autoregressive Transformer backbone, the layers of the backbone contain a cognitive driving causal attention module and a mixed expert feedforward network, the backbone is used for jointly generating future world tokens and instant action tokens, an anchor token is generated through the generation of the anchor token, a truncated diffusion sampling module performing deep truncated denoising sampling is initialized by the anchor token, and finally, the instant action token is generated, the cognitive driving causal reasoning mechanism makes the system upgrade from perception driving to cognitive driving, and the problem of decision making of automatic driving in a complex interactive scene is solved.

Description

TECHNICAL FIELD

[0001] The application relates to a cognitive-driven VLA world model automatic driving system based on mixed experts and truncated diffusion, and belongs to the technical field of artificial intelligence and automatic driving. BACKGROUND

[0002] At present, computer systems based on specific computing models, especially large models using autoregressive generation, have become the mainstream technical path for processing high-dimensional heterogeneous information flow. Through unified sequence modeling capability, they have shown effectiveness in various complex cognitive tasks. However, when such computing models are applied to dynamic environments that require real-time and continuous planning, their computing architecture exposes limitations. On the one hand, to achieve high-precision world state representation and complex logical reasoning, the model parameter size continues to grow, resulting in a delay in reasoning due to the dense activation of the computing method, which cannot meet the real-time requirements of high-frequency decision-making. On the other hand, to pursue real-time, the simplified model is limited in representation ability and cannot build an internal world model that can be used for long-term deduction, limiting its planning ability.

[0003] Attempts by the industry to solve this problem have introduced new problems. For example, models using multi-step iterative generation can simulate the multi-modal uncertainty of decision-making, but their own multi-round denoising process makes their computational overhead too large, and forcibly compressing the number of iterations often accompanies a decline in generation quality and a loss of diversity. At the same time, technical solutions attempt to use the reasoning ability of large language models (LLM) to decouple complex driving tasks, trying to manage the decision-making process through modularization such as perception, memory, planning, and reflection. For example, Chinese invention patent publication CN119203388A discloses a large language model-driven automatic driving motion planning method and system. Although this solution introduces cognitive and reflection modules, it is essentially a highly serial modular architecture that relies on large language models for high-level strategy generation and separates the understanding of the world state from the final motion planning through experience memory retrieval. This architecture not only has bottlenecks in information transmission and fusion between modules, but also makes it difficult to meet the high-frequency real-time decision-making requirements of autonomous driving based on retrieval and multi-step reasoning. This makes the world model's predictions unable to be fully utilized by the action model, and the two cannot form a deep coupling in the computational representation level.

[0004] Therefore, how to provide a computing model architecture that can balance model capacity and computational efficiency, decision-making diversity and generation real-time, and the coupling of world prediction and action generation, has become a technical problem to be solved by the present application. SUMMARY

[0005] This invention provides a cognitively driven VLA world model autonomous driving system that combines experts and truncated diffusion. Its main purpose is to solve the problem that it is difficult to balance model capacity and computational efficiency, decision diversity and real-time generation, and world prediction and action generation in the existing computational model architecture.

[0006] To achieve the above objectives, this invention provides a hybrid expert and truncated diffusion cognitive-driven VLA world model autonomous driving system. The system includes a unified token representation space, a world state encoder, an autoregressive Transformer backbone, and a truncated diffusion sampling module.

[0007] A unified token representation space is used to map visual observations, language commands, action sequences, and the world state of a 3D Gaussian field consisting of a set of 3D Gaussian units to a common vocabulary.

[0008] World State Encoder is used to compress 3D Gaussian field world state into future world tokens;

[0009] An autoregressive Transformer backbone receives the input sequence and autoregressively generates an output sequence containing a future world token and an immediate action token. The Transformer layer of the autoregressive Transformer backbone includes: a cognitively driven causal attention module, which embeds a causal graph prior for the driving domain generated by encoding a causal model of the driving domain structure through a graph neural network; and a hybrid expert feedforward network, which includes multiple expert networks and a routing network. The routing network dynamically selects the top-4 expert networks for sparse activation. When generating the immediate action token, the autoregressive Transformer backbone generates an anchor token.

[0010] The truncated diffusion sampling module initializes the Gaussian distribution by the distribution parameters mapped to the anchor token, and performs a fixed 10-step denoising sampling process starting from the initialized Gaussian distribution to generate an instant action token.

[0011] Preferably, the future world token is a sequence of 128 discrete tokens used to represent each future time step through Gaussian VQ-VAE compression, and the instant action token is a sequence of 8 discrete tokens representing macroscopic intentions, mesoscopic trajectories and microscopic adjustments respectively through three-level hierarchical quantization encoding.

[0012] Preferably, the multiple expert networks of the hybrid expert feedforward network are divided into 4 functionally specialized expert groups, including a group of perception experts for processing visual representations of different driving environments, a group of reasoning experts for performing causal reasoning and planning, a group of modeling experts for predicting geometric and semantic evolution of world states, and a group of decision experts for generating specific driving maneuver actions.

[0013] Preferably, the system also supports do-calculus intervention operations; intervention operations simulate forced interventions and support counterfactual reasoning by modifying the attention mask of the cognitive-driven causal attention module during reasoning to mask causal paths pointing to intervention variables in the causal graph priors of the driving domain.

[0014] Preferably, the system also adopts a load balancing auxiliary loss during training to ensure balanced usage of the multiple expert networks; the load balancing auxiliary loss is calculated as: where N is the total number of the multiple expert networks, is the actual frequency of expert j being selected by the routing network in a batch, is the probability of expert j being predicted to be selected by the routing network.

[0015] Preferably, the 3D Gaussian cell is constructed as where x is the position, is the covariance matrix, is the color, is the opacity, is the motion velocity; the system also includes a Gaussian VQ-VAE decoder for decoding and reconstructing the future world tokens generated by the autoregressive Transformer backbone into 3D Gaussian field world states.

[0016] Preferably, the 3D Gaussian cell is divided into static Gaussian cells and dynamic Gaussian cells, the static Gaussian cells are rigidly transformed with the ego motion, and the dynamic Gaussian cells are updated in position according to the predicted motion velocity of the system.

[0017] Preferably, during reasoning, the system generates multiple candidate action-world joint sequences by changing the random seed of the truncated diffusion sampling module; the system performs multiple steps of unrolling for each candidate sequence in the multiple imagined worlds corresponding to the multiple candidate action-world joint sequences reconstructed from the future world tokens, and selects the optimal sequence for execution according to a comprehensive scoring function containing safety and efficiency indicators.

[0018] Preferably, the system adopts a four-stage progressive training paradigm, and the four-stage progressive training mode comprises: stage one, self-supervised world modeling, used for learning a world state encoder and a world state prediction; stage two, supervised action alignment, used for aligning the action generated by the system with expert behavior; stage three, thought chain distillation, used for training the system to generate an interpretable reasoning chain token; and stage four, reinforcement learning optimization, used for optimizing the decision success rate of the system in a simulation environment.

[0019] Preferably, the autoregressive Transformer backbone jointly generates future world tokens and instant action tokens in the same autoregressive generation process to realize deep coupling of action generation and world state evolution.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] 1. The present application constructs a unified discrete token representation space, which maps heterogeneous information such as visual occupancy, world state, and action sequence to a common computational vocabulary. This kind of representation-level unification enables a single autoregressive Transformer architecture to jointly model world evolution and action generation, avoiding the inherent information loss and error accumulation when traditional modular computing systems transfer information between different representation spaces.

[0022] 2. The present application combines the hybrid expert architecture with the causal attention mechanism. The specialized reasoning expert group and decision expert group, guided by the causal graph prior, enable the computing model to go beyond surface data associations and use embedded causal structures for reasoning when processing information. This approach enables the system's output decisions to be based on predictions of the development logic of things rather than simple imitation of historical data, improving the model's computing effectiveness in unseen or complex interactive situations.

[0023] 3. By anchoring the Gaussian truncated diffusion method, the inherent computational delay problem of the computing model in generating multi-modal uncertainty output is solved. The model quickly generates a high-probability anchor distribution through autoregression, and only performs a limited number of denoising sampling at this time sequence truncation point. This mechanism combines the fast reasoning ability of autoregressive models with the fine sampling ability of diffusion models, without sacrificing output diversity, compressing the computational time of the generation process to a controllable low level, making it suitable for computing tasks with high real-time requirements. BRIEF DESCRIPTION OF DRAWINGS

[0024] Fig. 1 is a schematic diagram of the collaborative architecture of the autoregressive backbone and truncated diffusion sampling of the present application;

[0025] Fig. 2 is a schematic diagram of the comprehensive score of multi-hypothesis forward-looking evaluation of the present application;

[0026] Fig. 3 The training process of the model of the application and the schematic diagram of the vehicle-mounted deployment architecture. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the application clearer, the technical scheme of the application will be described in detail below in combination with the drawings and specific embodiments, but it should be understood that the detailed description and specific embodiments of the application are only used to explain the application, and not to limit the protection scope of the application.

[0028] The disclosed cognitive driving VLA world model automatic driving system of mixed experts and truncated diffusion is an end-to-end system architecture based on a specific computing model. The system mainly includes a unified token representation space, a world state encoder, an autoregressive Transformer backbone, and a truncated diffusion sampling module in engineering implementation. In the data flow of the system, the external multi-modal input is mapped to the unified token representation space via the respective encoder. The autoregressive Transformer backbone generates future world tokens representing future world evolution and immediate action tokens representing current decisions in an autoregressive manner, taking these tokens as context. The final generation of immediate action tokens is obtained by first outputting an anchor token from the backbone, initializing the truncated diffusion sampling module with the anchor token, and performing fast refinement sampling. In the specific computing task of automatic driving, the traditional modular system adopts heterogeneous intermediate representations in the perception, prediction, planning, and other links, resulting in loss and error accumulation when information is transmitted between modules. Therefore, the system constructs a unified token representation space to map all heterogeneous information streams to a common computing vocabulary V, so that a single computing model can be uniformly processed. Specifically, for visual observation, the system collects a sequence of 6-view RGB images A 4D space-time occupancy encoder is used to convert a continuous occupancy grid representing the environment occupancy into a discrete visual ; For language instructions, a standard sub-word tokenizer is used to encode language tokens ; For vehicle state, a vector containing 16 continuous values such as speed and orientation is quantized into 4 discrete state ; For action sequence, the system compresses the future trajectory into 8 discrete immediate actions The encoder is designed to divide the eight tokens into three levels, corresponding to macroscopic intent, mesoscopic trajectory, and microscopic adjustment. For the representation of the world state, this system employs a set of 3D Gaussian primitives. A 3D Gaussian field is constructed, and a specialized world-state encoder, namely Gaussian VQ-VAE, is used to process this Gaussian field. Compressed into a discrete future world The future world token is a sequence of 128 discrete tokens used to represent each future time step, thus all modalities are converted into a token sequence on the common vocabulary V.

[0029] After achieving representational unification, the core computation of the system is performed by an autoregressive Transformer backbone. This backbone addresses the problem of separation between world prediction and action planning in existing models, failing to form a cognitive loop. The backbone receives the input sequence... Visuals that include history ,language Current status And prompts for controlling the generation mode Based on the input sequence, the backbone jointly and autoregressively generates a unified output sequence. The output sequence also contains an interpretable inference chain. Current pending actions And the future world H time steps ahead. This computational method, which jointly generates actions and world states within the same sequence, enables the generation of actions... Able to evolve the future world This is a direct condition, and vice versa, thus achieving deep coupling and closed-loop optimization of the two at the representation level of the computational model. Furthermore, the internal structure of this autoregressive Transformer backbone is designed to simultaneously address the dual challenges of computational efficiency and cognitive depth in autonomous driving tasks. Firstly, to address the inference efficiency bottleneck caused by the expansion of large model parameters, the Transformer layer of this backbone employs a hybrid expert feedforward network (MoE-FFN), which contains multiple expert networks—32 independent in a specific implementation. And a routing network; during computation, the routing network calculates the routing score based on the hidden state h of the current token. The top-K highest-scoring expert networks in this scheme (K=4) are dynamically selected for sparse activation, and their outputs are... Only the contributions of these four experts are weighted, so that under the condition of having a total parameter capacity of 70B, only 8.75B of computing capacity is activated each time, improving computing efficiency; secondly, to ensure the balance of expert use and avoid overloading of a few experts, the system introduces a load balancing auxiliary loss during training , where N is the total number of experts, such as N = 32, is the actual frequency of expert j being selected in the batch, is the probability of the routing network predicting selecting this expert, and this loss function is used to punish between , prompting the routing network to evenly distribute the computing load; thirdly, for functional specialization, the N = 32 expert networks are divided into four groups of functionally specialized experts, namely a group of perception experts (E1-E8) for processing different driving environment visual representations, a group of reasoning experts (E9-E16) for performing causal inference and planning, a group of modeling experts (E17-E24) for predicting the geometric and semantic evolution of the world state, and a group of decision-making experts (E25-E32) for generating specific driving maneuver actions.

[0030] At the same time, in order for the computing model to obtain cognitive ability beyond the surface association of data, the Transformer layer of the autoregressive Transformer backbone also contains a cognitive-driven causal attention module; standard attention mechanisms can only capture correlations, and are easily misled by false correlations in complex interaction scenarios; for this reason, the system constructs a structural causal model (SCM) in the driving domain, which defines the causal relationships between variables such as ego vehicle actions, other vehicle behaviors, and road conditions, and encodes them into a causal graph prior matrix for the driving domain through a graph neural network (GNN); During attention calculation, the prior matrix C is embedded into the standard attention formula, i.e. , which is regulated by the parameter ; this prior matrix C can enhance the attention weight between tokens with real causal correlation, while suppressing false correlations; in addition, the system also supports do-calculus intervention operations to realize counterfactual reasoning; during reasoning, when an intervention is needed, the system modifies the attention maskTo achieve the graph surgery, i.e., to mask all the causal paths pointing to the intervened variable X in the causal graph prior, to simulate the effect of forced intervention within the computational model, so that the system can evaluate the possible consequences if a different action was taken at the time; for autonomous driving decisions, a simple deterministic action output cannot model the multi-modal nature of real-world driving decisions such as turning left or waiting at an intersection, while the multi-step iteration of 50-1000 steps of the standard diffusion model cannot meet the real-time requirement of more than 10 Hz; to address this computational delay bottleneck, the system uses a truncated diffusion sampling module in cooperation with an autoregressive backbone, whose execution process is as follows: in the first stage, when an immediate action token needs to be generated, the autoregressive Transformer backbone quickly generates an anchor point with the highest probability through its efficient autoregressive reasoning ; in the second stage, the anchor token is mapped to a set of Gaussian distribution parameters through codebook lookup, and this set of parameters determines the initialization Gaussian distribution of the truncated diffusion sampling module; in the third stage, the sampling process does not start from pure noise, but from the anchor distribution , and the number of denoising iterations is deeply truncated to a fixed small value, i.e. steps; the system performs these 10 steps of denoising sampling, and in a specific implementation, uses DDIM sampling , whose denoising calculation at each step is conditioned on the hidden state h of the Transformer backbone, predicts the noise through a lightweight denoising network , and iterates 10 times according to the update formula and to finally generate the refined immediate action token; this mechanism combines the fast positioning ability of autoregression with the fine sampling ability of diffusion model, while maintaining the multi-modal generation ability, and compresses the reasoning delay to within the real-time requirement (such as 50 ms).

[0031] The computational model of the system also needs a high-fidelity, real-time interactive internal world representation to support planning; for this purpose, the system uses a 3D Gaussian splatter field to represent the world state, and generates its token sequence through an autoregressive Transformer backbone ; when the future world needs to be explicitly visualized, the system also includes a Gaussian VQ-VAE decoder to decode and reconstruct the future world token generated by the backbone into a 3D Gaussian field world state ; the elements constituting the Gaussian field include position , covariance matrix , color , opacity , and key motion velocity ; To realize dynamic evolution, these Gaussian cells are divided into static and dynamic types; static Gaussian cells Like road, building location updates according to the motion of the vehicle Perform rigid transformation ; while dynamic Gaussian cells Like other vehicles are updated according to the system's predicted motion speed of their own Perform location update ; The 3D Gaussian field supports differentiable high-speed rendering of 150 FPS or more, enabling the system to perform low-cost trial and error in the imagined future world; To ensure that the mapping relationship between the effective activation of the function specialized expert group in the mixed expert feedforward network and the initialization Gaussian distribution of the anchor token generated by the autoregressive Transformer backbone diffusion sampling module is stable, the system introduces two auxiliary training procedures in the early stage of the four-stage progressive training, i.e., stage one self-supervised world modeling and stage two supervised action alignment, in addition to the load balancing auxiliary loss : One is the task-guided routing loss , which forces the routing score g of input tokens of a specific task type, such as world modeling task tokens, to be guided to a pre-set expert group, such as the modeling expert group, in the early stages of training, and the weight coefficient gradually decays until it is removed as the training step increases, and then relies on the routing network to perform autonomous dynamic routing based on and the main task loss; The second is the anchor distribution alignment procedure, which uses an independent action encoder to encode expert actions into target distribution parameters , and introduces an anchor alignment loss to minimize the KL divergence between the initialization distribution parameters mapped by the anchor points generated by the backbone and the , while the diffusion denoising loss optimizes the denoising network synchronously, enabling it to reconstruct from the distribution within a fixed 10-step denoising sampling .

[0032] To utilize the differentiable high-fidelity world model, the system uses a multi-hypothesis forward-looking evaluation mechanism during inference; first, the system generates multiple candidate action-world joint sequences, such as K=16, by changing the random seed of the truncated diffusion sampling module; second, the system evaluates the future world In the corresponding imagined world that has been decoded and reconstructed, each candidate sequence undergoes multiple steps, such as H=5 steps of rollout; finally, the system, based on a security-related... ,efficiency Comfort and adherence to rules Comprehensive scoring function for indicators such as The K=16 complete trajectories derived through H steps are evaluated, and the trajectory with the highest score is selected. This mechanism enables the system to evaluate a total of K×H (e.g., 16×5=80) future possibilities before executing physical actions, improving decision-making robustness in complex and edge scenarios. To ensure effective training of this complex system, a four-stage progressive training paradigm is adopted. Stage 1, self-supervised world modeling, utilizes a large number of unlabeled driving videos to train the world state encoder Gaussian VQ-VAE and its world state prediction capabilities through self-supervised tasks such as future frame generation and mask occupancy prediction. Stage 2, supervised action alignment, uses labeled expert driving clips and methods such as behavior cloning (BC) to align the system's generated instantaneous actions. Aligning expert behavior; Phase 3, thought chain distillation, using manually labeled reasoning process data, through sequence-to-sequence supervised learning, trains the system to generate interpretable reasoning chain tokens. Phase four involves reinforcement learning optimization. In high-fidelity simulation environments such as CARLA, algorithms such as Proximal Policy Optimization (PPO) are used, with security and traffic efficiency as reward signals, to optimize the system's decision success rate in complex interactive scenarios end-to-end.

[0033] To achieve real-time partitioning of 3D Gaussian Gaussian bits into static and dynamic Gaussian bits, the World State Encoder (Gaussian VQ-VAE) incorporates the following into its coding structure, except for the prediction position... Covariance In addition to parameters, it is also trained to output a dynamic attribute scalar for each Gaussian element i. Its value range is between 0 and 1; The training supervision signal comes from the self-supervised world modeling in stage one. Specifically, the system calculates the actual observed displacement of each primitive i between consecutive time steps t and t+1. And simultaneously calculate the vehicle's motion Resulting rigid transformation displacement When the residual between the two Greater than a preset motion threshold When the unit is in meters per frame, the target of the primitive i at time t. It is set to 1, and vice versa; during inference, the system relies on the encoder output. with a fixed classification threshold >0.5, the dynamic Gaussian primitive Gdyn(j) is partitioned according to the predicted velocity The position is updated while the static Gaussian primitive Gstat performs rigid transformation; the system adopts a joint multi-task objective function during training wherein is the aforementioned MoE load balancing loss; the remaining five loss objectives include: i.e., the autoregressive language modeling loss for supervised inference chain ; i.e., the action prediction loss (such as L2 loss and KL divergence) for aligning expert actions ; i.e., the world state reconstruction loss, which internally contains the rendering loss (the difference between the predicted rendered image and the real image ) and the token reconstruction loss (supervising the generation of the world ); i.e., the geometric consistency loss, which measures the geometric position difference between the predicted Gaussian field and the real scene by Chamfer distance ; and i.e., the diffusion denoising loss, which is in the form of , used to train the denoising network in the truncated diffusion sampling module .

[0034] In a typical complex decision-making scenario, the autonomous driving system of the present application faces a left-turn intersection without protection during a city traffic peak period. The system's main lane is green, but the opposing straight traffic flow is dense, and there are pedestrians on the right side of the crosswalk who are blocked by the A-pillar or the front car. This scenario presents a double contradiction to the calculation model. First, the decision must be made within a very short time, such as 100 ms, to respond to high-speed oncoming vehicles, which imposes strict constraints on calculation delay. Second, the decision space has a high degree of multimodality, and the system needs to evaluate multiple possibilities simultaneously, including waiting in place, slowly crawling to get a better view, and quickly passing through the gap between opposing traffic, as well as the complex interaction consequences derived therefrom. After receiving this high-dimensional heterogeneous input, the system encodes the 6-view image sequence, vehicle dynamics, and intersection topology provided by the high-precision map into a unified token sequence through its unified token representation space. This sequence is input into the autoregressive Transformer backbone. In the face of this complex scenario, the routing network of the mixed expert feedforward network (MoE-FFN) inside the backbone is activated, dynamically routing the calculation load to the expert group highly relevant to the task. Specifically, experts in the perception expert group for handling urban congestion environments, experts in the reasoning expert group for causal reasoning and risk assessment, and experts in the modeling expert group for predicting dynamic interactions are highly activated. Taking Top-4 as an example, while other irrelevant experts, such as decision-making experts for high-speed cruising, remain sparsely activated, thereby concentrating the calculation amount of the current reasoning step on key tasks without sacrificing the total capacity of the model. Inside the Transformer layer of the backbone, the cognitive-driven causal attention module begins to operate. This module uses its embedded priori causal graph in the driving field to strengthen the weight of key causal chains when calculating attention, not just focusing on the correlation at the surface level of data. The system will highly focus on the causal relationship between the speed of the oncoming vehicle and whether it has a deceleration trend when approaching the intersection, as well as the head direction of the pedestrian, such as whether the pedestrian's head is looking at the oncoming vehicle and the causal relationship between its future trajectory, thereby allowing the calculation model to predict that the oncoming vehicle A has no deceleration intention and will force its way through, and that pedestrian B is distracted and at risk of entering the lane.

[0035] Based on this cognitive judgment, the system enters the multi-hypothesis prospective evaluation stage. Instead of calculating a single optimal solution, the system generates K=16 candidate action-world joint sequences by changing the random seed of the truncated diffusion sampling module. These sequences semantically correspond to different decision branches: candidate A corresponds to immediately waiting in place, candidate B corresponds to slowly crawling 1 meter and then waiting, and candidate C corresponds to starting and turning left in the gap between vehicle A passing and vehicle D arriving. At this point, the system's core collaborative mechanism is triggered. The autoregressive Transformer backbone utilizes its efficient hybrid expert (MoE-FFN) architecture and causal attention module to quickly autoregressively generate H=5 future timestep future world tokens for each of these K=16 candidate branches. These token sequences are then decoded in parallel by the Gaussian VQ-VAE decoder into K=16 independent 3D Gaussian field imagined worlds. In the imagined world of candidate C, the system predicts the speed of vehicle D. The situation will change due to the intervention of the vehicle (candidate C), such as an emergency deceleration. In candidate B's imagined world, pedestrian B's... This indicates that it will enter the conflict zone of the vehicle (candidate B); the generation of this multimodal action-world sequence solves the real-time bottleneck of the traditional diffusion model; when generating each instantaneous action token, the system first uses the autoregressive Transformer backbone to quickly generate an anchor token, a high-probability neighborhood representing waiting, and the truncated diffusion sampling module only samples from the distribution mapped by this anchor token. Initially, a fixed 10-step denoising sampling process is performed to refine high-quality and diverse specific actions, such as waiting and keeping the steering wheel straight. This reduces the generation time of a single decision to less than 50ms, thus resolving the contradiction between decision diversity and real-time computation. After obtaining K=16 future trajectories that have been extrapolated through H=5 steps, the system, based on preset safety parameters... With efficiency The comprehensive scoring function is used for evaluation; candidate C, the forced insertion scheme, receives a very low score because it causes the collision time TTC of vehicle D to be below the safety threshold in the imagined world. The scoring process shows that candidate B, the creeping solution, receives a lower score due to the potential conflict with pedestrian B. Ultimately, candidate A, the solution of waiting in place for both vehicle A and pedestrian B to pass, receives the highest overall score. The system ultimately selects to execute the optimal sequence of candidate A, outputting a stable waiting action token, and the vehicle remains stationary.

[0036] Example 2: To objectively quantify the performance of the system of the present application in terms of computational efficiency and decision robustness, especially in the high dynamic and high interactive complexity urban scenarios, this example constructs a standardized test environment based on the CARLA simulation platform; this platform is chosen for its high-fidelity physics engine and sensor simulation capabilities, and the test scenario is set to a left-turn intersection in the Town05_Opt map, which reproduces the high-risk working conditions in Example 1, including dense opposite straight-flow and potential occluded pedestrians; to evaluate the performance of the system under different pressures, the experiment sets two traffic flow densities, i.e. 20 vehicles / minute for medium density opposite flow and 45 vehicles / minute for high density opposite flow, accompanied by intermittent pedestrian crossing, and each configuration is executed 1000 independent closed-loop simulation tests to obtain statistical data; the core of this experiment is to compare three different computational model configurations to isolate and verify the effect of the key technical features of the present application; the first group is the sample group of the present application, which fully deploys all the modules as described in the detailed description, including a mixed expert feedforward network MoE-FFN with a total parameter quantity of 70B, Top-4 sparse activation, cognitive-driven causal attention module and anchor Gaussian truncated diffusion sampling module Step; the second group is control group A, i.e. a dense baseline model, which uses a dense Transformer architecture with a total parameter quantity (about 8.75B) similar to the activated parameter quantity of the sample group of the present application in a single inference, but the control group does not have the MoE architecture and the causal attention module; the third group is control group B, i.e. a standard diffusion model, which is consistent with the sample group of the present application except that its truncated diffusion sampling module is replaced by a standard diffusion sampler that requires 100 steps of iteration (T=100); during the experiment, the system records the average single-frame inference delay (ms), the average activated parameter quantity (B), the collision rate (%) and the task success rate (%) of each simulation instance, where the task success is defined as completing the left turn safely and collision-free within 60 seconds; the experimental data are summarized in Table 1.

[0037] Table 1: Performance comparison test table under different computational model configurations

[0038]

[0039] Referring to Table 1, the experimental data show clear performance differences; in terms of computational efficiency, the average inference delay of the sample group of the present application is about 48.9ms, which is on the same order of magnitude as the control group A of about 45.5ms, and both satisfy the real-time decision requirement of 10Hz (100ms) for vehicle-mounted systems, while the control group B uses 100-step standard diffusion, with a delay of about 474.4ms, which is far beyond the real-time threshold, confirming the effectiveness of the anchor Gaussian truncated diffusion (AGTD) sampling mechanism of the present application plays a role in maintaining extremely low latency; second, in decision-making quality, although the reasoning delay of the inventive sample group and control group A is similar, the task success rates of the two groups show a difference of 90.3% and 58.4% in high-density scenarios, and the collision rate (1.9%) of the inventive sample group is much lower than that of control group A (15.6%); this group of data shows that the inventive sample group effectively utilizes the world modeling and causal reasoning ability carried by its 70B total parameter capacity through the sparse activation mechanism of MoE-FFN, which is further enhanced by the causal attention module, while the control group A with the same amount of calculation cannot handle this complex interaction scenario due to the limitation of model capacity.

[0040] Example 3: This example sets up a comparative example 1; the comparative example 1 uses the same test platform, high-density test scene and mixed expert feedforward network MoE-FFN containing 70B total parameters as the inventive sample group in example 2, Top-4 sparse activation and 10-step truncated diffusion calculation model; the only difference is that the cognitive-driven causal attention module is removed and replaced by the Transformer self-attention mechanism for sequence modeling in the art; the calculation model of the comparative example 1 is placed in the same 1000 closed-loop simulation tests as the high-density scene in example 2; it is observed in multiple tests that when facing the unprotected left turn working condition in example 1, the self-attention mechanism of the model only calculates based on the correlation of the data surface; in a typical failure case, the model observes two signals, an oncoming high-speed car (vehicle A) and a side-front occluded pedestrian (pedestrian B); due to the lack of guidance of the causal graph prior, the model gives high weight to the false correlation between the existence of the pedestrian and the deceleration of the oncoming car in the calculation, and fails to fully utilize the observation data of the vehicle A itself without attenuation; based on this calculation, the model outputs the immediate action token of starting left turn at the moment when the inventive sample group chooses to wait, directly leading to a side collision with vehicle A; after 1000 repeated tests, the key performance indicators are shown in Table 2.

[0041] Table 2: Comparative test table of cognitive-driven causal attention module stripping

[0042]

[0043] As shown in Table 2, the data indicates that, while keeping the number of computational parameters and inference latency almost unchanged (49.0ms vs 49.1ms), removing the cognitively driven causal attention module caused the system collision rate to increase from 1.9% to 12.8%, while the task success rate decreased from 90.3% to 65.2%. This result shows that the standard self-attention mechanism has limitations in handling such complex interaction scenarios, while the cognitively driven causal attention module is effective in solving such interactive decision-making problems.

[0044] Example 4: This example combines Figs. 1 to 3 An explanation of autonomous driving systems using a cognitively driven VLA world model with hybrid experts and truncated diffusion, such as... Fig. 1 As shown, the system receives multimodal inputs, including visual observations, language commands, and action sequences. These inputs are mapped to a common vocabulary via a unified token representation space. Simultaneously, the 3D Gaussian field world state is compressed through a world state encoder. The outputs of both are fed into an autoregressive Transformer backbone. The backbone contains a cognitively driven causal attention module for embedding causal graph priors and a hybrid expert feedforward network for sparse activation. When the backbone performs joint autoregressive generation of output sequences, it jointly generates future world tokens to predict future world state evolution and generates an anchor token. This anchor token initializes a truncated diffusion sampling module, which performs a fixed 10-step denoising sampling process to finally generate an instantaneous action token for real-time driving decisions.

[0045] like Fig. 2 As shown, the horizontal axis of the graph lists 16 options, from candidate 1 to candidate 16, and the vertical axis represents the evaluation value, ranging from 0 to 100. The graph specifically presents three scoring curves: the solid line represents the safety score. The curve fluctuates significantly, peaking at candidates 4, 8, and 11, while dropping to a distinct trough at candidate 6. The efficiency score is represented by the dashed line. The curve also exhibits strong volatility, and its trend often contradicts the safety score; when the safety score of candidate 4 reaches its peak, the efficiency score... It is at a low point, while the efficiency score of candidate 13 is low. When it is high, the safety score is The score is relatively low. In stark contrast to the previous two curves, the comprehensive score, represented by a dotted line, shows high stability, remaining at a high level of around 80 points across all 16 candidates. This demonstrates the system's trade-off and balance between safety and efficiency. Ultimately, the system selects the optimal sequence based on this relatively stable and high comprehensive score.

[0046] like Fig. 3As shown, the upper training flow demonstrates how the model is trained and deployed through Phase 1: Self-supervised world modeling, Phase 2: Supervised action alignment, Phase 3: Mind chain distillation, and Phase 4: Reinforcement learning optimization of the PPO algorithm in the CARLA simulator. The lower end-to-end system flow of the in-vehicle computing platform shows how, during actual operation, inputs such as visual observation of 6-view image sequences, language commands, and vehicle status are fed into the autoregressive Transformer backbone through a unified token representation space. This backbone utilizes its internal hybrid expert feedforward network and cognitively driven causal attention module to generate future world tokens through internal forward planning and to generate immediate action tokens through the anchor token initialization truncated diffusion sampling module, which are then ultimately executed by the vehicle actuators.

[0047] Example 5: The effectiveness of the cognitively driven causal attention module in the computational model of this invention depends on the embedded causal model of the driving domain. To ensure the accuracy and completeness of the model and to clarify the model construction process, this embodiment discloses a method for building, verifying, and maintaining the model. The reproducible procedure is executed offline before system deployment. Its first step is the definition of nodes and edges. The system's nodes are identified as key dynamic and static factors in the driving scenario, including 128 variables such as vehicle actions, other vehicle behaviors, pedestrian states, traffic lights, and road conditions like slipperiness and slope. The system's edges represent causal relationships, established through a two-step process: first, deterministic causal relationships are predefined based on physical laws such as vehicle dynamics and optics; second, a large-scale real-world driving log and accident database (up to 10 million kilometers) is used to mine causal chains hidden in statistical data through causal discovery algorithms, such as PC or FCI algorithms. These are then finalized by domain engineers, thus constructing an initial causal framework containing thousands of causal relationships. ;Should The prior is encoded into a graph neural network (GNN). To verify the validity of this prior, the system performs counterfactual verification in the CARLA simulator. Taking a standard following scenario as an example, the system performs an intervention query. The predicted collision probability (75% in this case) is compared with the actual ground collision frequency (72% in this case) caused by the delayed braking action in the simulator. The model's counterfactual prediction error is only considered valid if it falls below a preset threshold (5% in this case). Only then is it considered valid; at the same time, to ensure the long-term adaptability of the model, this... In a dynamic maintenance cycle, the system continuously extracts new interaction data from difficult scenarios or takeover events collected by the fleet, and periodically, which can be set to quarterly, re-verifies and incrementally updates the SCM.

[0048] Furthermore, the joint training objective function of the system of the present invention The convergence and final performance depend on six key hyperparameters, namely... to and The calibration of these weights is used to balance the relationship between the five main tasks and MoE load balancing. To clarify this calibration process, this embodiment adopts a phased parameter calibration procedure coupled with a four-stage progressive training paradigm. The first step of the procedure, in the first stage of training, namely the self-supervised world modeling stage, is to calibrate the weights related to actions, reasoning, and diffusion. The value is temporarily set to 0. At this point, the system focuses on optimizing the world model itself, using a grid search method to find the optimal value for world state reconstruction within an interval such as [0.1, 1.0]. With regard to geometric consistency The optimization objective is to maximize the occupancy prediction IoU on the validation set while ensuring that the rendering quality PSNR is not lower than 32dB. This step yields a preliminary set of weights. For example; in the second step of the procedure, during the second phase of training, namely the supervised movement alignment phase, the calibrated... , And introduces methods for action prediction. With the use of diffusion denoising The optimization objective at this point is to minimize the trajectory error while maintaining the performance of the world model (i.e., PSNR not lower than 32dB and no degradation). Through trade-offs, a set of weights is determined to... For example; in the third step of the procedure, during the third stage of training, namely the thought chain distillation stage, the preceding weights are fixed, and a set of weights for language modeling are introduced. This weight The calibration criteria are to ensure that the reasoning chain achieves a target value of human comprehensibility, which is 85% in this case, and to determine the minimum weight value to avoid interference with action decision-making. The fourth step of the procedure involves load balancing weights throughout all stages. It is always activated, and its calibration principle is to continuously monitor the routing percentage of all experts during training, and set a target that the coefficient of variation of expert load should be less than 0.2. The value, such as The calibration process is dynamically adjusted to meet this constraint, thus avoiding overload or underutilization of a few experts; this four-step calibration procedure can provide... Determine a set of reproducible weighting coefficients with engineering basis, such as... and thereby provide a basis for the stability and final performance of the computational model training process.

[0049] It is apparent for a person skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but that the present application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate but not limit the technical solutions of the present application, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those of ordinary skill in the art that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A hybrid expert and truncated diffusion cognitive-driven VLA world model autonomous driving system, characterized in that, The system includes a unified token representation space, a world-state encoder, an autoregressive Transformer backbone, and a truncated diffusion sampling module. A unified token representation space is used to map visual observations, language commands, action sequences, and the world state of a 3D Gaussian field consisting of a set of 3D Gaussian units to a common vocabulary. World State Encoder is used to compress 3D Gaussian field world state into future world tokens; An autoregressive Transformer backbone is used to receive the input sequence and jointly autoregressively generate an output sequence containing the future world token and the immediate action token. The Transformer layer of the autoregressive Transformer backbone contains: a cognitively driven causal attention module, which embeds a causal graph prior of the driving domain generated by encoding a causal model of the driving domain structure through a graph neural network. It also includes a hybrid expert feedforward network, which contains multiple expert networks and a routing network. The routing network is used to dynamically select the top-4 expert networks from the multiple expert networks for sparse activation. The autoregressive Transformer backbone generates an anchor token when generating an instant action token. The truncated diffusion sampling module initializes the Gaussian distribution by the distribution parameters mapped to the anchor token, and performs a fixed 10-step denoising sampling process starting from the initialized Gaussian distribution to generate an instant action token.

2. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The future world token is a sequence of 128 discrete tokens used to represent each future time step through Gaussian VQ-VAE compression. The instant action token is a sequence of 8 discrete tokens used to represent macroscopic intentions, mesoscopic trajectories and microscopic adjustments through three-level hierarchical quantization encoding.

3. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The multiple expert networks of the hybrid expert feedforward network are divided into four functionally specialized expert groups. These four groups include: a group of perception experts for handling visual representations of different driving environments; a group of reasoning experts for performing causal inference and planning; a group of modeling experts for predicting the geometric and semantic evolution of the world state; and a group of decision experts for generating specific driving maneuvers.

4. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The system also supports do-calculus intervention operations; these operations simulate forced intervention and support counterfactual reasoning by modifying the attention mask of the cognitively driven causal attention module during inference to mask causal paths pointing to intervention variables in the causal graph prior in the driving domain.

5. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The system also employs a load balancer to assist loss during training. To ensure that multiple expert networks are used in a balanced manner; load balancing assists in loss. The calculation method is as follows: Where N is the total number of expert networks. The actual frequency at which expert j is selected by the routing network in the batch. Predict the probability of selecting expert j for the routing network.

6. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, 3D Gaussian Composition, among which, For location, Let covariance matrix be the variance matrix. For color, For opacity, For motion speed; the system also includes a Gaussian VQ-VAE decoder to decode and reconstruct the future world tokens generated by the autoregressive Transformer backbone into 3D Gaussian field world states.

7. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 6, characterized in that, 3D Gaussian elements are divided into static Gaussian elements and dynamic Gaussian elements. Static Gaussian elements undergo rigidity transformations as the vehicle moves, while dynamic Gaussian elements are based on the motion velocity predicted by the system. Perform a location update.

8. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, During inference, the system generates multiple candidate action-world joint sequences by changing the random seed of the truncated diffusion sampling module. In the multiple imagined worlds corresponding to the multiple candidate action-world joint sequences reconstructed from the future world token, the system expands each candidate sequence in multiple steps and selects the optimal sequence to execute based on a comprehensive scoring function that includes security and efficiency indicators.

9. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The system adopts a four-stage progressive training paradigm, which includes: Stage 1, self-supervised world modeling, used to learn the world state encoder and world state prediction; Stage 2, supervised action alignment, used to align the actions generated by the system with expert behavior; Stage 3, thought chain distillation, used to train the system to generate interpretable reasoning chain tokens; and Stage 4, reinforcement learning optimization, used to optimize the system's decision success rate in a simulated environment.

10. The autonomous driving system of hybrid expert and truncated diffusion cognitive-driven VLA world model according to claim 1, characterized in that, The autoregressive Transformer backbone jointly generates future world tokens and instant action tokens within the same autoregressive generation process.

Images