Distillation training method and device of world model simulation system, computer equipment and medium

Abstract

The invention relates to a distillation training method and device for a world model simulation system, computer equipment and a medium, and the method comprises the steps: in a first stage, freezing a decoder, only training an encoder, and migrating the causal expression of a teacher world model simulation system for environment evolution to a student world model simulation system; in the second stage, the student world model simulation system runs on a target chip, freezing of the student world model simulation system is relieved, channel pruning is conducted on the student world model simulation system through a delay model configured by the target chip, and a quantization noise layer is inserted into a feature map channel; and in the third stage, deploying the student world model simulation system at the terminal master control, and enabling the configured terminal master control to collect driving data and record an environment prediction error. According to the method, through coordination of three steps of causal alignment migration, chip perception pruning quantification and vehicle end online fine tuning, the model not only meets the vehicle specification level real-time delay requirement, but also keeps high precision and is compatible with a vehicle-mounted chip, so that the overall delay, precision and hardware matching performance are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a distillation training method, apparatus, computer equipment, and medium for a world model simulation system. Background Technology

[0002] In existing technologies, autonomous driving world model simulation systems generally adopt cloud-based offline training methods, constructing ultra-large-scale neural networks on server clusters; subsequently, they rely on manually setting fixed pruning ratios to compress the model; and use the divergence of the overall output distribution as a distillation method. The above approach has the following shortcomings: First, the single inference time of the compressed model on the onboard chip far exceeds the real-time limit, failing to meet the requirements of closed-loop vehicle control; second, rule-based pruning lacks adaptability to scene differences, leading to a sharp drop in prediction accuracy for rare operating conditions; third, the distillation strategy that only aligns the final output cannot effectively convey the spatiotemporal causal relationships within the teacher model, resulting in the loss of key evolutionary logic; fourth, the training and deployment phases do not perform joint optimization for the quantization computing units of the onboard chip, leading to a mismatch between hardware computing power and network structure, further amplifying latency and errors. Summary of the Invention

[0003] To address the aforementioned shortcomings or deficiencies, this application provides a distillation training method, apparatus, computer equipment, and medium for a world model simulation system. This method, through the coordinated implementation of three steps—causal alignment transfer, chip-aware pruning quantization, and on-vehicle online fine-tuning—ensures the model meets automotive-grade real-time latency requirements while maintaining high accuracy and compatibility with automotive chips, thereby achieving a significant improvement in overall latency, accuracy, and hardware compatibility.

[0004] This application provides a distillation training method for a world model simulation system according to a first aspect. The method is based on a teacher world model simulation system and a student world model simulation system to be trained. The student world model simulation system has the same structure as the teacher world model simulation system but has fewer parameters. The student world model simulation system includes a decoder and an encoder. The method includes: In the first stage, the decoder is frozen, and only the encoder is trained. The causal expression of environmental evolution from the teacher world model simulation system is transferred to the student world model simulation system. In the second stage, the student world model simulation system is run on the target chip, the freeze on the student world model simulation system is lifted, and the channel pruning of the student world model simulation system is performed using the delay model configured on the target chip, and a quantization noise layer is inserted into the feature map path. In the third stage, the student world model simulation system is deployed on the terminal master controller, which collects driving data and records environmental prediction errors. In response to environmental prediction errors or inference delays exceeding a set threshold, an online fine-tuning process and cloud incremental distillation are triggered, followed by a backhaul update. Inference delay refers to the time required for the student world model simulation system to complete one forward inference on the target chip.

[0005] In some embodiments, the step of transferring the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system in the first stage includes: Calculate the spatiotemporal attention causal alignment loss; extract the spatiotemporal correlation weights of all attention heads in the teacher world model simulation system, and calculate the environmental causal importance score, which represents the degree to which the attention head captures key evolution patterns of dynamic obstacles or static environment; identify the attention heads with the highest scores as environmental causal key heads and establish a mapping with the corresponding layer of the student world model simulation system; during training, only align the attention weights of the environmental causal key heads to enable the student world model simulation system to reproduce the environmental evolution causal dependencies of the teacher world model simulation system.

[0006] In some embodiments, the second stage of channel pruning for the student world model simulation system includes: The theoretical inference time for each channel was measured offline on the target chip; the pruning index was calculated based on the time and the environmental causal importance score; and the channel with the smallest pruning index was removed. The pruning index is the product of the theoretical inference time of the channel and the gradient importance of the channel for the environmental state prediction task. The gradient importance is obtained through forward-to-backward propagation of a small batch of data and is used to measure the sensitivity of the channel to prediction errors of dynamic obstacles and static environment.

[0007] In some embodiments, in the second stage, the above method further includes: A quantization noise layer is inserted into the feedforward path of the pruned feature map; the parameters of the quantization noise layer are jointly optimized under the constraint of a preset training objective. The training objectives include: dynamic obstacle trajectory loss, which represents the future pose and velocity error based on dynamic obstacles; static environment geometry loss, which represents the error based on lane lines and passable areas; and quantized perception distribution loss, which represents the difference between the outputs of the teacher world model simulation system and the student world model simulation system.

[0008] In some embodiments, the optimization step of quantizing the distribution loss includes: The teacher world model simulation system outputs soft labels for the environment state, which include the future soft distribution of dynamic obstacles and static environment. The student world model simulation system outputs the corresponding soft distribution after processing with a quantized noise layer, and performs alternating optimization: In the first round, the quantized noise layer is frozen and the network weights of the student world model simulation system are updated; in the second round, the network weights of the student world model simulation system are frozen and the parameters of the quantized noise layer are updated; the above alternating optimization is repeated until the difference in the soft distribution of dynamic obstacles and static environment converges.

[0009] In some embodiments, the online fine-tuning process in the third phase includes: The student world model simulation system is deployed on the terminal main control unit, receiving real-time input from cameras, millimeter-wave radar, inertial measurement units, and high-precision maps. It outputs the future pose and velocity sequences of dynamic obstacles and the future lane lines and passable areas of the static environment frame by frame. The terminal output is compared with the true values ​​to calculate the average displacement error, velocity error of dynamic obstacles, and geometric error of the static environment. When any error or inference delay exceeds a set threshold, the following actions are taken: the original data of the high-error scene is cached on the terminal; the data is uploaded to the cloud via an encrypted link; the cloud performs incremental distillation based on the cached data to update the student world model simulation system; and the updated student world model simulation system is sent back to the terminal device for hot replacement via over-the-air download technology.

[0010] In some embodiments, the online fine-tuning process further includes: A circular cache is set up on the terminal to store high environmental error data from the most recent period. When the cache is full, the data is encrypted and uploaded to the cloud via cellular vehicle-to-everything (V2X) communication. The cloud uses a memory-playback mechanism to mix the new cached data with historical data for training, in order to maintain the memory of key patterns of dynamic obstacles and static environment. After incremental training, differential patches are generated and distributed to the terminal after differential compression. When the dynamic obstacle displacement error, static environment geometric error, and inference delay all meet the set thresholds within several consecutive frames on the terminal, the closed-loop optimization ends.

[0011] This application provides a distillation training device for a world model simulation system according to a second aspect. The device is based on a teacher's world model simulation system and a student's world model simulation system to be trained. The student's world model simulation system has the same structure as the teacher's world model simulation system but fewer parameters. The student's world model simulation system includes a decoder and an encoder. The device includes: The first-stage training module is used to freeze the decoder and train only the encoder, transferring the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system.

[0012] The second-stage training module is used to run the student world model simulation system on the target chip, unfreeze the student world model simulation system, use the delay model configured on the target chip to perform channel pruning on the student world model simulation system, and insert a quantization noise layer into the feature map path.

[0013] The third-stage training module is used to deploy the student world model simulation system on the terminal main controller, enabling the configured terminal main controller to collect driving data and record environmental prediction errors.

[0014] The backhaul update module is used to trigger the online fine-tuning process and cloud incremental distillation after backhaul update when the environmental prediction error or inference delay exceeds the set threshold. Inference delay refers to the time required for the student world model simulation system to complete one forward inference on the target chip.

[0015] Training data is constructed using pre-collected multi-source observation data, including image frame data, driving state data, and offline road data. The training data includes multiple training samples. An initial world model simulation system is built, comprising a generator network, a learnable optimization layer, and a driving condition constraint library, which provides fixed state transition constraints. The world model simulation system is iteratively trained using the training data until a preset termination condition is met. The generator network generates environmental observation information and learnable constraint vectors based on the multi-source observation data. The learnable optimization layer solves for optimized trajectory information based on environmental observation information and target constraint features. Target constraint features include state transition constraints. During each training iteration, the multi-source observation data related to the training samples used in this training is input into the world model simulation system. The path difference is determined based on the optimized trajectory information generated by the learnable optimization layer and the environmental observation information generated by the generator network. The parameters of the target network are updated based on the path difference. The target network is either the generator network or the learnable optimization layer.

[0016] In some embodiments, the driving state data includes the vehicle's longitudinal velocity data, lateral velocity data, yaw rate data, and longitudinal acceleration data; the offline road data includes lane centerline coordinate information, lane width data, and lane curvature information; and the generator network is configured with an encoder, an intermediate state transition network, and a decoder. The steps of the generator network generating environmental observation information and learnable constraint vectors based on multi-source observation data include: The encoder extracts spatial features from image frame data to obtain scene latent vectors. The scene latent vectors are then concatenated with driving state data and offline road data in the time dimension to obtain fused temporal features. The fused temporal features are then subjected to temporal evolution through an intermediate state transition network to obtain predicted latent vectors for future moments. The decoder maps the predicted latent vectors to environmental observation information and learnable constraint vectors. The environmental observation information includes the vehicle's position, speed, and acceleration information for future moments, as well as the position information of obstacles around the vehicle, road boundary information, lane line information, and passable area information.

[0017] In some embodiments, the learnable optimization layer includes a constraint embedding module, an optimization solution module, and an implicit gradient module; the step of the learnable optimization layer obtaining optimization trajectory information based on environmental observation information and target constraint features includes: The constraint embedding module concatenates or weights the learnable constraint vector with the state transition constraint to obtain the constraint vector. The position, velocity, and acceleration information from the environmental observation information are input into the optimization solution module along with the constraint vector. The optimization solution module adjusts the future trajectory of the vehicle and obstacles as a whole within the constraint space defined by the constraint vector to obtain the optimized trajectory information. The implicit gradient module performs differentiable mapping on the optimized trajectory information and calculates the gradient tensor through the implicit gradient approximation operator. The gradient tensor is used for backpropagation to update the parameters of the generator network.

[0018] In some embodiments, the steps of constructing the driving condition constraint library before training the world model simulation system include: The system collects vehicle sensor data and high-precision map data; it generates vehicle dynamics rules using the sensor data through a rigid body kinematics parameter estimation algorithm. The vehicle dynamics rules include longitudinal acceleration boundaries, lateral acceleration boundaries, and curvature continuity boundaries; it generates traffic rules using the high-precision map data, including lane keeping constraints, speed limit constraints, and collision avoidance constraints; it stores the vehicle dynamics rules and traffic rules as fixed state transition constraints in the driving condition constraint library; the state transition constraints remain frozen during the training and inference phases.

[0019] In some embodiments, updating the parameters of the target network based on path differences includes: The optimization solution module calculates the path difference vector between the optimized trajectory information and the environmental observation information. The path difference vector represents the path difference between the future position, velocity and acceleration of each object in the current driving scenario. The implicit gradient module uses the implicit gradient approximation operator to generate the gradient tensor. The gradient tensor is used to perform gradient descent update only on the parameters of the encoder, intermediate state transition network and decoder, while keeping the parameters of the learnable optimization layer, driving condition constraint library and state transition constraint frozen.

[0020] In some embodiments, during the pre-training phase, all parameters of the learnable optimization layer, the driving condition constraint library, and the state transition constraints are frozen, and only the generator network is trained; the parameters of the encoder, intermediate state transition network, and decoder are updated using gradient descent with the data distribution matching term as the loss function; during the joint fine-tuning phase, the freeze on the learnable optimization layer is lifted, while the parameters of the driving condition constraint library and the state transition constraints remain frozen; the steps of updating the parameters of the generator network and updating the parameters of the learnable optimization layer are alternately executed until the preset convergence condition is met.

[0021] In some embodiments, the steps of updating the parameters of the generator network and updating the parameters of the learnable optimization layer are performed alternately until a preset convergence condition is met, including: In the first round, the parameters of the learnable optimization layer, the driving condition constraint library, and the state transition constraints are frozen. The current environmental observation information, the learnable constraint vector, and the state transition constraints are input into the learnable optimization layer to obtain the first optimized trajectory. The first gradient tensor is generated through the implicit gradient module, and only the parameters of the encoder, intermediate state transition network, and decoder are updated. In the second round, the parameters of the generator network, the driving condition constraint library, and the state transition constraints are frozen. The updated environmental observation information, the learnable constraint vector, and the state transition constraints are input into the learnable optimization layer to obtain the second optimized trajectory. The second gradient tensor is generated through the implicit gradient module, and only the parameters of the constraint embedding module, the optimization solution module, and the implicit gradient module of the learnable optimization layer are updated. The first and second rounds are repeated until the loss function meets the preset training termination condition.

[0022] According to a second aspect, this application provides a construction apparatus for a world model simulation system, the apparatus comprising: The training data construction module is used to construct training data using pre-collected multi-source observation data; the multi-source observation data includes image frame data, driving status data, and offline road data; the training data includes multiple training samples.

[0023] The world model simulation system building module is used to build the initial world model simulation system. The world model simulation system includes a generative network, a learnable optimization layer, and a driving condition constraint library, which provides fixed state transition constraints.

[0024] The world model simulation system training module is used to iteratively train the world model simulation system using training data until the preset termination conditions are met.

[0025] The generator network is used to generate environmental observation information and learnable constraint vectors based on multi-source observation data; the learnable optimization layer is used to solve for the optimized trajectory information based on the environmental observation information and the target constraint features; the target constraint features include state transition constraints.

[0026] During each training session, the world model simulation system training module inputs multi-source observation data related to the training samples used in this training session into the world model simulation system. It determines the path difference based on the optimized trajectory information generated by the learnable optimization layer and the environmental observation information generated by the generator network, and updates the parameters of the target network based on the path difference. The target network is either the generator network or the learnable optimization layer.

[0027] According to a third aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the distillation training method for any of the world model simulation systems described above.

[0028] According to a fourth aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps of the distillation training method of any of the world model simulation systems described above.

[0029] The distillation training method of the aforementioned world model simulation system can be applied to the distillation training system of an autonomous vehicle (hereinafter referred to as the "system"). This vehicle is equipped with a LiDAR (Light Detection and Ranging) sensor (hereinafter referred to as "LiDAR") and a (vehicle-mounted) camera. The method includes: in the first stage, freezing the decoder and training only the encoder, transferring the causal expression of the teacher world model simulation system for environmental evolution to the student world model simulation system; in the second stage, running the student world model simulation system on the target chip, unfreezing the student world model simulation system, using the delay model configured on the target chip to perform channel pruning on the student world model simulation system and inserting a quantization noise layer into the feature map path; in the third stage, deploying the student world model simulation system on the terminal master control, allowing the configured terminal master control to collect driving data and record environmental prediction errors; in response to environmental prediction errors or inference delays exceeding a set threshold, triggering an online fine-tuning process and cloud incremental distillation followed by a data transmission update. Therefore, this method achieves a significant improvement in overall latency, accuracy, and hardware compatibility by working together three steps: causal alignment transfer, chip-aware pruning quantization, and on-vehicle fine-tuning. This is accomplished by meeting automotive-grade real-time latency requirements, maintaining high accuracy, and being compatible with automotive chips. Attached Figure Description

[0030] Figure 1 This is a flowchart of a distillation training method for a world model simulation system according to one or more embodiments of this application; Figure 2 This is a flowchart illustrating a method for transferring the causal expression of environmental evolution from a teacher world model simulation system to a student world model simulation system, according to one or more embodiments of this application. Figure 3 This is a flowchart of a method for channel pruning in a student world model simulation system according to one or more embodiments of this application; Figure 4 This is a flowchart illustrating a method for constructing a driving condition constraint library according to one or more embodiments of this application; Figure 5 This is a flowchart illustrating a method for optimizing quantized sensing distribution loss in one or more embodiments of this application. Figure 6 This is a flowchart of an online fine-tuning process according to one or more embodiments of this application; Figure 7 This is a flowchart of another online fine-tuning process in one or more embodiments of this application; Figure 8 This is a schematic diagram of the structure of the construction device for the world model simulation system in another embodiment of this application; Figure 9This is a schematic diagram of the internal structure of a computer device according to one or more embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] This application provides a distillation training method for a world model simulation system, which can be applied to a distillation training system for autonomous driving equipment (hereinafter referred to as the "system"). This autonomous driving equipment can be a vehicle, robot, or drone equipped with LiDAR sensors (hereinafter referred to as "LiDAR") and cameras (image sensors). The world model simulation system is a key component of the world model; specifically, it is a state transition model capable of predicting the future distribution of environmental states based on current environmental data and current policy data.

[0033] In some exemplary embodiments of this application, such as Figure 1 As shown, this method is based on a teacher world model simulation system and a student world model simulation system to be trained. The student world model simulation system has the same structure as the teacher world model simulation system but has fewer parameters. The student world model simulation system includes a decoder and an encoder. The method includes the following steps: Step 101: In the first stage, freeze the decoder, train only the encoder, and transfer the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system.

[0034] The first stage can refer to the "causal knowledge transfer stage," in which all decoding paths of the student world model simulation system are frozen, and only the encoding network parameters can be updated. The decoder can refer to the network part of the student world model simulation system responsible for mapping latent features to a complete sequence of environmental states. The encoder can refer to the network part of the student world model simulation system responsible for compressing multi-source observation data into latent features. The causal expression of environmental evolution can refer to the evolutionary patterns of the future poses and velocities of dynamic obstacles and the static environmental geometry at future moments, captured by the teacher world model simulation system through spatiotemporal attention weights.

[0035] The system can output the dynamic obstacle pose, velocity, and acceleration sequences, as well as the geometric evolution sequence of the static environment (lane lines, passable areas, and road boundaries) for the next 5 seconds, based on continuous image sequences, LiDAR point clouds, high-precision map semantics, and traffic participant interaction maps, through the teacher world model simulation system. Furthermore, the system can reproduce the above causal expression at the encoder end through spatiotemporal attention alignment loss via the student world model simulation system. The loss function is defined as a measure of the difference between the teacher model and the student model in the corresponding attention head outputs.

[0036] For example, if the teacher model's 7th head in layer 4 is sensitive to changes in the speed of traffic flow in the environment, the corresponding head in the student model should reproduce the same sensitivity after training, and the teacher model parameters should be kept frozen throughout the training process.

[0037] Step 102: In the second stage, the student world model simulation system is run on the target chip, the freeze on the student world model simulation system is lifted, the channel pruning of the student world model simulation system is performed using the delay model configured on the target chip, and a quantization noise layer is inserted into the feature map path.

[0038] The second phase can refer to the "hardware adaptation and fine-tuning phase," in which the overall parameters of the student world model simulation system are unfrozen and loaded onto the target chip for simultaneous optimization of inference latency and prediction accuracy. The target chip can refer to an embedded automotive-grade neural network processing unit deployed within a vehicle, robot, or drone. The latency model can refer to a computational model provided by the chip manufacturer for offline evaluation of the theoretical time consumption of each convolutional channel or tensor operation on the target chip. The feature map path can refer to the complete forward data flow of the feature map from input to output as data is transferred between convolutional layers.

[0039] A quantization noise layer is a trainable or fixed-parameter layer inserted into the feature map path to inject numerical perturbations during forward propagation to simulate quantization errors caused by low-bit integer computation. Specifically, the system first unfreezes the entire student world model simulation system and loads it onto the target automotive chip. Next, it offline evaluates the theoretical latency of each convolutional channel using the chip's built-in latency model, sorts the channels by "latency" multiplied by "environmental causality importance," and removes the channels at the bottom of the sort, completing dynamic pruning. After pruning, the system can insert a quantization noise layer into the feature map's forward path to simulate the numerical perturbations introduced by 8-bit integer calculations. Subsequently, it jointly optimizes the dynamic obstacle trajectory loss, static environment geometry loss, and quantization perception distribution loss until the student world model simulation system's forward inference time on the chip meets automotive-grade thresholds and both types of environmental errors are below set tolerances.

[0040] For example, after pruning, the number of channels in the 12th layer was reduced from 256 to 192. After inserting a noise layer, the measured single-frame inference time was reduced from 46 milliseconds to 38 milliseconds. The dynamic obstacle displacement error remained at 0.18 meters, and the static environment IoU (Intersection over Union) error remained at 0.92.

[0041] Step 103: In the third stage, the student world model simulation system is deployed on the terminal main controller, which is configured to collect driving data and record environmental prediction errors.

[0042] When the environmental prediction error or inference delay exceeds the set threshold, the system can trigger an online fine-tuning process and a cloud-based incremental distillation followed by a backhaul update. Inference delay refers to the time required for the student world model simulation system to complete one forward inference on the target chip.

[0043] The third stage can refer to the "terminal closed-loop verification stage." In this stage, the student world model simulation system is deployed to the terminal main control unit. It continuously outputs the future environmental state based on real-time multi-source observation data and simultaneously evaluates the prediction performance. Environmental prediction error refers to the deviation between the student world model simulation system's estimate of the future environmental state and the true value. Specifically, it includes: dynamic obstacle error, used to characterize the pose and speed deviations of vehicles, pedestrians, and cyclists in the future; and static environment error, used to characterize the geometric deviations of lane lines, passable areas, and road boundaries in the future. Forward inference refers to a complete inference process executed by the student world model simulation system on the terminal main control unit, i.e., the calculation process from receiving the current frame of multi-source observation data to outputting the future environmental state sequence.

[0044] The terminal may include an autonomous vehicle, mobile robot, or drone equipped with LiDAR and image sensors, and the terminal master controller may be a computing unit installed on the autonomous vehicle, mobile robot, or drone.

[0045] For example, the system deploys a student world model simulation system on the terminal main control unit, enabling the terminal main control unit to collect multi-source driving data in real time at a frequency of 30 Hz and record environmental prediction errors. These errors include: dynamic obstacle errors, used to characterize the pose and speed deviations of vehicles, pedestrians, and cyclists in the next 5 seconds; and static environmental errors, used to characterize lane line lateral deviations, crossover ratio deviations of passable areas, and road boundary geometric deviations.

[0046] If any error or inference latency (defined as the time required for the model to complete one forward inference on the target chip) exceeds a set threshold, the system's online fine-tuning process can be triggered: the system first caches high-error scene data on the terminal and uploads it to the cloud in encryption. Then, the cloud performs incremental distillation based on memory playback, generates difference patches, and sends them back to the terminal, allowing the terminal to verify that the dynamic obstacle displacement error, static environment cross-comparison error, and inference latency are all stable below the threshold within several consecutive frames, and the closed-loop optimization ends.

[0047] Through steps 101-103, the system achieves a significant improvement in overall latency, accuracy, and hardware compatibility by working together on three steps: causal alignment transfer, chip-aware pruning quantization, and on-vehicle fine-tuning. This ensures that the model meets automotive-grade real-time latency requirements while maintaining high accuracy and compatibility with automotive chips.

[0048] In some embodiments, in the first stage, such as Figure 2 As shown, the steps for transferring the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system include: Step 201: Calculate the spatiotemporal attention causal alignment loss.

[0049] Spatiotemporal attention causal alignment loss refers to the difference in accuracy between the student world model simulation system and the teacher world model simulation system in reproducing the causal dependence of environmental evolution at corresponding attention heads during the causal knowledge transfer stage. Specifically, this loss is achieved by aligning the spatiotemporal correlation weight matrices output by the attention heads of the two models at the same time and at the same layer, and using the matrix difference as a quantitative indicator. This forces the student world model simulation system to reproduce the evolution of the dynamic obstacle's future pose, velocity, and static environment geometry captured by the teacher world model simulation system at the encoder end.

[0050] The system takes the output matrices of the teacher world model simulation system and the student world model simulation system at the same time and at the same layer as the object, and calculates the difference between the two. This difference measure is the spatiotemporal attention causal alignment loss, which is used to quantify the accuracy of the student world model simulation system in reproducing the causal relationship of environmental evolution.

[0051] Step 202: Extract the spatiotemporal correlation weights of all attention heads in the teacher world model simulation system and calculate the environmental causal importance score.

[0052] Among them, the environmental causal importance score represents the degree to which the attention head captures key evolutionary patterns of dynamic obstacles or static environments. The spatiotemporal correlation weight refers to the quantitative value of the strength of correlation between elements at different positions and times in the input feature sequence in the spatiotemporal dimension; this weight is obtained by the query-key-value operation in the attention mechanism and is presented in matrix form. Its magnitude directly represents the model's sensitivity to the evolutionary patterns of dynamic obstacles or static environments at future times.

[0053] The system traverses every layer and every attention point of the teacher world model simulation system and extracts its spatiotemporal correlation weight matrix. Then, based on the sensitivity of this weight matrix to the future pose and velocity changes of dynamic obstacles and the geometric evolution of the static environment, the environmental causal importance score is calculated.

[0054] For example, if a certain attention head has the highest weight energy in the "sudden change in traffic speed" region in the 7th head of the 4th layer of the teacher model, then that head scores 0.92 and is marked as highly sensitive.

[0055] Step 203: Identify the top-scoring attention heads as environmental causal key heads and establish a mapping with the corresponding layer of the student world model simulation system.

[0056] In this context, the environmental causal keyheads refer to the top K attention heads selected from all attention heads in the teacher world model simulation system based on their environmental causal importance scores, ranked from highest to lowest, for transferring the causal expression of environmental evolution. The corresponding layer in the student world model simulation system refers to the attention heads in the student world model simulation system that are at the same network depth and have the same attention mechanism as the aforementioned environmental causal keyheads. The two are structurally mapped one-to-one and undertake the alignment task during the causal knowledge transfer stage.

[0057] The system sorts the environmental causal importance scores in descending order and selects the top K attention heads as environmental causal key heads; then, a one-to-one mapping relationship is established in the same layer of the student world model simulation system, so that subsequent alignment is only performed on this mapping subset.

[0058] For example, if K is 8, then the first 8 high-scoring heads of the 5th layer of the teacher model are mapped to the corresponding positions in the 5th layer of the student model, and the remaining heads do not participate in the alignment.

[0059] Step 204: During training, only the attention weights of the environmental causal keyheads are aligned so that the student world model simulation system can reproduce the environmental evolution causal dependencies of the teacher world model simulation system.

[0060] Among them, the causal dependence of environmental evolution can refer to the "cause-effect" relationship chain revealed by the teacher world model simulation system through spatiotemporal attention weights, that is: the currently observed dynamic obstacle pose / velocity and static environmental geometric features (cause) → the dynamic obstacle pose and velocity changes and static environmental geometric evolution that will inevitably occur in the future (effect).

[0061] During training, the system can first freeze the parameters of the teacher's world model simulation system, while backpropagation only updates the attention weights in the student's world model simulation system that are aligned with the environmental causal keyheads, until the spatiotemporal attention causal alignment loss converges.

[0062] Example: After 50 consecutive iterations, the difference between the weight matrix of the corresponding head in the student world model simulation system and the teacher world model simulation system decreased to 12% of the initial value, indicating that the causal dependence of environmental evolution has been successfully reproduced.

[0063] Through steps 201-204, the system can accurately transfer key environmental evolution causal patterns to the student world model simulation system while maintaining the integrity of the knowledge in the teacher world model simulation system, thus laying a high-precision foundation for subsequent hardware perception compression and terminal closed-loop verification.

[0064] In some embodiments, in the second phase, such as Figure 3 As shown, the steps for channel pruning in a student world model simulation system include: Step 301: Measure the theoretical inference time of each channel offline on the target chip.

[0065] The system calls the latency model provided by the chip manufacturer to evaluate the input size, stride, padding, and number of output channels for each convolutional channel, and obtains the theoretical time required for one complete operation of that channel on the target chip, in milliseconds.

[0066] Step 302: Calculate the pruning index based on the time consumption and environmental causal importance score.

[0067] The pruning index can be defined as follows: Pruning index = Theoretical reasoning time × Gradient importance; Gradient importance is obtained through forward-backward propagation of a mini-batch of data and is used to measure the sensitivity of the channel to prediction errors of dynamic obstacles and static environment; the larger the value, the more significant the impact of the channel on the environmental state prediction task.

[0068] For example, if the theoretical inference time for channel 48 of layer 12 is 0.08 milliseconds and the gradient importance is 1.5, then its pruning index is 0.12; if the theoretical inference time for channel 192 of layer 12 is 0.05 milliseconds and the gradient importance is 0.8, then its pruning index is 0.04.

[0069] Step 303: Remove the channel with the lowest pruning index.

[0070] The pruning index is the product of the theoretical inference time of the channel and the gradient importance of the channel for the environmental state prediction task. The gradient importance is obtained through forward-to-backward propagation of a small batch of data and is used to measure the sensitivity of the channel to prediction errors of dynamic obstacles and static environment.

[0071] Specifically, the system can sort channels by pruning index from smallest to largest, and remove the last channel in the sorting until the total number of remaining channels meets the delay budget of the target chip.

[0072] For example, when the target chip requires a single-frame inference time of no more than 38 milliseconds, after the system removes the 64 channels with the smallest pruning index, the number of remaining channels decreases from 256 to 192. The measured single-frame inference time decreases from 46 milliseconds to 38 milliseconds, the dynamic obstacle displacement error remains at 0.18 meters, and the static environment cross-connection ratio error remains at 0.92.

[0073] Through steps 301-303, the system can maintain the accuracy of environmental prediction while accurately pruning computing resources, providing a low-redundancy, high-time-efficiency network structure for subsequent quantitative sensing distillation.

[0074] In some embodiments, in the second stage, such as Figure 4 As shown, the training step also includes: Step 401: Insert a quantization noise layer into the forward path of the pruned feature map.

[0075] After channel pruning is completed, the system can insert a quantization noise layer between each convolution output feature map and the input of the next layer. This layer applies numerical perturbation to the feature map during forward propagation to simulate the quantization error generated by eight-bit integer calculation, and maintains a trainable or fixed scaling factor and zero offset during backpropagation, thereby approximating the actual numerical distribution of the target chip.

[0076] Step 402: Using the preset training objective as a constraint, jointly optimize the parameters of the quantization noise layer.

[0077] Specifically, the training objectives may include: dynamic obstacle trajectory loss, representing the future pose and velocity errors based on dynamic obstacles; static environment geometry loss, representing the errors based on lane lines and passable areas; and quantization-perceptual distribution loss, representing the difference between the outputs of the teacher's world model simulation system and the student's world model simulation system. In each training iteration, the system can first fix the quantization noise layer parameters and update the weights of the student's world model simulation system to reduce the weighted sum of the three losses. Then, it can fix the weights of the student's world model simulation system and update the quantization noise layer parameters to minimize the quantization-perceptual distribution loss. This process is then repeated alternately until all three losses converge and the inference time per frame does not exceed thirty-eight milliseconds.

[0078] For example, after 150 alternating iterations, the dynamic obstacle displacement error decreased from 0.22 meters to 0.18 meters, the static environment cross-comparison ratio error increased from 0.88 to 0.92, and the quantization perception distribution loss decreased to 15% of the initial value, meeting the requirements for automotive-grade deployment.

[0079] Through steps 401-402, the system can achieve numerical bit width compression and deep alignment of hardware numerical characteristics while maintaining the accuracy of environmental prediction, providing a low-latency, highly robust, lightweight world model simulation system for terminal closed-loop verification.

[0080] In some embodiments, such as Figure 5 As shown, the optimization steps for quantizing the distribution loss include: Step 501: Output environmental status soft labels through the teacher world model simulation system.

[0081] Among them, the environmental status soft label can include the future soft distribution of dynamic obstacles and static environment.

[0082] For example, for the same frame input, the variance of the soft distribution of dynamic obstacle speed output by the teacher model is 0.05 meters per second squared, and the variance of the soft distribution of lane lines in the static environment is 0.02 meters. Together, they constitute the soft label.

[0083] Step 502: Output the corresponding soft distribution after quantization noise layer processing through the student world model simulation system, and perform alternating optimization: In the first round, freeze the quantization noise layer and update the network weights of the student world model simulation system. In the second round, freeze the network weights of the student world model simulation system and update the parameters of the quantization noise layer. Repeat the above alternating optimization until the soft distribution differences between dynamic obstacles and static environment converge.

[0084] Specifically, in the first round, the system freezes the scaling factor and zero-point offset of the quantization noise layer and updates the network weights of the student world model simulation system only through backpropagation, thereby reducing the negative log-likelihood loss of the soft distributions of dynamic obstacles and static environments. In the second round, all weights of the student world model simulation system are frozen, and the parameters of the quantization noise layer are updated only through gradient descent to minimize the Kourbach-Leibler divergence between the teacher and student soft distributions. The system repeats the above alternating iterations until the difference in the soft distribution of dynamic obstacles drops below 0.03 and the difference in the soft distribution of the static environment drops below 0.01, with the change being less than one-thousandth for ten consecutive iterations.

[0085] For example, after 120 alternating iterations, the Kourbach-Leibler divergence between the dynamic obstacle velocity distribution output by the student world model simulation system and the teacher soft label decreased from the initial 0.18 to 0.02, and the static environment lane line distribution divergence decreased from 0.09 to 0.008, satisfying the convergence condition.

[0086] Through steps 501-502, the system can bring the teacher-student distribution error to an acceptable range while maintaining the low-bit representation of the numerical values, thereby ensuring that the quantized student world model simulation system still has the same environmental prediction capability as the teacher model on the target chip.

[0087] In some embodiments, in the third stage, such as Figure 6 As shown, the online fine-tuning process includes...

[0088] Step 601: Deploy the student world model simulation system on the terminal main control, receive input from the camera, millimeter-wave radar, inertial measurement unit and high-precision map in real time, and output the future pose and velocity sequence of dynamic obstacles, the future lane lines and passable area sequence of static environment frame by frame.

[0089] The future pose and velocity sequence of the obstacle can refer to the continuous numerical sequence of lateral displacement, longitudinal displacement, lateral velocity, and longitudinal velocity of vehicles, pedestrians, and cyclists within the next five seconds, with each frame interval of 0.2 seconds, for a total of twenty-five sets of data. The future lane line and passable area sequence of the static environment can refer to the continuous numerical sequence of lateral coordinates, curvature of the lane line, and vertex coordinates of the passable area outline within the next five seconds, with each frame interval of 0.2 seconds, for a total of twenty-five sets of data.

[0090] For example, the system burns the pruned and quantized student world model simulation system to the terminal main controller, which is integrated into the onboard computing unit of an autonomous vehicle, mobile robot, or drone. This main controller receives camera images, millimeter-wave radar point clouds, inertial measurement unit attitude data, and high-precision map semantics in real time at a frequency of 30 Hz, and outputs them frame by frame. The dynamic obstacle future pose and velocity sequence is used to characterize the lateral displacement, longitudinal displacement, lateral velocity, and longitudinal velocity of vehicles, pedestrians, and cyclists in the following five seconds. The static environment future sequence is used to characterize the geometric evolution of the lane line lateral coordinates and the outline of the passable area in the following five seconds.

[0091] Step 602: Compare the terminal output with the true value, and calculate the average displacement error, velocity error and static environment geometric error of the dynamic obstacle respectively.

[0092] The "true value" refers to the dynamic obstacle pose, velocity, and static environment geometry data, which are collected and fused in real time by vehicle-mounted LiDAR, millimeter-wave radar, combined inertial navigation, and high-precision maps, serving as the comparison benchmark. The average displacement error of the dynamic obstacle refers to the Euclidean distance between each predicted pose and the corresponding true pose of the dynamic obstacle output by the student world model simulation system within the next five seconds, averaged over all obstacles at all time steps, in meters. The velocity error of the dynamic obstacle refers to the absolute difference between each predicted velocity and the corresponding true velocity of the dynamic obstacle output by the student world model simulation system within the next five seconds, averaged over all obstacles at all time steps, in meters per second. The static environment geometry error refers to the average absolute error between the lateral coordinates of the lane lines and the true lateral coordinates output by the student world model simulation system within the next five seconds, and the intersection-over-union ratio (IoU) error between the traversable area contour and the true contour; both are combined and denoted as the static environment geometry error. The lateral error is in meters, and the IoU is a dimensionless value.

[0093] Specifically, when the terminal controller compares the above output with the true value, it may involve: The average displacement error of dynamic obstacles is used to characterize the average Euclidean distance between the predicted trajectory and the true trajectory of the lidar over the next five seconds. The velocity error of dynamic obstacles is used to characterize the average absolute value of the difference between the predicted velocity and the true velocity of the combined inertial navigation system. The geometric error of the static environment is used to characterize the average absolute value of the lane line lateral error and the intersection-combination ratio error of the passable area.

[0094] For example, if the displacement error of a dynamic obstacle in a certain frame is 0.21 meters, the speed error is 0.3 meters per second, the lateral error of the lane line is 0.07 meters, and the intersection ratio error of the passable area is 0.05, then all four errors are recorded.

[0095] Step 603: When any error or inference delay exceeds a set threshold, perform the following steps: The original data in high-error scenarios is cached on the terminal and then uploaded to the cloud via an encrypted link.

[0096] The cloud performs incremental distillation based on cached data to update the student world model simulation system.

[0097] The updated student world model simulation system is transmitted back to the terminal device and hot-swapped using over-the-air download technology.

[0098] Specifically, when any error or inference delay (defined as the time required for the model to complete one forward inference on the target chip) exceeds a set threshold (examples: dynamic obstacle displacement error 0.25 meters, speed error 0.35 meters per second, lane line lateral error 0.1 meters, passable area intersection-over-comparison error 0.08, inference delay 50 milliseconds), the system can execute: Terminal caching packages all raw sensor data and truth labels corresponding to high-error scenarios into a cache package; Encrypted upload: The data is uploaded to the cloud training node in an end-to-end encrypted manner via cellular vehicle-to-everything (V2X) or satellite link. Incremental distillation in the cloud involves mixing cached data with historical data proportionally based on a memory replay strategy, performing fifty rounds of incremental distillation, updating only the weight increments of the student world model simulation system, and generating difference patches. Over-the-air download and hot replacement: after differential compression, the difference patch is sent back to the terminal via over-the-air download technology. After the terminal verifies that the hash consistency is correct, it updates the model by hot replacement. After verifying that the four errors and inference latency are all stable below the threshold within 20 consecutive frames, the closed-loop fine-tuning process ends.

[0099] Through steps 601 to 603, the system can continuously self-evolve during the operation of vehicles, robots, or drones, ensuring both the accuracy of environmental prediction and maintaining automotive-grade real-time performance.

[0100] In some embodiments, such as Figure 7 As shown, the online fine-tuning process also includes: Step 701: Set up a circular cache on the terminal to store high environmental error data from the most recent period.

[0101] The circular buffer refers to a fixed-capacity storage area allocated locally by the terminal controller, which is cyclically overwritten in chronological order to temporarily store sensor data packets within the most recent time period; for example, a capacity of 256 megabytes can store data from the most recent ten minutes. High environmental error data refers to the complete data packet corresponding to the situation where, after the terminal controller compares the model output with the true value in real time, the dynamic obstacle displacement error exceeds 0.25 meters or the static environmental geometric error exceeds a set threshold. This includes synchronized image sequences, LiDAR point clouds, millimeter-wave radar data, inertial measurement unit data, and corresponding high-precision map tiles.

[0102] The terminal controller opens a fixed-capacity circular buffer locally and stores the original data packets that meet the "high environmental error" criteria in the most recent time period in chronological order. Each packet contains synchronized image sequences, LiDAR point clouds, millimeter-wave radar data, inertial measurement unit data, and corresponding high-precision map tiles.

[0103] For example, the buffer capacity is set to 256 megabytes, which can store high-error scenarios from the most recent ten minutes.

[0104] Step 702: After the cache is full, the data is encrypted and uploaded to the cloud via cellular vehicle-to-everything (V2X) communication. The cloud uses a memory playback mechanism to mix the new cached data with historical data for training, so as to maintain the memory of key patterns of dynamic obstacles and static environment.

[0105] Cellular vehicle-to-everything (V2X) refers to a vehicle-to-everything (V2X) wireless communication system based on fourth- or fifth-generation cellular mobile communication technology. It establishes an encrypted data channel between the terminal and the cloud, enabling remote uploading of high-environment-error data and over-the-air distribution of differential patches. The memory-playback mechanism refers to the cloud's practice of mixing newly cached high-environment-error data with historical data at a preset ratio during incremental distillation training, and then re-feeding it to the network. This prevents the student world model simulation system from forgetting previously learned key patterns such as dynamic obstacle sudden stops, cutting-in, and static environment mutations.

[0106] When the remaining space of the ring buffer is less than 10%, the terminal uploads the complete cache package to the cloud training node through the cellular vehicle network in an end-to-end encrypted manner. The cloud adopts a memory playback mechanism to mix the new cache data with the cloud historical data in a 1:3 ratio to form an incremental training set, so as to maintain long-term memory of key patterns such as sudden stops of dynamic obstacles, cutting in, and lane line changes in static environments.

[0107] Step 703: After incremental training, generate differential patches, and distribute them to the terminal after differential compression.

[0108] Incremental training refers to the cloud-based retraining of the student world model simulation system using only newly uploaded high-environment-error data and a small amount of historical data to correct recently emerging prediction biases, without retraining all parameters. Differential patching refers to the process where, after incremental training is completed in the cloud, only the difference matrix between the current and previous weights of the student world model simulation system is extracted, and differentially compressed to form an incremental file. The terminal controller then obtains this file via over-the-air download and adds it to the current weights, achieving hot updating of the model.

[0109] The cloud performs fifty rounds of incremental distillation based on the incremental training set, only calculating the difference matrix of the student world model simulation system weights relative to the weights of the previous round, and generating a difference patch; after differential compression, the size of the patch is reduced to 15% of the original weights, and then it is downloaded to the terminal main control via over-the-air download technology.

[0110] Step 704: When the dynamic obstacle displacement error, static environment geometric error and inference delay all meet the set threshold within several consecutive frames of the terminal, the closed-loop optimization ends.

[0111] For example, after receiving a difference patch, the terminal controller completes hash verification and hot replacement, and then monitors in real time for twenty consecutive frames: The displacement error of dynamic obstacles shall not exceed 0.25 meters; Static environment geometric errors (lateral lane line error not greater than 0.1 meters, intersection ratio error of passable area not greater than 0.08). Inference delay is no more than fifty milliseconds.

[0112] Once all three indicators stably meet the threshold, the terminal controller sends a closed-loop completion flag to the cloud, ending this round of optimization.

[0113] Through steps 701 to 704, the system can continuously self-evolve during operation, avoiding catastrophic forgetting while ensuring automotive-grade real-time performance and prediction accuracy.

[0114] This application provides a construction apparatus for a world model simulation system according to the second aspect, such as Figure 8 As shown, the device is based on a teacher world model simulation system and a student world model simulation system to be trained. The student world model simulation system has the same structure as the teacher world model simulation system but has fewer parameters. The student world model simulation system includes a decoder and an encoder. The device includes: The first-stage training module 110 is used to freeze the decoder and train only the encoder, transferring the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system.

[0115] The second-stage training module 120 is used to run the student world model simulation system on the target chip, unfreeze the student world model simulation system, use the delay model configured on the target chip to perform channel pruning on the student world model simulation system, and insert a quantization noise layer into the feature map path.

[0116] The third-stage training module 130 is used to deploy the student world model simulation system on the terminal main control, enabling the configured terminal main control to collect driving data and record environmental prediction errors.

[0117] The backhaul update module 140 is used to trigger the online fine-tuning process and cloud incremental distillation backhaul update when the environmental prediction error or inference delay exceeds the set threshold. Inference delay refers to the time required for the student world model simulation system to complete one forward inference on the target chip.

[0118] During each training session, the world model simulation system training module 130 inputs multi-source observation data related to the training samples used in this training into the world model simulation system, determines the path difference based on the optimized trajectory information generated by the learnable optimization layer and the environmental observation information generated by the generator network, and updates the parameters of the target network based on the path difference; the target network is either the generator network or the learnable optimization layer.

[0119] In some embodiments, the first-stage training module 110 is further configured to: Calculate the spatiotemporal attention causal alignment loss; extract the spatiotemporal correlation weights of all attention heads in the teacher world model simulation system, and calculate the environmental causal importance score, which represents the degree to which the attention head captures key evolution patterns of dynamic obstacles or static environment; identify the attention heads with the highest scores as environmental causal key heads and establish a mapping with the corresponding layer of the student world model simulation system; during training, only align the attention weights of the environmental causal key heads to enable the student world model simulation system to reproduce the environmental evolution causal dependencies of the teacher world model simulation system.

[0120] In some embodiments, the second-stage training module 120 is further configured to: The theoretical inference time for each channel was measured offline on the target chip; the pruning index was calculated based on the time and the environmental causal importance score; and the channel with the smallest pruning index was removed.

[0121] In some embodiments, the second-stage training module 120 is further configured to: A quantization noise layer is inserted into the feedforward path of the pruned feature map; the parameters of the quantization noise layer are jointly optimized under the constraint of a preset training objective.

[0122] The training objectives include: dynamic obstacle trajectory loss, which represents the future pose and velocity error based on dynamic obstacles; static environment geometry loss, which represents the error based on lane lines and passable areas; and quantized perception distribution loss, which represents the difference between the outputs of the teacher world model simulation system and the student world model simulation system.

[0123] In some embodiments, the second-stage training module 120 is further configured to: The teacher world model simulation system outputs soft labels for the environment state, which include the future soft distribution of dynamic obstacles and static environment. The student world model simulation system outputs the corresponding soft distribution after processing with a quantized noise layer, and performs alternating optimization: In the first round, the quantized noise layer is frozen and the network weights of the student world model simulation system are updated; in the second round, the network weights of the student world model simulation system are frozen and the parameters of the quantized noise layer are updated; the above alternating optimization is repeated until the difference in the soft distribution of dynamic obstacles and static environment converges.

[0124] In some embodiments, the third-stage training module 130 is further configured to deploy the student world model simulation system on the terminal main control, receive input from cameras, millimeter-wave radar, inertial measurement units, and high-precision maps in real time, and output the future pose and velocity sequences of dynamic obstacles and the future lane lines and passable areas of the static environment frame by frame; compare the terminal output with the true values, and calculate the average displacement error, velocity error of the dynamic obstacles, and geometric error of the static environment respectively; when any error or inference delay exceeds a set threshold, perform the following: cache the original data of the high-error scene on the terminal; upload it to the cloud via an encrypted link; perform incremental distillation on the cached data in the cloud to update the student world model simulation system; and send the updated student world model simulation system back to the terminal device for hot replacement via over-the-air download technology.

[0125] In some embodiments, the third-stage training module 130 is further configured to set up a circular cache on the terminal to store high environmental error data from the most recent time period; after the cache is full, the data is encrypted and uploaded to the cloud via cellular vehicle-to-everything (V2X) communication; the cloud uses a memory playback mechanism to mix the new cached data with historical data for training, so as to maintain the memory of key patterns of dynamic obstacles and static environment; after incremental training, a differential patch is generated and distributed to the terminal after differential compression; when the dynamic obstacle displacement error, static environment geometric error and inference delay all meet the set threshold within several consecutive frames on the terminal, the closed-loop optimization ends.

[0126] Specific limitations regarding the distillation training method applicable to world model simulation systems can be found in the above section on limitations for the distillation training method applicable to world model simulation systems, and will not be repeated here. Each module in the above-mentioned distillation training method applicable to world model simulation systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0127] According to a third aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the distillation training method for any of the world model simulation systems described above.

[0128] According to a fourth aspect, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps of the distillation training method of any of the world model simulation systems described above.

[0129] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data related to the distillation training of the world model simulation system. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the distillation training methods of the world model simulation system described above.

[0130] Any references to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (RAMCUs), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0133] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

Claims

1. A distillation training method for a world model simulation system, characterized in that, The method is based on a teacher world model simulation system and a student world model simulation system to be trained. The student world model simulation system has the same structure as the teacher world model simulation system but has fewer parameters. The student world model simulation system includes a decoder and an encoder. The method includes: In the first stage, the decoder is frozen, and only the encoder is trained to transfer the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system. In the second stage, the student world model simulation system is run on the target chip, the freeze on the student world model simulation system is lifted, and the channel pruning of the student world model simulation system is performed using the delay model configured on the target chip, and a quantization noise layer is inserted into the feature map path. In the third stage, the student world model simulation system is deployed on the terminal main control, which collects driving data and records environmental prediction errors. In response to environmental prediction errors or inference delays exceeding a set threshold, an online fine-tuning process and cloud incremental distillation are triggered, followed by a data transmission update. The inference delay refers to the time required for the student world model simulation system to complete one forward inference on the target chip.

2. The method according to claim 1, characterized in that, In the first stage, the step of transferring the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system includes: Calculate the spatiotemporal attention causal alignment loss; The spatiotemporal correlation weights of all attention heads in the teacher world model simulation system are extracted, and the environmental causal importance score is calculated. The environmental causal importance score represents the degree to which the attention head captures key evolution patterns of dynamic obstacles or static environment. The attention heads with the highest scores are identified as key environmental causal heads, and a mapping is established with the corresponding layer of the student world model simulation system. During training, attention weights are aligned only for environmental causal keyheads, enabling the student world model simulation system to reproduce the environmental evolution causal dependencies of the teacher world model simulation system.

3. The method according to claim 2, characterized in that, In the second stage, the process of channel pruning in the student world model simulation system includes: The theoretical inference time for each channel was measured offline on the target chip; The pruning index is calculated based on the time consumption and the causal importance score of the environment; Remove the channel with the lowest pruning index; The pruning index is the product of the theoretical inference time of the channel and the gradient importance of the channel for the environmental state prediction task. The gradient importance is obtained through forward-to-backward propagation of a small batch of data and is used to measure the sensitivity of the channel to prediction errors of dynamic obstacles and static environment.

4. The method according to claim 3, characterized in that, In the second stage, the method further includes: A quantization noise layer is inserted into the forward path of the pruned feature map; The parameters of the quantization noise layer are jointly optimized under the constraint of a preset training objective. The training objectives include: Dynamic obstacle trajectory loss characterizes the future pose and velocity error based on dynamic obstacles; Static environment geometric loss is characterized by errors based on lane lines and traversable areas. The quantified perceptual distribution loss characterizes the difference between the outputs of the teacher world model simulation system and the student world model simulation system.

5. The method according to claim 4, characterized in that, The optimization steps for the quantized sensing distribution loss include: The teacher world model simulation system outputs environmental state soft labels, which include the future soft distribution of dynamic obstacles and static environment. The student world model simulation system outputs the corresponding soft distribution after processing by the quantization noise layer, and performs alternating optimization: In the first round, the quantized noise layer is frozen, and the network weights of the student world model simulation system are updated. In the second round, the network weights of the student world model simulation system are frozen, and the parameters of the quantized noise layer are updated. Repeat the above alternating optimization until the soft distribution differences between dynamic obstacles and static environment converge.

6. The method according to claim 5, characterized in that, In the third phase, the online fine-tuning process includes: The student world model simulation system is deployed on the terminal main control, receiving inputs from cameras, millimeter-wave radar, inertial measurement units and high-precision maps in real time, and outputting the future pose and velocity sequence of dynamic obstacles, the future lane lines and passable area sequence of static environment frame by frame; The terminal output is compared with the true value, and the average displacement error, velocity error and static environment geometric error of the dynamic obstacle are calculated respectively. When any error or inference delay exceeds a set threshold, execute: In terminal caching, raw data is used in high-error scenarios. Uploaded to the cloud via an encrypted link; The cloud performs incremental distillation based on cached data to update the student world model simulation system; The updated student world model simulation system is transmitted back to the terminal device and hot-swapped using over-the-air download technology.

7. The method according to claim 6, characterized in that, The online fine-tuning process also includes: A circular cache is set up on the terminal to store high environmental error data from the most recent period; Once the cache is full, the data is encrypted and uploaded to the cloud via cellular vehicle-to-everything (V2X) connectivity. The cloud employs a memory-replay mechanism, mixing new cached data with historical data for training, in order to maintain the memory of key patterns of dynamic obstacles and static environments; After incremental training, differential patches are generated and distributed to the terminal after differential compression. The closed-loop optimization ends when the dynamic obstacle displacement error, static environment geometric error, and inference delay all meet the set thresholds within several consecutive frames of the terminal.

8. A distillation training device for a world model simulation system, characterized in that, include: A student world model simulation system for training, wherein the student world model simulation system has the same structure as the teacher world model simulation system but has fewer parameters than the teacher world model simulation system, and the student world model simulation system includes a decoder and an encoder; the device includes: The first-stage training module is used to freeze the decoder and train only the encoder to transfer the causal expression of environmental evolution from the teacher world model simulation system to the student world model simulation system. The second-stage training module is used to run the student world model simulation system on the target chip, unfreeze the student world model simulation system, use the delay model configured on the target chip to perform channel pruning on the student world model simulation system and insert a quantization noise layer into the feature map path; The third-stage training module is used to deploy the student world model simulation system on the terminal main control, enabling the configured terminal main control to collect driving data and record environmental prediction errors; The back-up update module is used to trigger the online fine-tuning process and cloud incremental distillation and back-up update when the environmental prediction error or inference delay exceeds a set threshold. The inference delay refers to the time required for the student world model simulation system to complete one forward inference on the target chip.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images