High-fidelity lightweight world model construction method for end-to-end autonomous driving test

Abstract

The application relates to a world model construction method, in particular to an end-to-end automatic driving test high-fidelity lightweight world model construction method, which constructs a high-fidelity world model, solves the problems of large world model parameters and low inference efficiency, performs knowledge distillation on the world model, reduces model parameters on the basis of retaining world model generation capacity, and improves inference efficiency; a CUDA operator is self-defined for a world model calculation bottleneck part, memory allocation is optimized, and a single-device multi-thread scheduling and multi-device cooperative calculation method are used to improve the inference efficiency of the high-fidelity world model. The application can construct an end-to-end automatic driving test high-fidelity lightweight world model, effectively solve the problems of low multi-modal information alignment accuracy, poor cross-view and cross-frame consistency, and low inference efficiency of the existing world model, improve the confidence of an end-to-end automatic driving system test process, and greatly accelerate the test efficiency of the end-to-end automatic driving system.

Description

TECHNICAL FIELD

[0001] The application relates to a world model construction method, in particular to a high-fidelity lightweight world model construction method for end-to-end automatic driving test. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the end-to-end automatic driving architecture has become the mainstream development direction in the field of automatic driving. It takes multi-view images as input and directly outputs bottom control signals, effectively reduces the cumulative error of the modular automatic driving paradigm, and significantly enhances the ability to cope with complex scenes in the real world. However, the driving scenes in the real world are endless and rich, and sufficient and reliable testing is a necessary prerequisite for the industrialization of end-to-end automatic driving vehicles.

[0003] The change of the automatic driving architecture paradigm makes the traditional modular testing method relying on single modal information no longer applicable. Existing research uses a world model to expand the modalities of the test scene and uses it in the testing process of the end-to-end automatic driving system. However, the world model constructed by existing researches faces the challenge of difficulty in accurately aligning multi-modal scene information, which is specifically manifested in the following aspects: poor alignment accuracy of trajectory information and multi-view image information, strong randomness of vehicle generation caused by difficulty in mapping vehicle pose, and violation of real-world laws caused by difficulty in analyzing instance depth; at the same time, the world model of existing researches faces the challenge of poor cross-view and cross-frame consistency, and there are obvious mutations in key information between multi-views and time-series frames; in addition, the world model has a large amount of operation parameters and a long inference time, which seriously restricts the improvement of the testing efficiency of the end-to-end automatic driving system. SUMMARY

[0004] In order to solve the above technical problems, the application provides a high-fidelity lightweight world model construction method for end-to-end automatic driving test, which comprises the following steps:

[0005] Step 1, constructing a high-fidelity world model;

[0006] A diffusion model is used as the basis to construct a world model basic skeleton. On the basis of the world model basic skeleton, a conditional control module is constructed to realize controllable generation of multi-view images of the world model. An instance control module and a multi-view and time-series frame control module are integrated, instance depth and 3D masks are introduced, instance features are extracted combined with instance attributes, cross-view consistency loss functions and cross-frame consistency loss functions are constructed, and a world model training loss function is designed to construct a high-fidelity world model. Figure One Further, the specific steps are as follows:

[0007] Step 1.1, constructing a world model basic skeleton;

[0008] A diffusion model is used to construct a basic skeleton of the world model, which includes a forward diffusion process and a backward denoising process. In the forward diffusion process, Gaussian noise is gradually added to the original data, as shown in equation (1):

[0009] (1)

[0010] In the equation, N represents a Gaussian distribution; q(z t |z t-1 ) represents a transition probability distribution in the forward diffusion process; z t represents a latent variable; I represents a unit covariance matrix; β t represents a noise variance at the t-th time step; the latent variable z t at any time step is calculated according to the original sample x0; and the marginal distribution is shown in equation (2):

[0011] (2)

[0012] In the equation, q(z t |x0) represents a marginal distribution in the forward diffusion process, α t represents a single-step signal retention coefficient, and α t =1-β t .

[0013] In the backward denoising process, a neural network is used to generate the backward conditional distribution q(z t-1 |z t ) at each step, to gradually denoise and reconstruct the original sample. A denoising neural network ε θ (z t ,t) is trained to predict the Gaussian noise ε introduced in the forward diffusion process, and then the predicted mean is obtained, as shown in equation (3):

[0014] (3)

[0015] In the equation, μ θ (z t ,t) represents the predicted mean, which is used to construct the inverse distribution p θ (z t-1 |z t ) for sampling, as shown in equation (4):

[0016] (4)

[0017] In the equation, C(z ) represents a covariance control term of the distribution, and the latent variable z , .

[0018] Step 1.2, constructing a conditional control module;

[0019] Integrate the conditional control module on the basic skeleton of the world model, define six categories of conditional control signals that can be integrated into the world model skeleton, which are scene 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt prom, road layout L road , reference frame X ref , and express the six categories of conditional control signals as comprehensive conditional variables c as shown in equation (5):

[0020] (5)

[0021] Build encoders for the six categories of conditional control signals to obtain the encoded six categories of conditional control information as shown in equation (6):

[0022] (6)

[0023] In the equation, e 3D , e pose , e cam , e prom , e road , e ref are the encoders of 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt prom, road layout L road , reference frame X ref , respectively; E 3D , E pose , E cam , E prom , E road , E ref are the encoded conditional control information of 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt prom, road layout L road , reference frame X ref , respectively.

[0024] Normalize and fuse the conditional control information as shown in equation (7):

[0025] (7)

[0026] In the equation, e c represents the aggregated comprehensive conditional control information; f represents the conditional control information aggregator.

[0027] Further, the comprehensive conditional control information e cThe conditional control information fusion is realized by using FiLM modulation in the world model skeleton network, as shown in equation (8).

[0028] (8)

[0029] In the formula, h l represents the feature representation after the condition control information is integrated; h l represents the intermediate feature of the lth layer in the world model skeleton; γ l represents the channel scaling and offset functions, respectively; when the model is trained, the modulation structure is embedded into the diffusion model denoising network, and learning is performed by minimizing the diffusion reconstruction loss under the control condition, and the diffusion reconstruction loss function L cond As shown in equation (9):

[0030] (9)

[0031] In the formula, represents the expected calculation operator, and ζ represents the Gaussian sampling noise;

[0032] Step 1.3, constructing an instance control module;

[0033] To realize accurate control of instances and avoid the world model from generating multi-view images that violate the laws of reality, an instance depth D (k) and a 3D mask M (k) ∈0,1 H×W are introduced for the instance object, combined with the instance attribute c (k) , and the instance features are extracted through multiple encoders, as shown in equation (10):

[0034] (10)

[0035] In the formula, z (k) represents the fusion feature vector of the kth instance, E inst , E D , and E M represent the instance attribute feature encoder, the instance depth feature encoder, and the 3D mask feature encoder, respectively; represents the fusion mapping function; to enhance the training stability and improve the model's perception ability of instance pose and depth, an instance loss function L inst is introduced, as shown in equation (11):

[0036] (11)

[0037] In the formula, , represent the model prediction results of the 3D mask and the instance depth; λ m, λ d represents the loss weights for 3D mask and instance depth; BCE represents the binary cross-entropy loss; K represents the set of instance objects;

[0038] Step 1.4: Construct the multi-view and time sequence frame control module;

[0039] At the multi-view level, let the camera matrix under the i-th view be P. i =K i [R i |t i ], K i R represents the camera's intrinsic parameters. i t i These represent the rotation matrix and translation vector in the camera extrinsic parameters, respectively. To enhance the modeling ability of the same scene instance from different perspectives, a cross-view attention mechanism is introduced into the world model network structure. By aligning instance regions under different perspectives through geometric relationships, the network is guided to fuse contextual information from other views at the feature level, thereby improving the consistency and physical alignability of images under different camera perspectives.

[0040] At the continuous frame level, a cross-frame attention mechanism is introduced into the world model network structure to model the dynamic evolution of instances between consecutive frames. Let the inter-frame vehicle motion be δ, and align the features based on geometric transformation. Map the feature map at time t+1 to the coordinate system at time t. Combined with the attention mechanism, information fusion between consecutive frames is achieved, improving the temporal consistency and dynamic stability of the generated image sequence.

[0041] A supervised loss based on instance representation is introduced to constrain the consistency of the model's generated results across different views and time frames. For instance k at each time t and each view i, the consistency is achieved by masking the instance representation. Applied to feature maps Extract the instance vector representation, as shown in equation (12):

[0042] (12)

[0043] In the formula, Let represent the semantic vector representation of instance k from perspective i at time t, and AugPool represents average pooling.

[0044] Building cross-view Figure One The consistency loss function and the cross-frame consistency loss function are used to measure the consistency of instances among multiple views at the same time and the consistency of instances among multiple views at consecutive times, as shown in equations (13) and (14), respectively:

[0045] (13)

[0046] (14)

[0047] In the formula, L view L temp They represent cross-view Figure One Consistency loss function and cross-frame consistency loss function; cos represents cosine similarity; Let represent the set of instances viewed from the i-th camera at time t.

[0048] Step 1.5: Construct the world model training loss function;

[0049] The complete training loss function for constructing the world model is shown in equation (15):

[0050] (15)

[0051] In the formula, L total Indicates total loss; L cond L represents the diffusion reconstruction loss function; inst L represents the instance loss function; view Indicates cross-view Figure One Consistency loss function; L temp Denotes the cross-frame consistency loss function; λ cond , λ inst , λ view , λ temp These represent the weighting coefficients for each type of loss.

[0052] Step 2: Perform knowledge distillation on the world model;

[0053] A basic knowledge distillation architecture of "teacher model-student model" is constructed, a knowledge distillation loss function is designed, the multi-view images output by the student model are aligned with the output of the teacher model, and the high-fidelity world model in step 1 is lightweighted to reduce the complexity of the world model and the inference overhead. Further, the specific steps are as follows:

[0054] Step 2.1: Construct the basic framework for knowledge distillation;

[0055] A knowledge distillation architecture of "teacher model-student model" is constructed. The teacher model T is a high-fidelity world model trained in step 1 with fixed parameters, which is used only for forward reasoning. The student model S is a lightweight world model with the same interface as the teacher model T but with a smaller parameter size. Its parameters can be trained and continuously learn and approximate the output performance of the teacher model T under the same input conditions during the knowledge distillation process.

[0056] In the distillation process, the input information is standardized to a unified numerical precision, tensor dimension and data type to ensure consistency between the teacher model T and the student model S. During the training process, the input data is sampled, encoded and normalized, and a random variable is introduced in the diffusion modeling. The input data is randomly transmitted to the teacher model T and the student model S. The teacher model T and the student model S are consistent in structure and use the same control module combination to ensure that the distillation training is performed under the premise of complete alignment.

[0057] To achieve efficient training of multi-camera perspectives, a batch perspective merging strategy is used, which concatenates the original batch dimension and perspective dimension. The original structure is restored in the output stage to ensure that the model prediction output and the supervision signal are strictly corresponding in the sample dimension and perspective dimension.

[0058] Step 2.2, design the knowledge distillation loss function;

[0059] After building the basic architecture of knowledge distillation, the output of the teacher model T is used as the supervision target, and the prediction output of the student model S is aligned with the supervision target.

[0060] Further, the teacher model T samples Gaussian noise for each sample and multi-perspective image at time t, and uses it as the supervision target. Under the same input conditions, the student model S outputs the prediction tensor The distillation loss is calculated by mean square error, and the sample and perspective dimensions are averaged to obtain the distillation loss L KD (t) as shown in equation (16):

[0061] (16)

[0062] In the formula, represents the expected calculation operator, represents the prediction output of the student model, and η represents the output of the teacher model;

[0063] When the batch size is B and the number of perspectives is N cam , equation (16) is equivalent to the finite sum form shown in equation (17):

[0064] (17)

[0065] In the formula, b represents the sample index within the batch B, n represents the camera perspective index, represents the prediction output of the student model, represents the output of the teacher model, which is the supervision target of the student model;

[0066] To consider the importance and training stability of different time steps, without changing the input conditions, each time loss is given a non-negative weight wt The weighted total loss shown in equation (18) is obtained. :

[0067] (18)

[0068] Step 3: Develop custom CUDA operators;

[0069] To address the bottleneck in the world model that consumes the most computational resources, custom CUDA operators were developed, and key computational kernels were rewritten. Cascaded computations of two or more consecutive operators were merged into a single CUDA kernel function to fully leverage the parallel computing performance of the GPU. Simultaneously, thread block partitioning and shared memory usage were optimized. A dedicated CUDA kernel was written for multi-view projection computations. A mixed-precision computing strategy was introduced, utilizing memory reuse and timely release techniques to optimize GPU memory usage logic. Further, the specific steps are as follows:

[0070] Step 3.1: Identify the computational bottlenecks in the world model;

[0071] The operation flow of the world model is analyzed, the complexity of each module of the world model is analyzed and the actual runtime is calculated, and each sub-module of the world model is defined as a set o. j Execute the complete inference process of the world model and record the running time T of each submodule. j The module with the longest execution time The module is considered the main computational bottleneck in the world model. At the same time, the peak memory usage of each module is evaluated to identify potential memory bottlenecks.

[0072] Step 3.2: Develop CUDA operators for the identified computational bottlenecks;

[0073] For identified computational bottleneck modules, custom CUDA operators are used to rewrite key computational kernels to fully leverage the parallel computing performance of GPUs. The cascaded computations of two or more consecutive operators are merged into a single CUDA kernel function, avoiding repeated writing and reading of intermediate results into and from GPU memory. During large convolution operations, the thread block partitioning and shared memory usage are optimized, enabling multiple computation steps to be completed in a single traversal. In multi-view projection computation, a dedicated CUDA kernel is written to merge geometric transformations and interpolation into a single parallel computation, replacing pixel-by-pixel serial operations.

[0074] Step 3.3: Optimize video memory usage logic;

[0075] While improving the computational efficiency of the world model bottleneck module, the model memory occupancy is optimized through various means. A mixed precision calculation strategy is introduced to compress the model weights from 32-bit floating point numbers to 16-bit floating point numbers for storage and operation. On the premise that the model accuracy is almost not affected, the storage space is greatly saved. Memory reuse and timely release techniques are used to reduce redundant intermediate results. For intermediate tensors generated during the gradual calculation of the world model that are no longer needed subsequently, they are released. For intermediate results that need to be calculated repeatedly, some intermediate variables are selectively discarded during forward reasoning, and are recalculated when back propagation is performed, so as to appropriately reduce the memory occupancy.

[0076] By combining the above optimization methods, the memory occupancy can be significantly reduced while ensuring the inference performance of the world model, thereby providing protection for long-term testing of the end-to-end autonomous driving system.

[0077] Step 4, single-device multi-thread scheduling and multi-device collaborative calculation;

[0078] A multi-thread parallel scheduling mechanism is constructed for a single-device environment, and a multi-device collaborative calculation mechanism is constructed for a multi-device environment. The two mechanisms work together to further improve the inference efficiency of the world model. The specific steps are as follows:

[0079] Step 4.1, single-device multi-thread scheduling;

[0080] For a single-device environment, a multi-thread parallel scheduling mechanism is constructed to fully utilize the computing resources and pipeline job capabilities of the device. The sub-tasks that can be calculated in parallel during the world model inference process are divided into independent execution units, and they are allocated to different computing threads to reduce resource waste caused by serial execution of each sub-task.

[0081] On this basis, a thread synchronization and buffer mechanism is designed to achieve task overlap between input preparation, core calculation and post-processing. In the image generation, diffusion sampling and perspective rendering process, multiple sub-tasks are executed concurrently to effectively compress the model inference time. At the same time, the task execution order is dynamically adjusted according to the task dependency relationship and device resource state to ensure the cooperative operation of each thread and the efficient use of resources, and to improve the data throughput capacity in the model inference process.

[0082] Step 4.2, multi-device collaborative calculation;

[0083] For a multi-device environment, a multi-device collaborative calculation mechanism is constructed to improve the inference speed of the world model. Through task division, model decomposition and perspective allocation, multiple devices are assigned different calculation tasks, and necessary data exchange and result integration are completed during execution. Meanwhile, parallel optimization is implemented in the calculation tasks of each stage.

[0084] The compact intermediate feature representation is preferentially transmitted, the data is quickly transmitted through the high-speed interconnection, the resource utilization of each computing node is coordinated, the data is synchronized, the load balancing and the computing continuity are ensured, the communication overhead between devices is reduced, and the model has higher inference efficiency.

[0085] The present application has the following advantages:

[0086] The application provides a high-fidelity lightweight world model construction method for end-to-end automatic driving test, which is used for solving the problems of low multi-modal information alignment accuracy, poor cross-view and cross-frame consistency and low inference efficiency of the world model. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 It is a whole flowchart of the application.

[0088] Figure 2 It is a whole architecture diagram of the high-fidelity world model.

[0089] Figure 3 It is an image generation result of the high-fidelity lightweight world model under single-frame multi-view.

[0090] In the figure, (a)-(j) are single-frame multi-view image generation results 1-10.

[0091] Figure 4 It is an image generation result of the high-fidelity lightweight world model under multi-frame multi-view. Figure 1

[0092] Figure 5 It is an image generation result of the high-fidelity lightweight world model under multi-frame multi-view. Figure 2

[0093] In the figure, t-1, t and t+1 are different time steps. DETAILED DESCRIPTION

[0094] ​​The embodiment will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the embodiment, but not to limit the embodiment.

[0095] Referring to Figure 1 , the embodiment provides a high-fidelity lightweight world model construction method for end-to-end automatic driving test, including the following steps:

[0096] Step 1, constructing a high-fidelity world model;

[0097] Referring to Figure 2 , a world model basic skeleton is constructed based on a diffusion model, a conditional control module is constructed based on the world model basic skeleton to realize controllable generation of multi-view images of the world model, an instance control module, a multi-view and time sequence frame control module are integrated, instance depth and 3D mask are introduced, instance features are extracted combined with instance attributes, cross-view Figure One consistency loss function and cross-frame consistency loss function are constructed, a world model training loss function is designed, and a high-fidelity world model is constructed, and the specific steps are as follows:

[0098] Step 1.1, constructing a world model basic skeleton;

[0099] The world model basic skeleton is constructed by using a diffusion model, and the diffusion model includes a forward diffusion and a reverse denoising process. In the forward diffusion process, Gaussian noise is gradually added to the original data, as shown in formula (1):

[0100] (1)

[0101] In the formula, N represents a Gaussian distribution; q(z t |z t-1 ) represents a transition probability distribution in the forward diffusion process; z t represents a latent variable; I represents a unit covariance matrix; β t represents the noise variance of the t-th time step; the latent variable z t at any time step is calculated according to the original sample x0, and the marginal distribution is as shown in formula (2):

[0102] (2)

[0103] In the formula, q(z t |x0) represents the marginal distribution in the forward diffusion process, α t represents a single-step signal retention coefficient, and α t =1-β t ;

[0104] In the reverse denoising process, a neural network is used to approximate the reverse conditional distribution q(zt-1 |z t ), step-by-step denoising reconstruction of the original sample, training of the denoising neural network ε θ (z t , t) prediction of the Gaussian noise ε introduced in the forward diffusion process, and further obtaining the approximate mean prediction, as shown in equation (3):

[0105] (3)

[0106] In the formula, μ θ (z t , t) represents the predicted mean, and the mean is used to construct the inverse distribution p θ (z t-1 |z t ), as shown in equation (4):

[0107] (4)

[0108] In the formula, C represents the covariance control term of the distribution, and the latent variable , ;

[0109] Step 1.2, constructing a conditional control module;

[0110] This embodiment integrates a conditional control module on the basic skeleton of the world model, so that the world model generation result can meet the set scene conditions. Specifically, six types of parallel encoding and conditional control signals that can be integrated into the world model skeleton are defined, which are scene 3D layout L 3D , vehicle pose pose, camera sensor parameter M cam , text prompt word prom, road layout L road , reference frame X ref . These six types of conditional control signals are represented as comprehensive conditional variables c, as shown in equation (5):

[0111] (5)

[0112] An encoder is constructed for each of the six types of conditional control signals to obtain the encoded six types of conditional control information, as shown in equation (6):

[0113] (6)

[0114] In the formula, e 3D , e pose , e cam , e prom , e road , e ref are 3D layout L 3D, vehicle pose, camera sensor parameters M cam , text prompt words prom, road layout L road , reference frame X ref encoded conditional control information, E 3D , E pose , E cam , E prom , E road , E ref respectively 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt words prom, road layout L road , reference frame X ref encoder;

[0115] The conditional control information is normalized and fused, as shown in equation (7):

[0116] (7)

[0117] In the formula, e c represents the aggregated comprehensive conditional control information; f represents the conditional control information aggregator; e c is integrated into the world model skeleton network, to improve the fusion effect, the embodiment adopts the FiLM modulation method to realize the conditional control information fusion, as shown in equation (8):

[0118] (8)

[0119] In the formula, represents the feature representation after the conditional control information is integrated; h l represents the intermediate feature of the l-th layer in the world model skeleton; γ l and d l respectively represent the channel scaling and offset functions; when the model is trained, the modulation structure is embedded into the diffusion model denoising network, and learning is performed by minimizing the diffusion reconstruction loss under the control condition, and the diffusion reconstruction loss function L cond as shown in equation (9):

[0120] (9)

[0121] In the formula, represents the expected calculation operator, and ζ represents the Gaussian sampling noise;

[0122] Step 1.3, constructing an instance control module;

[0123] To realize precise control of instances and avoid the world model from generating multi-view images that violate real laws, the embodiment introduces an instance depth D (k)and 3D mask M (k) ∈0,1 H×W Combined with instance property c (k) (Direction, position), instance features are extracted through multiple encoders, as shown in Equation (10):

[0124] (10)

[0125] In the formula, z (k) E represents the fused feature vector of the k-th instance. inst E D E M These represent the instance attribute feature encoder, the instance depth feature encoder, and the 3D mask feature encoder, respectively. The fusion mapping function is used to enhance training stability and improve the model's ability to perceive instance pose and depth. This embodiment introduces an instance loss function L. inst As shown in equation (11):

[0126] (11)

[0127] In the formula, , The model prediction results represent the 3D mask and instance depth; λ m , λ d represents the loss weights for 3D mask and instance depth; BCE represents the binary cross-entropy loss; K represents the set of instance objects;

[0128] Step 1.4: Construct the multi-view and time sequence frame control module;

[0129] To enhance the generation capability of world models and improve cross-view image processing. Figure One To improve structural consistency and cross-frame consistency, this embodiment introduces cross-view attention mechanism and cross-frame attention mechanism into the network structure to enhance the structural consistency of images under different camera views and the dynamic coherence between consecutive time frames.

[0130] At the multi-view level, let the camera matrix under the i-th view be P. i =K i [R i |t i ], K i R represents the camera's intrinsic parameters. i t i These represent the rotation matrix and translation vector in the camera extrinsic parameters, respectively. To enhance the modeling ability of the same scene instance from different perspectives, a cross-view attention mechanism is introduced. By aligning instance regions under different perspectives through geometric relationships, the network is guided to fuse contextual information from other views at the feature level, thereby improving the consistency and physical alignability of images under different camera perspectives.

[0131] At the continuous frame level, a cross-frame attention mechanism is introduced to model the dynamic evolution process of instances between consecutive frames. Let the inter-frame vehicle motion be δ, and align the features based on geometric transformation. Map the feature map at time t+1 to the coordinate system at time t. Combined with the attention mechanism, information fusion between consecutive frames is achieved, improving the temporal consistency and dynamic stability of the generated image sequence.

[0132] Furthermore, to ensure the consistency of the model generation results across different views and time frames, this embodiment introduces a supervised loss based on instance representation. For instance k at each time t and each view i, the loss is calculated by masking the instance representation. Applied to feature maps Extract the instance vector representation, as shown in equation (12):

[0133] (12)

[0134] In the formula, Let represent the semantic vector representation of instance k from perspective i at time t, and AugPool represents average pooling.

[0135] Furthermore, build cross-view Figure One The consistency loss function and the cross-frame consistency loss function are used to measure the consistency of instances among multiple views at the same time and the consistency of instances among multiple views at consecutive times, as shown in equations (13) and (14), respectively:

[0136] (13)

[0137] (14)

[0138] In the formula, L view L temp They represent cross-view Figure One Consistency loss function and cross-frame consistency loss function; cos represents cosine similarity; Let represent the set of instances viewed from the i-th camera at time t.

[0139] Step 1.5: Construct the world model training loss function;

[0140] The complete training loss function for the world model constructed in this embodiment is shown in equation (15):

[0141] (15)

[0142] In the formula, L total Indicates total loss; L cond Let L represent the diffusion reconstruction loss function, and its calculation formula is shown in equation (9); instrepresents an instance loss function, and a calculation formula is shown in equation (11); L view represents a cross-view Figure One represents a cross-view temp represents a cross-frame consistency loss function, and a calculation formula is shown in equation (14); λ cond , λ inst , λ view , λ temp respectively represent weight coefficients of each loss.

[0143] Step 2, knowledge distillation is performed on the world model;

[0144] A basic architecture of “teacher model-student model” knowledge distillation is constructed, a knowledge distillation loss function is designed, the multi-view image output by the student model is aligned with the output of the teacher model, the high-fidelity world model in step 1 is processed in a lightweight manner, and the complexity and reasoning overhead of the world model are reduced. The specific steps are as follows:

[0145] Step 2.1, a basic architecture of knowledge distillation is constructed;

[0146] A “teacher model-student model” knowledge distillation architecture is built, the teacher model T is the high-fidelity world model trained in step 1, the model parameters remain fixed and are only used for forward reasoning, and the student model S is a lightweight world model that is consistent with the teacher model T in the interface but has a significantly reduced parameter scale. The parameters can be trained and continuously learn and approximate the output performance of the teacher model T under the same input condition during the knowledge distillation process;

[0147] To ensure the effectiveness and stability of the distillation process, the input information is standardized to a unified numerical precision, tensor dimension and data type to ensure the consistency between the teacher model T and the student model S. During the training process, the input data is sampled, encoded and normalized, and a random variable in diffusion modeling is introduced to randomly transmit the input data into the teacher model T and the student model S. The teacher model T and the student model S are consistent in structure and use the same control module combination to ensure that the distillation training is performed under the premise of complete alignment;

[0148] To achieve efficient training of multiple camera perspectives, the batch perspective merging strategy is adopted in this embodiment, that is, the original batch dimension and perspective dimension are spliced, and the original structure is restored in the output stage to ensure that the model prediction output and the supervision signal are strictly corresponding in the sample dimension and the perspective dimension;

[0149] Step 2.2, a knowledge distillation loss function is designed;

[0150] After constructing the basic architecture of knowledge distillation, the output of the teacher model T is taken as the supervision target, and the predicted output of the student model S is aligned with the supervision target. Specifically, at time t, a Gaussian perturbation is sampled for each sample and multi-view image, which is taken as the supervision target, denoted as Under the same input condition, the student model S outputs the predicted tensor The distillation loss is calculated in the form of mean square error, and the sample and view angle dimensions are averaged to obtain the distillation loss L KD (t) as shown in formula (16):

[0151] (16)

[0152] In the formula, represents the expected calculation operator, represents the predicted output of the student model, and η represents the output of the teacher model;

[0153] When the batch size is B and the number of views is N cam , formula (16) is equivalent to the finite sum form shown in formula (17):

[0154] (17)

[0155] In the formula, b represents the sample index within the batch B, n represents the camera view index, represents the predicted output of the student model, represents the output of the teacher model, i.e., the supervision target of the student model;

[0156] In order to consider the importance and training stability of different time steps, without changing the input condition, the embodiment gives non-negative weights w t to each time loss, and obtains the weighted total loss shown in formula (18):

[0157] (18)

[0158] Step 3, custom development of CUDA operator;

[0159] Locate the bottleneck link with the largest consumption of computing resources in the world model, and develop CUDA operators accordingly. Rewrite the key calculation kernel to fully utilize the parallel computing performance of GPU, and optimize the memory occupation logic. The specific steps are as follows:

[0160] Step 3.1, determine the world model calculation bottleneck;

[0161] The embodiment analyzes the running process of the world model, locates the bottleneck link with the largest calculation resource consumption, specifically, analyzes the complexity of each module of the world model and calculates the actual running time, defines each sub-module of the world model as a set o j , executes the complete reasoning process of the world model, records the running time T j of each sub-module, regards the module with the longest time consumption as the main calculation bottleneck module of the world model, and at the same time, evaluates the peak value of the video memory occupation of each module to find the potential memory bottleneck;

[0162] Step 3.2, developing CUDA operators for the determined calculation bottleneck;

[0163] For the determined calculation bottleneck module, the embodiment uses a custom CUDA operator to rewrite the key calculation kernel to fully utilize the parallel computing performance of the GPU. Specifically, the cascade calculation of two or more continuous operators is fused into a single CUDA kernel function to avoid repeated writing and reading of intermediate results to and from the video memory. In the process of large convolution operation, the thread block division and shared memory usage mode are optimized so that multiple calculation steps can be completed in one traversal. In the multi-view projection calculation, a special CUDA kernel is written to fuse the geometric transformation and interpolation into a single parallel calculation, replacing the pixel-by-pixel serial operation.

[0164] Through operator fusion and special CUDA kernel design, the number of memory access times and kernel launch overhead are significantly reduced, the parallel computing capability of a large number of threads of the GPU and the high-speed memory bandwidth are fully utilized, and the calculation efficiency of the bottleneck module of the world model is greatly improved.

[0165] Step 3.3, optimizing video memory occupation;

[0166] While improving the calculation efficiency of the bottleneck module of the world model, the embodiment optimizes the model memory occupation by various means. Specifically, a mixed precision calculation strategy is introduced to compress the model weights from 32-bit floating-point numbers to 16-bit floating-point numbers for storage and operation, which greatly saves storage space while ensuring that the model accuracy is almost not affected. Memory reuse and timely release techniques are used to reduce redundant intermediate results. For intermediate tensors generated during the gradual calculation of the world model that are no longer needed subsequently, they are released. For intermediate results that need to be calculated repeatedly, some intermediate variables are selectively discarded during forward reasoning, and are recalculated when back propagation is performed, so as to appropriately reduce the video memory occupation.

[0167] The embodiment combines the above optimization methods to significantly reduce the video memory occupation while ensuring the reasoning performance of the world model, providing protection for the long-time testing of the end-to-end automatic driving system.

[0168] Step 4, single-device multi-thread scheduling and multi-device collaborative calculation;​

[0169] A multi-thread parallel scheduling mechanism is constructed for a single device environment, and a multi-device cooperative computing mechanism is constructed for a multi-device environment, which further improves the world model reasoning efficiency by mutual cooperation. The specific steps are as follows:

[0170] Step 4.1, single device multi-thread scheduling;

[0171] This embodiment is for a single device environment, and a multi-thread parallel scheduling mechanism is constructed to fully utilize the computing resources and pipeline job capabilities of the device. Specifically, the subtasks that can be calculated in parallel in the world model reasoning process are divided into independent execution units, and they are allocated to different computing threads to reduce resource waste caused by serial execution of each subtask.

[0172] On this basis, this embodiment further designs a thread synchronization and buffer mechanism to realize task overlap between input preparation, core calculation and post-processing. In the image generation, diffusion sampling and perspective rendering process, multiple subtasks are run simultaneously in a concurrent execution mode, thereby effectively compressing the model reasoning time. At the same time, the task execution order is dynamically adjusted according to the task dependency relationship and device resource state to ensure the cooperative operation of each thread and the efficient use of resources, and to improve the data throughput capacity in the model reasoning process.

[0173] Step 4.2, multi-device cooperative computing;

[0174] This embodiment is for a multi-device environment, and a multi-device cooperative computing mechanism is constructed to further improve the reasoning speed of the world model. Specifically, through task division, model decomposition and perspective allocation, multiple devices are assigned different computing tasks, and necessary data exchange and result integration are completed during execution. At the same time, parallel optimization is implemented in the computing tasks of each stage.

[0175] To reduce the communication overhead between devices, compact intermediate feature representations are preferentially transmitted, and high-speed interconnection is used to realize fast data transmission. The resource utilization of each computing node is coordinated and data synchronization is performed to ensure load balancing and computing continuity, so that the model has higher reasoning efficiency.

[0176] Effect verification:

[0177] The world model architecture is constructed using the method of the present application, the nuScenes dataset is used to train the world model, the trained world model is subjected to knowledge distillation, and CUDA operators are self-defined for the world model computing bottleneck part. Single device multi-thread scheduling and multi-device cooperative computing strategies are executed on 4*NVIDIA L20 GPUs.

[0178] The single-frame multi-perspective image generated by the high-fidelity lightweight world model constructed in this embodimentFigure 3 As shown in Figure 3 As can be seen, the world model can accurately generate corresponding instances according to the 3D layout, while Figure 3 The red oval region in FIG. 8 shows strong cross-view consistency, effectively solving the challenge of low alignment accuracy and poor cross-view consistency of multi-modal information of existing world models. Figure One Figure One The red oval region in FIG. 8 shows strong cross-frame consistency, effectively solving the challenge of poor cross-frame consistency of existing world models. Figures 4-5 Figures 4-5 The red oval region in FIG. 8 shows strong cross-frame consistency, effectively solving the challenge of poor cross-frame consistency of existing world models.

[0179] To quantitatively verify the generation effect and inference efficiency of the high-fidelity lightweight world model constructed in this embodiment, the method of this embodiment is compared with mainstream world models such as DriveDreamer and DrivingDiffusion. The FID (Frechet Inception Distance) and FVD (Frechet Video Distance) indicators are constructed to evaluate the quality of multi-view images and videos generated by the model, and the single-frame multi-view image generation time is used to evaluate the inference efficiency of the model. The comparison results are shown in Table 1.

[0180] Table 1 Evaluation indicators of generation results of different world models

[0181]

[0182] As can be seen from Table 1, compared with existing mainstream world models, the FID and FVD indicators of the high-fidelity world model constructed in this embodiment (steps 2, 3, and 4 are not performed) are greatly reduced, achieving the best image and video generation results, but the single-frame inference time is comparable to existing models. Although the high-fidelity, lightweight world model (steps 2, 3, and 4 are performed) reduces the model parameters and performs inference acceleration, its performance does not decrease significantly and is similar to that of the high-fidelity world model (steps 2, 3, and 4 are not performed). Compared with existing methods, the image and video generation quality is still greatly improved. In addition, through knowledge distillation, custom development of CUDA operators, single-device multi-thread scheduling, and multi-device collaborative computing, the model inference efficiency is greatly improved, and the single-frame inference time can reach 112.758 ms, verifying the effectiveness of the method of this embodiment.

[0183] ​​In conclusion, the high-fidelity lightweight world model construction method for end-to-end automatic driving test provided by the present application effectively solves the problems of low multi-modal information alignment accuracy, poor cross-view and cross-frame consistency of the existing world model, and through knowledge distillation of the world model, custom development of CUDA operators for the calculation bottleneck, and cooperation of single-device multi-thread scheduling and multi-device collaborative calculation methods, a high-fidelity lightweight world model with high efficient inference ability is established, which not only improves the confidence of the end-to-end automatic driving system test process, but also greatly speeds up the test efficiency of the end-to-end automatic driving system, and accelerates the industrialization landing of the end-to-end automatic driving car.

Claims

1. A high-fidelity lightweight world model construction method for end-to-end autonomous driving testing, characterized in that: Comprising the following steps: Step 1, constructing a high-fidelity world model; Based on the diffusion model, the basic skeleton of the world model is constructed, and the conditional control module is constructed based on the basic skeleton of the world model to realize the controllable generation of the world model multi-view image, and the instance control module, multi-view and time sequence frame control module are integrated, the instance depth and 3D mask are introduced, the instance characteristics are extracted combined with the instance attributes, the cross-view consistency loss function and the cross-frame consistency loss function are constructed, and the world model training loss function is designed to construct a high-fidelity world model; comprising the following steps: Step 1.1, constructing a basic skeleton of the world model; The basic skeleton of the world model is constructed by using the diffusion model, which includes forward diffusion and reverse denoising process. In the forward diffusion process, Gaussian noise is gradually added to the original data: where N denotes a Gaussian distribution; q(z t |z t-1 ) denotes a transition probability distribution in the forward diffusion process; z t denotes a latent variable; I denotes an identity covariance matrix; β t denotes a noise variance at the t-th time step; the latent variable z t at any time step is calculated according to the original sample x0, and the marginal distribution is: where q(z t represents the edge distribution in the forward diffusion process, a t represents the single-step signal preservation coefficient, a t = 1 - β t ; In the reverse denoising process, the neural network is used to generate the reverse conditional distribution q(z t-1 |z t ) of each step, and the original sample is reconstructed by step-by-step denoising. The denoising neural network ε θ (z t ,t) is trained to predict the Gaussian noise ε introduced in the forward diffusion process, and the predicted mean is obtained: where μ θ (z t ,t) denotes the predicted mean, with which the inverse distribution p θ (z t-1 |z t ) is constructed for sampling. In the formula, covariance control term representing the distribution, latent variable , ; Step 1.2, constructing a conditional control module; Integrate conditional control module on the basic skeleton of world model, define six types of conditional control signals which can be parallel coded and integrated into the skeleton of world model, including scene 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt word prom, road layout L road , reference frame X ref , and express the six types of conditional control signals as comprehensive conditional variables c: An encoder is constructed for each of the six types of conditional control signals to obtain the encoded six types of conditional control information: wherein e 3D , e pose , e cam , e prom , e road , e ref are encoders of 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt prom, road layout L road , reference frame X ref encoded conditional control information, E 3D , E pose , E cam , E prom , E road , E ref are encoders of 3D layout L 3D , vehicle pose, camera sensor parameters M cam , text prompt prom, road layout L road , reference frame X ref , respectively. The conditional control information is normalized and fused: In the formula, e c represents the integrated condition control information after polymerization; f represents a condition control information aggregator; The integrated condition control information e c is incorporated into the world model skeleton network; Step 1.3, constructing an instance control module; Introducing instance depth D for instance objects (k) and 3D mask M (k) ∈0,1 H×W , combining instance properties c (k) extract instance features through multiple encoders: wherein z (k) represents the fusion feature vector of the kth instance, E inst , E D , E M respectively represent the instance attribute feature encoder, the instance depth feature encoder, and the 3D mask feature encoder. represents the fusion mapping function; in order to enhance the training stability and improve the perception ability of the model to the instance pose and depth, an instance loss function L inst is introduced. wherein , represents the model prediction results of 3D mask, instance depth; λ m , λ d is the loss weight of 3D mask, instance depth; BCE represents the binary cross entropy loss; K represents the set of instance objects; Step 1.4, constructing a multi-view and time sequence frame control module; In the multi-view level, let the camera matrix under the i-th view be P i = K i [R i |t i ], K i denotes the camera intrinsic, R i , t i denote the rotation matrix and translation vector in the camera extrinsic respectively; in the world model network structure, a cross-view attention mechanism is introduced, the instance regions under different views are aligned through geometric relationship, the network is guided to fuse the context information from other views at the feature level, and the consistency and physical alignability of the image under different camera views are improved; In the continuous frame level, the cross-frame attention mechanism is introduced into the world model network structure to model the dynamic evolution process of the instance between continuous frames, and the inter-frame self-vehicle motion is δ. Based on geometric transformation, the features are aligned, the feature map at time t+1 is mapped to the coordinate system at time t, and the information fusion between the front and rear frames is realized combined with the attention mechanism, which improves the temporal consistency and dynamic stability of the generated image sequence; Introduce a supervision loss based on instance representation, constrain the consistency of model generation results under different views and time frames, for each instance k under each time t, each view i, through the instance mask Applied to the feature map , extract the instance vector representation: wherein, represents the semantic vector representation of instance k at time t and view i, and AugPool represents average pooling. The cross-view consistency loss function and the cross-frame consistency loss function are constructed to measure the consistency of the instance between multiple views at the same time and the consistency of the instance between multiple views at continuous times: In the formula, L view , L temp respectively represent the cross-view consistency loss function and the cross-frame consistency loss function; cos represents the cosine similarity. represents the instance set under the i-th camera view at the t-th moment. Step 1.5, constructing a world model training loss function; The complete training loss function of the world model is constructed: wherein L total represents the total loss; L cond represents the diffusion reconstruction loss function; L inst represents the instance loss function; L view represents the cross-view consistency loss function; L temp represents the cross-frame consistency loss function; λ cond , λ inst , λ view , λ temp respectively represent the weight coefficients of each loss. Step 2, knowledge distillation of the world model; The basic architecture of the "teacher model-student model" knowledge distillation is constructed, the knowledge distillation loss function is designed, the multi-view images output by the student model are aligned with the teacher model output, the high-fidelity world model in step 1 is processed to reduce the complexity and reasoning overhead of the world model; Step 3, custom development of CUDA operator; Locate the bottleneck link with the largest calculation resource consumption in the world model, develop CUDA operator specifically, rewrite the key calculation kernel, fuse the cascade calculation of two or more continuous operators into a single CUDA kernel function, fully utilize the parallel computing performance of GPU, optimize thread block division and shared memory usage, and write a special CUDA kernel during multi-view projection calculation; introduce a mixed precision calculation strategy, use memory reuse and timely release technology to optimize memory occupation logic; Step 4, single-device multi-thread scheduling and multi-device collaborative computing; For single-device environment, a multi-thread parallel scheduling mechanism is constructed, and for multi-device environment, a multi-device collaborative computing mechanism is constructed, which cooperates to improve the reasoning efficiency of the world model.

2. The high-fidelity lightweight world model construction method for end-to-end autonomous driving testing of claim 1, wherein: In step 1.2, the integrated condition control information e c The condition control information is integrated into the world model skeleton network in a FiLM modulation manner: wherein represents a feature representation after fusion of the conditional control information; h l represents an intermediate feature of the l-th layer in the world model skeleton; γ l with d l denote the channel scaling and offset functions, respectively; when the model is trained, the modulation structure is embedded into the diffusion model denoising network, learning is performed by minimizing the diffusion reconstruction loss under the control condition, and the diffusion reconstruction loss function L cond : wherein represents the desired computational operator, and ζ represents Gaussian sampling noise.

3. The high-fidelity lightweight world model construction method for end-to-end autonomous driving testing of claim 1, wherein: Step 2 comprises the following steps: Step 2.1, constructing a knowledge distillation basic architecture; A "teacher model-student model" knowledge distillation architecture is built, the teacher model T is a high-fidelity world model trained in step 1, the model parameters remain fixed and are only used for forward inference, and the student model S is a lightweight world model consistent with the teacher model T in the interface but with a reduced parameter size, the parameters of which can be trained and continuously learned and approximated to the output performance of the teacher model T under the same input conditions in the knowledge distillation process; In the distillation process, the input information is standardized to a unified numerical precision, tensor dimension and data type to ensure consistency between the teacher model T and the student model S, and in the training process, the input data is sampled, encoded and normalized, and random variables in diffusion modeling are introduced, the input data is randomly transmitted to the teacher model T and the student model S, and the teacher model T and the student model S remain consistent in structure and use the same control module combination to ensure that the distillation training is performed under complete alignment; In order to realize efficient training of multiple camera perspectives, a batch perspective merging strategy is adopted, that is, the original batch dimension and perspective dimension are spliced, and the original structure is restored in the output stage to ensure that the model prediction output and the supervision signal are strictly corresponding in the sample dimension and the perspective dimension; Step 2.2, design a knowledge distillation loss function; After building the basic architecture of knowledge distillation, the output of the teacher model T is taken as the supervision target, and the prediction output of the student model S is aligned with the supervision target.

4. The high-fidelity lightweight world model construction method for end-to-end autonomous driving testing of claim 3, wherein: Step 2.2 is: the teacher model T samples Gaussian perturbation for each sample and multi-view image at time t, and takes it as a supervised target, and the student model S outputs a predicted tensor under the same input condition The distillation loss is calculated in the form of mean square error, and is averaged in the sample and view angle dimensions to obtain the distillation loss L KD (t): wherein represents the desired computation operator, represents the prediction output of the student model, and η represents the output of the teacher model; When the batch size is B, the number of views is N cam and the distillation loss L KD (t) is equivalent to the finite sum form: where b represents a sample index within batch B, n represents a camera view index, represents a prediction output of the student model, represents an output of the teacher model, i.e., a supervision target of the student model; In order to balance the importance of different time steps and training stability, non-negative weights w are given to the loss of each time step without changing the input condition t , and the weighted total loss is obtained : 。 5. The high-fidelity lightweight world model construction method for end-to-end autonomous driving testing of claim 1, wherein: Step 3 includes the following steps: Step 3.1, determine the world model calculation bottleneck; The running flow of the world model is analyzed, the complexity of each module of the world model is analyzed, and the actual running time is calculated, each sub-module of the world model is defined as a set o j , the complete reasoning process of the world model is executed, and the running time T j of each sub-module is recorded The module with the longest time consumption is regarded as the main calculation bottleneck module of the world model, and at the same time, the peak value of the display memory occupation of each module is evaluated to find the potential memory bottleneck; Step 3.2, develop a CUDA operator for the determined calculation bottleneck; For the determined calculation bottleneck module, a custom CUDA operator is used to rewrite the key calculation kernel to fully utilize the parallel computing performance of the GPU, and the cascade calculation of two or more consecutive operators is fused into a single CUDA kernel function to avoid repeated writing and reading of intermediate results to and from the memory; in the process of large convolution operation, the thread block division and shared memory usage mode are optimized, so that multiple calculation steps can be completed in one traversal; in the multi-view projection calculation, a special CUDA kernel is written to fuse geometric transformation and interpolation into a single parallel calculation, replacing the pixel-by-pixel serial operation; Step 3.3, optimize the memory occupation logic; While improving the calculation efficiency of the world model bottleneck module, a mixed precision calculation strategy is introduced to compress the model weights from 32-bit floating point numbers to 16-bit floating point numbers for storage and operation; memory reuse and timely release techniques are used to reduce redundant intermediate results, and intermediate tensors generated in the world model calculation process that are no longer needed are released, and intermediate variables that need to be calculated repeatedly are selectively discarded during forward inference and recalculated during backpropagation to appropriately reduce memory occupation.

6. The high-fidelity lightweight world model construction method for end-to-end autonomous driving testing of claim 1, wherein: Step 4 includes the following steps: Step 4.1, single-device multi-thread scheduling; For a single-device environment, a multi-thread parallel scheduling mechanism is constructed to fully utilize the computing resources and pipeline job capabilities of the device. The sub-tasks that can be parallel computed in the world model inference process are divided into independent execution units and allocated to different computing threads, reducing resource waste caused by serial execution of each sub-task. On this basis, a thread synchronization and buffer mechanism is designed to achieve task overlap between input preparation, core calculation and post-processing. In the image generation, diffusion sampling and perspective rendering process, multiple sub-tasks are run simultaneously in a concurrent execution mode to compress the model inference time. At the same time, the task execution order is dynamically adjusted according to the task dependency relationship and device resource state to ensure the collaborative operation of each thread and the efficient use of resources, and to improve the data throughput capacity in the model inference process. Step 4.2, multi-device collaborative computing; For a multi-device environment, a multi-device collaborative computing mechanism is constructed to improve the inference speed of the world model. Through task division, model decomposition and perspective allocation, multiple devices are assigned different computing tasks and perform necessary data exchange and result integration during execution. Parallel optimization is also implemented in the computing tasks of each stage. Compact intermediate feature representations are transmitted preferentially, and high-speed interconnection is used to achieve fast data transmission. The resource utilization of each computing node is coordinated and data synchronization is performed to ensure load balancing and computing continuity, and to reduce communication overhead between devices.

Images