World model-based unmanned logistics vehicle intelligent driving model training method and system

Abstract

The application provides a world model-based unmanned logistics vehicle intelligent driving model training method and system. The method generates instructions by obtaining historical video data and long-tail scene data, merges the historical video data with a preset amount of Internet video data as sample data, trains a preset architecture generative world model through the sample data, determines a structured generation instruction, controls the first generative world model to generate multi-modal long-tail scene data as an input condition, determines first long-tail scene state data, and trains and evaluates the first generative world model through the first long-tail scene state data to obtain a teacher model, determines a first student model, and deploys the first student model on a vehicle end to make decision control during unmanned logistics vehicle driving. The application effectively solves the problems of difficult long-tail data acquisition and high cost, and improves the accuracy of long-tail scene recognition of the student model.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a training method and system for an intelligent driving model of an unmanned logistics vehicle based on a world model. Background Technology

[0002] With the rapid development of artificial intelligence and autonomous driving technologies, Level 4 autonomous driving technology is gradually moving from the laboratory to practical applications. Unmanned logistics vehicles, as a key application of autonomous driving technology, have become a priority for the deployment of Level 4 autonomous driving technology due to their relatively closed operating environment, clear mission objectives, and significant economic benefits. However, to achieve large-scale commercial operation of Level 4 unmanned logistics vehicles, it is essential to address the "long-tail effect" challenge faced by autonomous driving systems in open and dynamic environments. Long-tail scenarios refer to extremely rare, diverse, and unpredictable extreme situations that occur in massive traffic environments, such as non-standard obstacles (e.g., road debris, animals crossing), illegal behavior by traffic participants (e.g., pedestrians, non-motorized vehicles), and adverse perception conditions such as rain, fog, backlighting, and changing lighting in tunnels. Although these long-tail scenarios occur infrequently, they are the main causes of autonomous driving system failures and even safety accidents.

[0003] Currently, the industry mainly relies on the paradigm of "large-scale real-world road data collection + manual annotation + rule patching" to address the above challenges. However, this approach has significant bottlenecks: First, data acquisition is costly, requiring a large fleet of vehicles for long-term testing, and it is difficult to exhaustively cover all extreme conditions. Second, data utilization is low, with massive amounts of normal driving data overwhelming scarce fault segments, resulting in insufficient generalization ability of the model for long-tail scenarios. Third, while traditional simulation testing can generate some virtual scenes, it is often limited by the modeling accuracy of the physics engine and the simplification of traffic flow logic, often resulting in a "sim-to-real gap," making it difficult to reproduce the complex textures and physical interactions of the real world. Meanwhile, although world model technology based on generative AI provides a new approach to data synthesis, existing solutions still suffer from poor physical consistency and a lack of dynamic interaction logic, and most remain at the offline data augmentation stage, lacking closed-loop linkage with on-vehicle real-world operating data.

[0004] Therefore, it is necessary to improve the training methods for intelligent driving models of unmanned logistics vehicles in related technologies. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, this application provides a training method and system for an intelligent driving model of unmanned logistics vehicles based on a world model, in order to solve the above-mentioned technical problems.

[0006] According to one aspect of the embodiments of this application, a method for training an intelligent driving model for an unmanned logistics vehicle based on a world model is provided. The method includes: acquiring historical video data collected by the unmanned logistics vehicle during its operation and long-tail scene data generation instructions input by target personnel; the target personnel include: developers and operators; merging the historical video data with a preset number of internet video data as sample data; training a generative world model with a preset architecture using the sample data to obtain a first generative world model; performing semantic parsing on the long-tail scene data generation instructions based on a long-tail scene generator to obtain structured generation instructions, and controlling the first generative world model to generate multimodal long-tail scene data using the structured generation instructions as input conditions; injecting the multimodal long-tail scene data into a pre-constructed simulation environment based on a digital twin training engine to obtain first long-tail scene state data; training and evaluating the first generative world model using the first long-tail scene state data to obtain a teacher model; performing knowledge distillation on the teacher model to obtain a first student model, and deploying the first student model on the vehicle to perform decision control through the first student model during the operation of the unmanned logistics vehicle.

[0007] According to another aspect of the embodiments of this application, a training system for an intelligent driving model of an unmanned logistics vehicle based on a world model is also provided, comprising: a data acquisition module, used to acquire historical video data collected during the operation of the unmanned logistics vehicle and long-tail scene data generation instructions input by target personnel; the target personnel include: developers and operators; a first model training module, used to merge the historical video data with a preset number of internet video data as sample data; and to train a generative world model with a preset architecture using the sample data to obtain a first generative world model; and a data generation module, used to perform semantic analysis on the long-tail scene data generation instructions based on a long-tail scene generator. The system analyzes and obtains structured generation instructions, and controls the first generative world model to generate multimodal long-tail scene data using the structured generation instructions as input conditions. A second model training module injects the multimodal long-tail scene data into a pre-built simulation environment based on a digital twin training engine to obtain first long-tail scene state data. The first generative world model is trained and evaluated using the first long-tail scene state data to obtain a teacher model. A model deployment module performs knowledge distillation on the teacher model to obtain a first student model, and deploys the first student model on the vehicle to perform decision control during the unmanned logistics vehicle's operation.

[0008] The beneficial effects of this application are as follows: This application acquires historical video data collected during the operation of an unmanned logistics vehicle and long-tail scene data generation instructions input by the target personnel. The historical video data is merged with a preset number of internet video data as sample data. The sample data is then used to train a generative world model with a preset architecture to obtain a first generative world model. Based on a long-tail scene generator, the long-tail scene data generation instructions are semantically parsed to obtain structured generation instructions. The first generative world model is then controlled to generate multimodal long-tail scene data using these structured generation instructions as input. Based on a digital twin training engine, the multimodal long-tail scene data is injected into a pre-constructed simulation environment to obtain first long-tail scene state data. The first generative world model is then trained and evaluated using the first long-tail scene state data to obtain a teacher model. Knowledge distillation is performed on the teacher model to obtain a first student model. This first student model is then deployed on the vehicle to perform decision-making and control during the operation of the unmanned logistics vehicle. The process, by introducing a generative world model and a long-tail scene generator, transforms scarce long-tail scene data generation instructions into multimodal long-tail scene data. This reduces reliance on historical video data and generates long-tail scene data on demand, meeting data needs under extreme conditions and effectively solving the problems of difficult and costly long-tail data acquisition, thus improving data coverage and diversity. The generative world model with a pre-defined architecture is trained using sample data composed of historical video data and a preset number of internet video data. The first generative world model is then trained and evaluated using the first long-tail scene state data, preventing the massive amount of sample data from overwhelming the first long-tail scene state data, thereby improving the accuracy of long-tail scene recognition. An evaluation system incorporating physical dynamics compliance scoring enhances the physical consistency and realism of the teacher model's generated data, narrowing the "virtual-real gap." Distilling the teacher model into a lightweight student model and deploying it on the vehicle facilitates an efficient closed loop of "cloud training - vehicle deployment," balancing model performance and vehicle real-time performance.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0011] Figure 1This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application;

[0012] Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application of a method for training an intelligent driving model for unmanned logistics vehicles based on a world model;

[0013] Figure 3 This is a schematic diagram illustrating a world model-based intelligent driving model training method for unmanned logistics vehicles, which is another exemplary embodiment of this application.

[0014] Figure 4 This is a flowchart illustrating a difficult case detection method as shown in an exemplary embodiment of this application;

[0015] Figure 5 This is a block diagram illustrating an exemplary embodiment of the world model-based intelligent driving model training system for unmanned logistics vehicles. Detailed Implementation

[0016] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0017] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0018] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0019] Figure 1 This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application.

[0020] Reference Figure 1As shown, the system architecture may include a storage device 101 and a processing device 102. The processing device 102 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, etc. Technical personnel can use the processing device 102 to acquire historical video data collected during the operation of the unmanned logistics vehicle and long-tail scene data generation instructions input by the target personnel. The historical video data is merged with a preset number of internet video data as sample data. The sample data is then used to train a generative world model with a preset architecture to obtain a first generative world model. Based on the long-tail scene generator, the long-tail scene data generation instructions are semantically parsed to obtain structured generation instructions. The first generative world model is then controlled to generate multimodal long-tail scene data using the structured generation instructions as input conditions. Based on the digital twin training engine, the multimodal long-tail scene data is injected into a pre-constructed simulation environment to obtain the first long-tail scene state data. The first generative world model is then trained and evaluated using the first long-tail scene state data to obtain a teacher model. Knowledge distillation is performed on the teacher model to obtain a first student model. The first student model is then deployed on the vehicle to make decisions and control the unmanned logistics vehicle during operation. The storage device 101 is used to store the historical video data collected by the unmanned logistics vehicle during its operation and the long-tail scene data generation instructions input by the target personnel, and to provide the historical video data collected by the unmanned logistics vehicle during its operation and the long-tail scene data generation instructions input by the target personnel to the processing device 102 for processing.

[0021] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0022] Figure 2 This is a flowchart illustrating an exemplary embodiment of the training method for an intelligent driving model of an unmanned logistics vehicle based on a world model, as shown in this application. (Refer to...) Figure 2 As shown, the training method for the intelligent driving model of the unmanned logistics vehicle based on the world model includes at least steps S210 to S250, which are described in detail below:

[0023] In step S210, historical video data collected by the unmanned logistics vehicle during its operation and long-tail scene data generation instructions input by the target personnel are obtained. In one embodiment of this application, the historical video data is data actually collected by the unmanned logistics vehicle during its operation, and the long-tail scene data generation instructions are in natural language form, for example, "narrow road at night, tricycle traveling in the opposite direction" is used as the long-tail scene data generation instruction.

[0024] In step S220, historical video data is merged with a preset number of internet video data to form sample data; and a generative world model with a preset architecture is trained using the sample data to obtain a first generative world model. In one embodiment of this application, the preset number is set according to actual conditions, and the generative world model with the preset architecture is a large-scale generative world model based on a diffusion model or an autoregressive Transformer architecture. The generative world model with the preset architecture includes: a preset spatiotemporal encoder, a preset conditional injection module, a preset diffusion generation module, and a preset spatiotemporal decoder. Training the generative world model with the preset architecture using sample data helps improve the generative world model's basic cognitive ability of the real road traffic environment, allowing the generative world model to learn the environmental generation rules under different road conditions, weather, and lighting, laying the foundation for subsequent generation of long-tail scenes that conform to real physical rules.

[0025] In step S230, based on the long-tail scene generator, the long-tail scene data generation instructions are semantically parsed to obtain structured generation instructions, and the first generative world model is controlled to generate multimodal long-tail scene data with the structured generation instructions as input conditions. In one embodiment of this application, the process of obtaining structured generation instructions by semantically parsing the long-tail scene data generation instructions based on the long-tail scene generator includes: inputting the long-tail scene data generation instructions into a Large Language Model (LLM) to extract scene features: including moving targets (such as vehicles and pedestrians), environmental attributes (such as weather type and light intensity), and behavioral logic (such as lane change and collision); based on a predefined domain knowledge graph or scene template library, logical completion and parameterized mapping are performed on the identified scene features to determine the spatial position, trajectory, and interaction relationship of each moving target; the spatial position, trajectory, and interaction relationship of each moving target are encapsulated into a structured data format (such as JSON, XML, or scene graph structure) to generate structured generation instructions. The generation of multimodal long-tail scene data solves the problems of high cost and difficulty in acquiring scarce long-tail data in real-world scenarios. Furthermore, the generation method described in this application enables the on-demand acquisition of different types of extreme condition data, significantly expanding the coverage of the training dataset. Multimodal long-tail scene data comprises multi-dimensional data including visual images, point cloud data, target motion trajectories, and environmental sensor response parameters, directly supporting model training and testing in subsequent simulation environments.

[0026] In step S240, multimodal long-tail scene data is injected into a pre-constructed simulation environment based on the digital twin training engine to obtain first long-tail scene state data. The first generative world model is then trained and evaluated using this first long-tail scene state data to obtain a teacher model. In one embodiment of this application, injecting multimodal long-tail scene data into a pre-constructed simulation environment based on the digital twin training engine facilitates the verification of the rationality of the multimodal long-tail scene data in terms of vehicle dynamics and the movement patterns of traffic participants through the virtual-real mapping relationship of the digital twin. Simultaneously, it allows the generated first long-tail scene state data to adapt to the simulation interaction training process of the intelligent driving model and be directly used for iterative testing of the intelligent driving decision model. When training and evaluating the first generative world model based on the first long-tail scene state data, multi-dimensional evaluation indicators including physical dynamics compliance, visual fidelity, and long-tail scene scarcity are introduced to filter out low-quality generated data that does not conform to real physical rules. Simultaneously, the generation capability of the first generative world model is optimized in reverse, ultimately resulting in a teacher model with reliable data generation quality and high scene fidelity.

[0027] In step S250, knowledge distillation is performed on the teacher model to obtain a first student model, which is then deployed on the vehicle to make decisions and control the vehicle during its operation. In one embodiment of this application, knowledge distillation is a model compression method that transfers the generalization ability and decision-making knowledge learned by a large-parameter teacher model to a small-parameter student model, which is particularly suitable for transferring decision logic in long-tail scenarios. Knowledge distillation preserves the decision-making accuracy of the teacher model in various long-tail scenarios while compressing the model parameters to a range suitable for edge computing on the vehicle, ensuring that the model's inference speed meets the real-time decision-making requirements of the unmanned logistics vehicle. This results in a first student model that balances performance and inference efficiency, achieving an efficient collaborative closed loop between high-performance training of the cloud-based model and real-time inference of the small model on the vehicle. The deployment of the first student model on the vehicle is achieved through OTA (Over-the-Air) technology.

[0028] This application introduces a generative world model and a long-tail scene generator to transform scarce long-tail scene data generation instructions into multimodal long-tail scene data. This reduces reliance on historical video data and generates long-tail scene data on demand, meeting data needs under extreme conditions. It effectively solves the problems of difficult and costly long-tail data acquisition and improves data coverage and diversity. The generative world model with a pre-defined architecture is trained using sample data composed of historical video data and a preset number of internet video data. The first generative world model is trained and evaluated using the first long-tail scene state data, avoiding the flooding of the first long-tail scene state data by massive sample data, thereby improving the accuracy of long-tail scene recognition. By constructing an evaluation system that includes physical dynamics compliance scoring, the physical consistency and realism of the teacher model's generated data are improved, narrowing the "virtual-real gap". By distilling the teacher model into a lightweight student model and deploying it on the vehicle, it is conducive to achieving an efficient closed loop of "cloud training - vehicle deployment", balancing model performance and vehicle real-time performance.

[0029] In one embodiment of this application, the process of decision control through a first student model includes:

[0030] The system acquires the real-time driving status and perception information of the unmanned logistics vehicle, as well as the decision-making and control results of the first student model. In one embodiment of this application, the real-time driving status includes: the unmanned logistics vehicle's speed, acceleration, automatic emergency braking status, forward collision warning status, and manual control status. The decision-making and control results include: prediction results, prediction confidence levels, and planned trajectories. The perception information includes: image information, point cloud data, radar detection information, and unmanned logistics vehicle's positioning information surrounding the vehicle. The real-time driving status, perception information, and decision-making and control results are monitored and updated at a frequency of 10-20Hz.

[0031] If the real-time driving status, decision control results, or perception information meet the criteria for difficult case data, then the real-time driving status, decision control results, and perception information are considered as difficult case data. In one embodiment of this application, the difficult case detection process is implemented through a difficult case detection module deployed on the vehicle. When the difficult case detection module determines that the real-time driving status, decision control results, or perception information meet the criteria for difficult case data, it packages the real-time driving status, decision control results, and perception information into a difficult case data package and uploads it to the cloud world model. The generation of long-tail scene augmentation data, the training and evaluation of the teacher model are all completed through the cloud world model, which is deployed on a cloud supercomputing platform. The cloud world model includes a long-tail scene generator and a digital twin training engine, etc.

[0032] Based on the long-tail scene generator, semantic parsing is performed on the difficult example data to obtain structured scene fragments, and the first generative world model is controlled to generate long-tail scene augmentation data with the structured scene fragments as input conditions. In one embodiment of this application, the process of semantically parsing difficult example data to obtain structured scene fragments based on a long-tail scene generator includes: scanning video frames in the difficult example data using a large visual model (e.g., an image-text matcher) to obtain scene features, including moving targets (such as vehicles and pedestrians), environmental attributes (such as weather type and light intensity), and behavioral logic (such as lane changing and collision); extracting sensor metadata from the difficult example data, including unmanned logistics vehicle positioning information, data collection timestamps, and unmanned logistics vehicle speed; performing deep semantic analysis on the scene features and sensor metadata using a large language model in the long-tail scene generator to obtain the causes of the difficult examples; performing logical completion and parameter correction on the scene features and sensor metadata based on the causes of the difficult examples and a predefined unmanned logistics domain knowledge graph to determine the spatial location range and motion parameter range of each moving target; and encapsulating the spatial location range and motion parameter range of each moving target into a structured data format (such as JSON or XML) to generate structured scene fragments.

[0033] In one embodiment of this application, if the first generative world model includes a spatiotemporal encoder, a conditional injection module, a diffusion generation module, and a spatiotemporal decoder, then the process of controlling the first generative world model to generate long-tail scene augmentation data with structured scene fragments as input conditions includes: acquiring moving target information and environmental parameters in the structured scene fragments, and using the scene parameters in the structured scene fragments as target parameters; compressing and encoding the original image sequence of the structured scene fragments using the spatiotemporal encoder to obtain basic latent space features; constructing a second random noise tensor in the latent space, wherein the dimension of the second random noise tensor is consistent with the dimension of the basic latent space features; superimposing the second random noise tensor with the basic latent space features and controlling the noise. The amplitude of the superposition is used to obtain the basic latent space features after noise superposition. The target parameters and moving target information are feature-encoded through a conditional injection module to obtain a conditional feature embedding vector. Using a cross-attention mechanism, the conditional feature embedding vector is fused with the basic latent space features after noise superposition to obtain a second attention fusion feature. The diffusion generation module, guided by the conditional feature embedding vector, denoises the second attention fusion feature to obtain latent space augmented video features. The latent space augmented video features are decoded through a spatiotemporal decoder to obtain long-tail scene augmented data. In the above process, noise superposition and conditional injection achieve multi-dimensional perturbation of the hard example data (e.g., environmental parameter perturbation, moving target perturbation, etc.). The generation of long-tail scene augmented data can be derived and extended based on already collected real hard example data. Only a small number of real hard examples are needed to generate a large amount of long-tail scene data of the same type but with different parameters. This ensures the close fit between the generated data and the real scene, and can quickly cover long-tail conditions with more parameter ranges, further improving the richness of the dataset.

[0034] The long-tail scene augmentation data and multimodal long-tail scene data are merged to obtain merged data. Based on a digital twin training engine, the merged data is injected into a pre-built simulation environment to obtain second long-tail scene state data. The teacher model is trained and evaluated using the second long-tail scene state data to obtain an adjusted teacher model. In one embodiment of this application, the process of injecting the merged data into a pre-built simulation environment based on a digital twin training engine to obtain second long-tail scene state data is the same as the process of injecting multimodal long-tail scene data into a pre-built simulation environment based on a digital twin training engine to obtain first long-tail scene state data. The process of training and evaluating the teacher model using the second long-tail scene state data to obtain an adjusted teacher model is the same as the process of training and evaluating a first generative world model using the first long-tail scene state data to obtain the teacher model. By adjusting the teacher model, new challenging case data accumulated from real-world driving can be continuously incorporated, constantly optimizing the teacher model's scenario generation and intelligent driving decision-making capabilities. This forms a dynamic iterative closed loop of challenging case data collection, long-tail scenario expansion data generation, teacher model training and evaluation, and second student model deployment, allowing the intelligent driving capabilities of unmanned logistics vehicles to continuously improve as the mileage increases.

[0035] The adjusted teacher model undergoes knowledge distillation to obtain a second student model, which is then deployed on the vehicle for decision-making during the autonomous logistics vehicle's operation. In one embodiment of this application, the process of performing knowledge distillation on the adjusted teacher model to obtain the second student model is the same as the process of performing knowledge distillation on the teacher model to obtain the first student model, and the process of deploying the second student model on the vehicle is the same as the process of deploying the first student model on the vehicle.

[0036] In one embodiment of this application, after obtaining the adjusted teacher model, knowledge distillation is performed on the adjusted teacher model. Once the decision-making ability of the adjusted teacher model reaches a preset evaluation threshold, a second student model is output and deployed on the vehicle. This avoids frequent iterations and upgrades due to insufficient improvement in the teacher model, reducing unnecessary OTA traffic consumption and vehicle deployment overhead. Through this dynamic iteration mechanism, the intelligent driving model of the unmanned logistics vehicle can continuously absorb new problems encountered in real-world road operations, constantly optimizing its decision-making robustness for various long-tail, rare scenarios, and gradually improving the safety and efficiency of unmanned logistics delivery in complex and open environments. The training method of this application relies on the powerful generation and computing capabilities of cloud-based generative world models. It not only solves the industry pain point of scarce data in long-tail scenarios but also meets the deployment requirements of vehicle-side edge computing through knowledge distillation. Simultaneously, it forms a complete data closed-loop iteration chain, providing a feasible technical path for the continuous evolution of the intelligent driving capabilities of unmanned logistics vehicles.

[0037] In one embodiment of this application, the process of determining whether the real-time driving status, decision control result, or perception information meets the criteria for difficult data determination includes:

[0038] If the decision control result includes an optimal planning trajectory and a suboptimal planning trajectory, and the cost difference between the cost of the optimal planning trajectory and the cost of the suboptimal planning trajectory is less than a preset difference threshold, then the decision control result is determined to meet the hard case data judgment condition. In one embodiment of this application, the preset difference threshold is set according to the actual situation, for example, 0.1, 0.15, etc. The formula for calculating the cost difference is as follows:

[0039] Equation (1)

[0040] in, Indicates the difference in costs, This represents the cost of the suboptimal programming trajectory. This represents the cost of the optimal planning trajectory.

[0041] If the decision control result includes a prediction confidence level, and the prediction confidence level is less than a preset confidence threshold, then the decision control result is determined to meet the criteria for difficult case data. In one embodiment of this application, the preset confidence threshold is set according to the actual situation, for example, 0.6. The formula for calculating the prediction confidence level is as follows:

[0042] Equation (2)

[0043] in, Indicates the prediction confidence level. Entropy represents the predicted probability distribution. This represents the normalization function.

[0044] The formula for calculating the entropy of the predicted probability distribution is as follows:

[0045] Equation (3)

[0046] in, Entropy represents the predicted probability distribution. Indicates the first The predicted probability distribution of each prediction item.

[0047] If the real-time driving status is in automatic emergency braking or forward collision warning state, then the real-time driving status is determined to meet the difficult example data determination conditions. In one embodiment of this application, the automatic emergency braking status can be obtained from devices such as ESC (Electronic Stability Control) and ESP (Electronic Stability Program). The forward collision warning status can be obtained from devices such as a forward-view camera and millimeter-wave radar, which have the function of calculating the collision time based on perception information.

[0048] If the real-time driving state is under manual control, then the real-time driving state is determined to meet the difficult case data determination conditions. In one embodiment of this application, the manual control state can be determined from the rotation signal of the steering wheel angle sensor, the pedal position sensor's pedal input signal, the torque signal of the torque sensor, or the monitoring results of the driver's behavior and actions by the camera.

[0049] If the perceived information includes the location information of the unmanned logistics vehicle, and the difference between the location information and the location information in the preset precision map is greater than a preset distance threshold, then the perceived information is determined to meet the criteria for difficult data determination. In one embodiment of this application, the preset distance threshold is set according to the actual situation, for example, 10 meters.

[0050] In one embodiment of this application, any situation that meets the criteria for determining difficult case data will trigger the upload of difficult case data. When there are multiple situations that meet the criteria for determining difficult case data, they can be triggered according to their respective priorities. For example, the priority of triggering difficult case data upload in automatic emergency braking state or forward collision warning state is higher than that in manual control state; the priority of triggering difficult case data upload in manual control state is higher than that in perception-triggered (the positional difference between the location information and the position information in the preset accuracy map is greater than a preset distance threshold); the priority of perception-triggered difficult case data upload is higher than that in prediction confidence-triggered (the prediction confidence is less than a preset confidence threshold); and the priority of prediction confidence-triggered difficult case data upload is higher than that in decision uncertainty-triggered (the cost difference between the cost of the optimal planning trajectory and the cost of the suboptimal planning trajectory is less than a preset difference threshold).

[0051] In one embodiment of this application, when difficult case data upload is triggered, the real-time driving status, decision control results, and perception information within a preset time range at the trigger time are packaged into a difficult case data package, and uploaded to the cloud supercomputing platform in a data encryption manner to ensure the security of difficult case data upload. If the difficult case data package contains sensitive information such as faces or license plates, it needs to be blurred.

[0052] In one embodiment of this application, the process of training a generative world model with a preset architecture using sample data to obtain a first generative world model includes:

[0053] The sample data is preprocessed to obtain preprocessed sample data. In one embodiment of this application, the preprocessing operations include: deduplication, denoising, and format unification. The deduplication process of the sample data includes: using a perceptual hash algorithm to extract and compare features of the videos in the sample data, identifying redundant video groups with similarity higher than a preset threshold; in the redundant video groups, selecting videos as retained videos according to a preset strategy (e.g., selecting videos with higher resolution, larger file size, or better image quality), and removing the remaining redundant videos, thereby obtaining the deduplicated sample data. The denoising process of the deduplicated sample data includes: using an image quality assessment algorithm to assess the quality of each frame of the deduplicated sample data, and removing images with low assessment quality (e.g., images with blurriness, jitter, or abnormal exposure), thereby obtaining the denoised sample data.

[0054] The pre-trained detection model automatically detects moving targets, static obstacles, and roads in the pre-processed sample data, obtaining sample data with labeled information. In one embodiment of this application, after automatically detecting and obtaining sample data with labeled information, the labeled information can be supplemented and corrected by manual annotation to obtain the final sample data with labeled information. The labeled information includes: the type, bounding box, and trajectory of the moving target; the type of static obstacle; the type of road; the weather type; and the light intensity. The types of moving targets include: vehicles, pedestrians, cyclists, etc.; the types of static obstacles include: bollards, billboards, road posts, temporary construction barriers, etc.; the types of roads include: main roads, non-motorized vehicle lanes, park walkways, intersections, etc.; the weather types include: sunny, rainy, foggy, snowy, etc.; and the light intensity includes: strong light, weak light, no lighting at night, etc.

[0055] The sample data with annotation information is format-converted to obtain converted sample data. In one embodiment of this application, the converted sample data is in COCO (Common Objects in Context) format, KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute Dataset) format, etc.

[0056] The transformed sample data is input into a generative world model with a preset architecture to obtain predicted video data. In one embodiment of this application, the generative world model predicts continuous video frame data within a set time period based on the frame image data in the input transformed sample data, thus completing the deduction of the future state of the input scene.

[0057] With the objective of minimizing the difference between the predicted video data and the transformed sample data, a generative world model with a preset architecture is iteratively trained to obtain a first generative world model. In one embodiment of this application, if the difference between the predicted video data and the transformed sample data is represented by a first loss function, then the expression of the first loss function includes:

[0058] Equation (4)

[0059] in, Denotes the first loss function. This represents the pixel-level reconstruction loss weights. Indicates pixel-level reconstruction loss. Represents the weights of the perceptual feature loss. This represents the loss of perceptual features. Indicates the weight of the time-series consistency loss. This represents the temporal consistency loss. The sum of the pixel-level reconstruction loss weight, the perceptual feature loss weight, and the temporal consistency loss weight equals 1.

[0060] The expression for pixel-level reconstruction loss includes:

[0061] Equation (5)

[0062] in, Indicates pixel-level reconstruction loss. The image represents the predicted video data. The image represents the transformed sample data, and the image of the predicted video data is aligned with the image of the transformed sample data in both time and space. Represents the L1 norm;

[0063] The expressions for the perceptual feature loss weights include:

[0064] Equation (6)

[0065] in, This represents the loss of perceptual features. The image represents the predicted video data. An image representing the transformed sample data. Indicates the first Each level of feature extractor Indicates the first Each level of feature extractor extracts image features from the predicted video data. Indicates the first Each level of feature extractor extracts image features from the transformed sample data. denoted as L2 norm; the feature extractor is a pre-trained neural network model.

[0066] The expressions for timing consistency loss include:

[0067] Equation (7)

[0068] in, Indicates the time-series consistency loss. Indicating the prediction of video data Images of moments Indicating the prediction of video data Images of moments Indicating the transformed sample data Images of moments Indicating the transformed sample data Images of moments This represents the L1 norm.

[0069] In one embodiment of this application, with the goal of minimizing the difference between the predicted video data and the transformed sample data, the generative world model with a preset architecture is iteratively trained so that the first generative world model obtained after training can learn the spatial layout rules of various scenes in real open roads and the temporal evolution rules of moving targets. It has the ability to infer the future scene development and changes based on the existing scene context, providing a generative basis for the subsequent generation of massive and diverse long-tailed difficult example scenes.

[0070] In one embodiment of this application, if the first generative world model includes: a spatiotemporal encoder, a conditional injection module, a diffusion generation module, and a spatiotemporal decoder, then the process of controlling the first generative world model to generate multimodal long-tail scene data with structured generation instructions as input conditions includes:

[0071] The video data required for the structured generation instructions is obtained. In one embodiment of this application, the structured generation instructions include indication information for the required video data, such as the number, content description, and storage location of the indication information.

[0072] Video data is compressed and encoded using a spatiotemporal encoder to obtain latent space features; a first random noise tensor is then constructed within the latent space. In one embodiment of this application, the dimension of the first random noise tensor is consistent with the dimension of the latent space features. The process of compressing and encoding video data using a spatiotemporal encoder to obtain latent space features includes: segmenting the video data into a sequence of image frames and normalizing the pixel values ​​in each frame to obtain a normalized frame sequence; extracting spatial features from the normalized frame sequence using a spatial encoder to obtain a spatial feature map; extracting temporal features from the spatial feature map using a temporal encoder to obtain a spatiotemporal fusion feature map; and compressing the spatiotemporal dimension of the spatiotemporal fusion feature map using a downsampling module to obtain latent space features.

[0073] The first random noise tensor is superimposed with the latent space features to obtain the noise-superimposed latent space features. In one embodiment of this application, the first random noise tensor and the latent space features are superimposed by channel concatenation, and the expression of the noise-superimposed latent space features is as follows:

[0074] Equation (8)

[0075] in, This represents the latent space features after noise superposition. Represents latent space features. Represents the first random noise tensor. This represents the channel splicing function.

[0076] The structured generation instructions are encoded into text using a conditional injection module to obtain text feature embedding vectors. A cross-attention mechanism is then used to calculate the cross-attention between the text feature embedding vectors and the latent space features after noise superposition, resulting in a first attention fusion feature. In one embodiment of this application, the conditional injection module can employ a Transformer-based pre-trained text encoder, such as CLIP (Contrastive Language-Image Pre-training) text encoder or T5 text encoder. The process of calculating the cross-attention between the text feature embedding vectors and the latent space features after noise superposition to obtain the first attention fusion feature includes: projecting the text feature embedding vectors onto the query space to obtain a query vector; projecting the latent space features after noise superposition onto the key space and value space respectively to obtain a key vector and a value vector; and performing weighted attention calculation based on the query vector, key vector, and value vector to obtain the first attention fusion feature.

[0077] The formula for calculating the first attention fusion feature is as follows:

[0078] Equation (9)

[0079] in, This represents the first attention fusion feature. Represents the query vector. Represents the key vector. Represents a value vector. This represents the dimension of the key vector. This represents the normalization function.

[0080] The expression for the query vector is as follows:

[0081] Equation (10)

[0082] in, Represents the query vector. Indicates querying the projection matrix. This represents the text feature embedding vector.

[0083] The expression for the key vector is as follows:

[0084] Equation (11)

[0085] in, Represents the key vector. Represents the key projection matrix. This represents the latent space features after noise superposition.

[0086] The expression for the value vector is as follows:

[0087] Equation (12)

[0088] in, Represents a value vector. Represents the value projection matrix, This represents the latent space features after noise superposition.

[0089] The diffusion generation module, guided by text feature embedding vectors, denoises the first attention-fused features to obtain latent space video features. In one embodiment of this application, the diffusion generation module can employ a spatiotemporal diffusion model based on the U-Net architecture to achieve the process of denoising the first attention-fused features using text feature embedding vectors to obtain latent space video features.

[0090] The spatiotemporal decoder decodes latent space video features to obtain multimodal long-tail scene data. In one embodiment of this application, the spatiotemporal decoder may be a decoder based on a 3D convolutional neural network, a generative decoder based on DiffusionTransformer (DiT), etc.

[0091] In one embodiment of this application, the multimodal long-tail scene data includes various types of data such as image information and radar point clouds. The generation of multimodal long-tail scene data is based on the regular characteristics of real scenes and is virtual scene data that conforms to real physical rules under the guidance of instructions. It can cover various rare long-tail scenes that are difficult to collect in reality, effectively solve the problems of insufficient distribution of long-tail difficult examples and high collection costs in real road data, and provide sufficient and diverse samples for intelligent driving model training.

[0092] In one embodiment of this application, if the pre-built simulation environment is constructed based on a map of preset accuracy, and the pre-built simulation environment conforms to the dynamic constraints in the vehicle dynamics model and the traffic flow constraints in the traffic flow model, then the process of injecting multimodal long-tail scene data into the pre-built simulation environment based on the digital twin training engine to obtain the first long-tail scene state data includes:

[0093] The visual features of the multimodal long-tail scene data are compared with a pre-constructed precision map in the simulation environment to determine the target location of the long-tail scene in the pre-constructed simulation environment. Then, using the distortion model, camera intrinsic matrix, and camera extrinsic matrix in the sensor model, the multimodal long-tail scene data is mapped to the target location to obtain aligned multimodal long-tail scene data. In one embodiment of this application, the pre-constructed precision map is a high-precision map, etc. If the visual features include semantic object features (e.g., vehicles, pedestrians, traffic lights) and geometric depth features (e.g., wall distance, road surface undulation), the process of comparing the visual features of multimodal long-tail scene data with a pre-built simulation environment with a preset precision map to determine the target position of the long-tail scene in the pre-built simulation environment includes: semantically associating the semantic object features with the vector layers in the preset precision map to obtain a candidate position set; under the preset of the candidate position set, spatially registering the geometric depth features with the 3D point cloud data in the preset precision map to obtain the registration result; and calculating the target pose of the long-tail scene relative to the global coordinate system of the pre-built simulation environment using a pose calculation algorithm (such as the PnP algorithm or the iterative nearest point algorithm) to obtain the target position of the long-tail scene in the pre-built simulation environment. The process of mapping multimodal long-tail scene data to the target location using the distortion model, camera intrinsic matrix, and camera extrinsic matrix in the sensor model to obtain aligned multimodal long-tail scene data includes: inputting distortion coefficients (including radial and tangential distortion coefficients) and image data from the multimodal long-tail scene data into the distortion model in the sensor model to obtain distortion-free image data; based on the camera intrinsic matrix (including focal length and principal point coordinates) and the extrinsic matrix of the camera relative to the target position, mapping the pixel coordinates in the distortion-free image data to the three-dimensional world coordinate system where the target position is located through back projection transformation to obtain mapped image data; and spatially registering the mapped image data with radar point cloud data in the three-dimensional world coordinate system to obtain aligned multimodal long-tail scene data.

[0094] The aligned multimodal long-tail scene data is used to perform state calculation to obtain the environmental parameters of the long-tail scene and the motion state of the moving targets in the long-tail scene. In one embodiment of this application, the process of performing state calculation on the aligned multimodal long-tail scene data includes: performing temporal recursion and state update on the motion state (such as position and velocity) of the moving targets in the aligned multimodal long-tail scene data using an extended Kalman filter algorithm or an unscented Kalman filter algorithm to obtain the motion state of the moving targets in the long-tail scene; using a deep learning network model based on an attention mechanism (e.g., Transformer architecture) to extract semantic and geometric information from the aligned multimodal long-tail scene data, and parsing the environmental parameters (e.g., road curvature, obstacle type, etc.) of the long-tail scene through a regression head or decoder.

[0095] The motion state of a moving target is corrected using a vehicle dynamics model to obtain a corrected motion target state. This corrected motion target state, along with environmental parameters, is then input into a traffic flow model to obtain the interactive response strategy for the moving target. In one embodiment of this application, during the correction of the motion state of the moving target using the vehicle dynamics model, the feasible range of speed and acceleration in the motion state is verified through the constraints of tire forces, air resistance, and the power transmission system in vehicle dynamics. Abnormal values ​​that do not conform to the laws of physical motion are eliminated. Simultaneously, the position and heading angle of the moving target are fine-tuned according to the motion equations of vehicle dynamics to ensure that the corrected state conforms to the motion characteristics of a real vehicle. After the corrected motion target state and environmental parameters are input into the traffic flow model, the traffic flow model calculates the interactive response strategy corresponding to each moving target based on the current road traffic density, traffic rules, and the interaction relationships between different moving targets. Examples of strategies include following and decelerating, changing lanes and yielding, and stopping to give way.

[0096] In one embodiment of this application, the dynamic constraints include: steering system dynamic constraints, braking system dynamic constraints, and drive system dynamic constraints; traffic flow constraints include: following model constraints, lane-changing model constraints, and pedestrian model constraints. The process of pre-constructing the simulation environment includes: collecting sensor data, vehicle dynamics data, and traffic behavior data; denoising the sensor data, vehicle dynamics data, and traffic behavior data; experimentally calibrating the denoised vehicle dynamics data to obtain a vehicle dynamics model; fitting the denoised traffic behavior data to obtain a traffic flow model; rendering the physics engine using the denoised sensor data as a benchmark against a pre-defined precision map, and integrating the vehicle dynamics model and traffic flow model into the physics engine; and performing consistency verification on the integrated simulation environment based on real data to ensure the accuracy of the simulation environment.

[0097] The corrected moving target state, environmental parameters, and interaction response strategy are merged to obtain the first long-tail scene state data. In one embodiment of this application, all state information of the virtual long-tail scene in the simulation environment is fully preserved, providing a complete and realistic simulation foundation for the closed-loop training of the subsequent unmanned logistics vehicle intelligent driving model.

[0098] In one embodiment of this application, the process of training and evaluating a first generative world model using first long-tail scene state data to obtain a teacher model includes:

[0099] A second generative world model is obtained by training a first generative world model using pre-set expert data, which includes driving rule data and driving experience data. In one embodiment of this application, training the first generative world model using pre-set expert data helps the first generative world model learn the decision-making logic and behavioral patterns of experts in complex long-tail scenarios, making the generated scenario data more closely resemble the actual situation of real human driving and traffic interaction.

[0100] The second generative world model is obtained by training the first long-tail scene state data. In one embodiment of this application, training the second generative world model with the first long-tail scene state data helps the second generative world model to continuously iterate and optimize its ability to predict future scene evolution trends during simulation interaction, further improving the physical rationality and interactive realism of the generated scene.

[0101] The second generative world model is subjected to visual fidelity evaluation to obtain a visual fidelity score; the second generative world model is subjected to physical dynamics compliance evaluation to obtain a physical dynamics compliance score; and the second generative world model is subjected to long-tail scene scarcity evaluation to obtain a long-tail scene scarcity score. In one embodiment of this application, the process of visual fidelity evaluation of the second generative world model includes: calculating the Fraser initial distance, cosine similarity, and learned perceptual image patch similarity between the generated image and the real image; and performing a weighted summation of the Fraser initial distance, cosine similarity, and learned perceptual image patch similarity to obtain a visual fidelity score. The formula for calculating the visual fidelity score is as follows:

[0102] Equation (13)

[0103] in, Indicates visual fidelity score. Indicates the weight of the distributional similarity score. The distributional similarity score is represented by the Fraser initial distance between the generated image and the real image. This indicates the weighting of perceived quality and aesthetic ratings. The perceived quality and aesthetic score are represented by the cosine similarity between the generated image and the real image. Indicates the perceived difference distance weight. The perceived difference distance is represented by the similarity of learned perceptual image patches between the generated image and the real image. The normalization function is represented by the sum of the weights of the distributional similarity score, the perceived quality and aesthetic score, and the perceived difference distance, which equals 1.

[0104] The process of calculating the initial Fréchet distance between the generated image and the real image includes: extracting feature vectors from the generated image using a feature extractor to obtain a feature vector of a preset dimension; extracting feature vectors from the real image using a feature extractor to obtain a feature vector of a preset dimension; calculating the mean vector of the generated image feature vectors and the covariance between the generated image feature vectors based on the preset dimension feature vectors, and forming a covariance matrix using the covariance between the generated image feature vectors; calculating the mean vector of the real image feature vectors and the covariance between the real image feature vectors based on the preset dimension feature vectors, and forming a covariance matrix using the covariance between the real image feature vectors; and determining the initial Fréchet distance between the generated image and the real image based on the mean vector of the generated image feature vectors, the covariance matrix of the generated image feature vectors, the mean vector of the real image feature vectors, and the covariance matrix of the real image feature vectors. The formula for calculating the initial Fréchet distance between the generated image and the real image is shown below:

[0105] Equation (14)

[0106] in, This represents the initial Frescher distance between the generated image and the real image. The mean vector representing the feature vectors of a real image. This represents the mean vector of the generated image feature vectors. Denotes the square of the L2 norm. The covariance matrix represents the feature vectors of a real image. The covariance matrix represents the feature vector of the generated image. Represents the trace of a matrix. Represents the square root of a matrix.

[0107] The process of calculating the cosine similarity between the generated image and the real image includes: after obtaining the feature vectors of the generated image and the real image in a preset dimension, calculating the cosine similarity based on these two feature vectors. The formula for calculating the cosine similarity between the generated image and the real image is shown below:

[0108] Equation (15)

[0109] in, This represents the cosine similarity between the generated image and the real image. This represents the feature vector of the generated image in a preset dimension. A feature vector representing a predefined dimension of a real image. Represents the L2 norm. Represents the first generation of the generated image 1 eigenvector The first representing the real image 1 eigenvector This represents the total number of eigenvectors.

[0110] The process of calculating the learned perceptual patch similarity between a generated image and a real image includes: extracting feature maps of different levels from the generated image using a feature extractor; extracting feature maps of different levels from the real image using a feature extractor; and determining the learned perceptual patch similarity between the generated image and the real image based on the feature maps of different levels in both images. The formula for calculating the learned perceptual patch similarity between the generated image and the real image is shown below:

[0111] Equation (16)

[0112] in, This represents the learned perceptual image patch similarity between the generated image and the real image. Indicates the first The height of the hierarchical feature map, Indicates the first Width of the hierarchical feature map Indicates the first Channel weights at different levels Indicates the generated image at the 1st... Hierarchy, spatial location The feature vector at that location, Represents the real image in the first... Hierarchy, spatial location The feature vector at that location, This represents the square of the L2 norm.

[0113] In one embodiment of this application, the process of evaluating the physical dynamics compliance of the second generative world model includes: calculating the trajectory physics error between the generated trajectory and the real trajectory; scoring the collision and contact rationality of the generated trajectory using a pre-trained model (e.g., VLMs (Vision-Language Models) or a discriminator) to obtain a collision and contact rationality score, and normalizing the collision and contact rationality score; calculating the optical flow smoothness error of the generated trajectory using frame interpolation; detecting physical violations of the generated video using a modular detection method based on specific physical rules to obtain a physical logic consistency score, and normalizing the physical logic consistency score; the modular detection method based on specific physical rules can be: geometry and physics engine simulation method, kinematics and dynamics analysis method, temporal coherence and object persistence detection method, semantic-physical consistency check method, etc. After obtaining each score, the scores are normalized, and then the normalized scores are weighted and summed to obtain the physical logic consistency score.

[0114] The physical dynamics compliance score is obtained by weighted summing of the trajectory physics error, the normalized collision and contact rationality score, the optical flow smoothness error, and the normalized physical logic consistency score. The formula for calculating the physical dynamics compliance score is shown below:

[0115] Equation (17)

[0116] in, Indicates the physical dynamics compliance score. Represents the physical error weights of the trajectory. The physical error of the trajectory is represented by the dynamic time-warped distance between the generated trajectory and the actual trajectory. This indicates the weighting of the collision and contact rationality scores. This represents the normalized collision and contact plausibility score. Indicates the weight of optical flow smoothness error. This represents the optical flow smoothness error. The optical flow smoothness error of the generated trajectory is used to measure the continuity of the motion field. This indicates the weight of the physical-logical consistency score. The normalized physical-logical consistency score is used to characterize physical violations in the generated video. The sum of the trajectory physical error weight, the collision and contact rationality score weight, and the optical flow smoothness error weight is equal to 1.

[0117] The formula for calculating the trajectory physical error is as follows:

[0118] Equation (18)

[0119] in, Indicates the physical error of the trajectory. This indicates the number of matching point pairs between the generated trajectory and the actual trajectory. Indicates the first Euclidean distance between matching point pairs Indicates the first A number of matching point pairs. Formula (18) represents the average Euclidean distance between the generated trajectory and the true trajectory under the optimal alignment path. By introducing a normalization factor... This eliminates the influence of trajectory length on error assessment, thus enabling an objective characterization of the average physical position error within a unit time step.

[0120] The formula for calculating optical flow smoothness error is as follows:

[0121] Equation (19)

[0122] in, Indicates optical flow smoothness error. This represents the total number of sampling points. Indicates the first At each position, the odd-numbered frame values ​​of the generated optical flow are... Indicates the first At each position, predict the odd-numbered frame values ​​of optical flow.

[0123] The process of evaluating the scarcity of long-tail scenes in the second generative world model includes: calculating the probability of a single element appearing in the overall dataset, calculating the probability of a single element appearing in the presence of a preset context, and calculating the long-tail scene scarcity score based on the probability of a single element appearing in the overall dataset and the probability of a single element appearing in the presence of a preset context.

[0124] The formula for calculating the scarcity score of long-tail scenarios includes:

[0125] Equation (20)

[0126] in, This indicates the scarcity score of long-tail scenarios. Represents the marginal probability weights. Marginal probability represents the probability of a single feature appearing in the entire dataset; it is the ratio of the total number of times a single feature appears to the total number of times all features appear. Indicates the scarcity of a single factor. Represents the conditional probability weights. This represents the conditional probability, used to characterize the probability of a single element appearing in the presence of a predefined context. It is the ratio of the number of times a single element and the predefined context appear simultaneously to the total number of times the predefined context appears simultaneously. Used to characterize context scarcity This represents the Sigmoid function, where the sum of the marginal probability weights and the conditional probability weights equals 1. The preset context can be scenarios such as highways, intersections, and tunnels.

[0127] The visual fidelity score, physical dynamics compliance score, and long-tail scene scarcity score are weighted and summed to obtain the comprehensive score of the second generative world model. In one embodiment of this application, the formula for calculating the comprehensive score includes:

[0128] Equation (21)

[0129] in, This indicates the overall score. Indicates the weight of the visual fidelity score. Indicates visual fidelity score. Indicates the weight of the physical dynamics compliance score. Indicates the physical dynamics compliance score. This indicates the weighting of the scarcity score for long-tail scenarios. This represents the scarcity score of long-tail scenarios.

[0130] Based on the comprehensive score, the second generative world model is selected to obtain the teacher model. In one embodiment of this application, the second generative world model with the highest comprehensive score is used as the teacher model. This ensures that the output world model has scene generation quality, physical rationality, and long-tail scarcity, better meeting the needs of subsequent intelligent driving model training. It can effectively improve the decision robustness and generalization ability of the unmanned logistics vehicle intelligent driving model in rare and dangerous scenarios.

[0131] Figure 3 This is a schematic diagram illustrating a world-model-based intelligent driving model training method for unmanned logistics vehicles, as shown in another exemplary embodiment of this application. Figure 3The training method for the intelligent driving model of unmanned logistics vehicles based on the world model includes: (1) constructing a generative world model with a preset architecture through the world model construction module; (2) training the generative world model with the preset architecture through sample data to obtain the first generative world model; (3) performing semantic parsing on the long-tail scene data generation instructions based on the long-tail scene generator to obtain structured generation instructions, and controlling the first generative world model to generate multimodal long-tail scene data with the structured generation instructions as input conditions; (4) injecting the multimodal long-tail scene data into the pre-constructed simulation environment based on the digital twin training engine to obtain the first long-tail scene state data; training and evaluating the first generative world model through the first long-tail scene state data to obtain the teacher model; (5) performing knowledge distillation on the teacher model to obtain the first student model, and deploying the first student model on the vehicle; (6) the difficult example detection module based on the real-time driving status, (7) If the real-time driving status, decision control results, or perception information meet the criteria for difficult case data, then the real-time driving status, decision control results, and perception information are sent back to the cloud supercomputing platform as difficult case data; (8) Based on the long-tail scene generator, the difficult case data is semantically parsed to obtain structured scene fragments, and the first generative world model is controlled to generate long-tail scene augmentation data with the structured scene fragments as input conditions; (9) The long-tail scene augmentation data is merged with the multimodal long-tail scene data to obtain merged data; Based on the digital twin training engine, the merged data is injected into the pre-built simulation environment to obtain the second long-tail scene state data; The teacher model is trained and evaluated through the second long-tail scene state data to obtain the adjusted teacher model; (10) The adjusted teacher model is knowledge distilled to obtain the second student model, and the second student model is deployed on the vehicle.

[0132] In one embodiment of this application, steps (1) to (5) are executed through a cloud supercomputing platform, steps (6) to (7) are executed through a vehicle-side intelligent driving system, and steps (8) to (10) are executed through a cloud supercomputing platform. Therefore, an automated and continuously evolving closed loop is formed from "data generation - model training - vehicle-side deployment - difficult example feedback - difficult example data amplification - model adjustment - redeployment", realizing the collaborative closed loop processing flow of the cloud supercomputing platform and the vehicle-side intelligent driving system.

[0133] Figure 4 This is a flowchart illustrating a difficult case detection method as shown in an exemplary embodiment of this application, such as... Figure 4As shown, the difficult case detection process includes: (1) obtaining the real-time driving status and perception information of the unmanned logistics vehicle, as well as the decision control results of the first student model; (2) if the real-time driving status is an automatic emergency braking state or a forward collision warning state, then the real-time driving status is determined to meet the difficult case data judgment conditions; (3) if the real-time driving status is a manual control state, then the real-time driving status is determined to meet the difficult case data judgment conditions; (4) if the perception information includes: the positioning information of the unmanned logistics vehicle, and the position difference between the positioning information and the position information in the preset precision map is greater than the preset distance threshold, then the perception information is determined to meet the difficult case data judgment conditions; (5) if the decision control results include: prediction confidence, and the prediction confidence is less than the preset confidence threshold, then the decision control results are determined to meet the difficult case data judgment conditions; (6) if the decision control results include: the optimal planning trajectory and the suboptimal planning trajectory, and the cost difference between the cost of the optimal planning trajectory and the cost of the suboptimal planning trajectory is less than the preset difference threshold, then the decision control results are determined to meet the difficult case data judgment conditions.

[0134] The advantages of the long-tail scenario data acquisition method in this application compared with traditional long-tail scenario data acquisition methods are as follows:

[0135] Table 1

[0136]

[0137] As can be seen from Table 1, the long-tail scenario data acquisition method of this application has lower data acquisition cost, shorter data acquisition cycle, wider long-tail scenario coverage, and higher data diversity compared with the traditional long-tail scenario data acquisition method.

[0138] The simulation technology of this application has the following advantages compared with traditional simulation technology:

[0139] Table 2

[0140]

[0141] As can be seen from Table 2, the simulation technology of this application has a higher degree of physical realism, more complex interaction logic, a significantly reduced Sim-to-Real Gap, and a greatly improved scene generation capability compared to traditional simulation technology.

[0142] The advantages of the world model in this application compared to existing world models are as follows:

[0143] Table 3

[0144]

[0145] As can be seen from Table 3, the world model in this application is more lightweight, has stronger scene generation capabilities, a more complete closed-loop iteration mechanism, and higher engineering feasibility than existing world models.

[0146] This application generates multimodal long-tail scene data through a cloud-based world model. The first generative world model is trained and evaluated using this multimodal long-tail scene data. When difficult examples are generated, the model is augmented. The teacher model is then trained and evaluated using the augmented long-tail scene data and the multimodal long-tail scene data. This significantly improves the adaptability of unmanned logistics vehicles to long-tail scenes, reduces reliance on real-world road data, and accelerates technological iteration. It possesses significant technical advantages and application value, thereby significantly improving the performance and safety of unmanned logistics vehicles and reducing operating costs. Furthermore, the technical solution of this application is reproducible and applicable to different scenarios and vehicle models.

[0147] The following describes an embodiment of the apparatus described in this application, which can be used to execute a world-model-based intelligent driving model training system for unmanned logistics vehicles as described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the world-model-based intelligent driving model training method for unmanned logistics vehicles described above in this application.

[0148] Figure 5 This is a block diagram illustrating an exemplary embodiment of the world model-based intelligent driving model training system for unmanned logistics vehicles.

[0149] like Figure 5 As shown, the exemplary world-model-based unmanned logistics vehicle intelligent driving model training system 500 includes:

[0150] The data acquisition module 501 is used to acquire historical video data collected by the unmanned logistics vehicle during its operation and long-tail scene data generation instructions input by the target personnel; the target personnel include: developers and operations personnel;

[0151] The first model training module 502 is used to merge historical video data with a preset number of Internet video data as sample data; and to train a generative world model with a preset architecture using the sample data to obtain the first generative world model.

[0152] The data generation module 503 is used to perform semantic parsing on the long-tail scene data generation instructions based on the long-tail scene generator to obtain structured generation instructions, and control the first generative world model to generate multimodal long-tail scene data with the structured generation instructions as input conditions.

[0153] The second model training module 504 is used to inject multimodal long-tail scene data into a pre-built simulation environment based on the digital twin training engine to obtain the first long-tail scene state data; and to train and evaluate the first generative world model using the first long-tail scene state data to obtain the teacher model.

[0154] The model deployment module 505 is used to perform knowledge distillation on the teacher model to obtain the first student model, and deploy the first student model on the vehicle to make decisions and control during the operation of the unmanned logistics vehicle.

[0155] In one embodiment of this application, the historical video data is data actually collected by the unmanned logistics vehicle during its operation, and the long-tail scene data generation instruction is in natural language form. For example, "narrow road at night, tricycle going in the opposite direction" is used as the long-tail scene data generation instruction.

[0156] In one embodiment of this application, the preset quantity is set according to actual conditions, and the generative world model of the preset architecture is a large-scale generative world model based on a diffusion model or an autoregressive Transformer architecture. The generative world model of the preset architecture includes: a preset spatiotemporal encoder, a preset conditional injection module, a preset diffusion generation module, and a preset spatiotemporal decoder. Training the generative world model of the preset architecture with sample data helps to improve the generative world model's basic cognitive ability of the real road traffic environment, allowing the generative world model to learn the environmental generation rules under different road conditions, weather, and lighting, laying the foundation for subsequent generation of long-tail scenes that conform to real physical rules.

[0157] In one embodiment of this application, the process of semantically parsing long-tail scene data generation instructions based on a long-tail scene generator to obtain structured generation instructions includes: inputting the long-tail scene data generation instructions into a large language model to extract scene features: including moving targets (such as vehicles and pedestrians), environmental attributes (such as weather type and light intensity), and behavioral logic (such as lane changing and collision); based on a predefined domain knowledge graph or scene template library, performing logical completion and parameterized mapping on the identified scene features to determine the spatial position, trajectory, and interaction relationship of each moving target; encapsulating the spatial position, trajectory, and interaction relationship of each moving target into a structured data format (such as JSON, XML, or scene graph structure) to generate structured generation instructions. The generation of multimodal long-tail scene data solves the problems of high cost and difficulty in acquiring scarce long-tail data in real-world scenarios, and can obtain different types of extreme working condition data on demand through the generation method of this application, significantly expanding the coverage of the training dataset. Multimodal long-tail scene data includes multi-dimensional data such as visual images, point cloud data, target motion trajectories, and environmental sensor response parameters, which can directly support model training and testing in subsequent simulation environments.

[0158] In one embodiment of this application, multimodal long-tail scene data is injected into a pre-constructed simulation environment based on a digital twin training engine. This facilitates the verification of the rationality of the multimodal long-tail scene data in terms of vehicle dynamics and the movement patterns of traffic participants through the virtual-real mapping relationship of the digital twin. Simultaneously, the generated first long-tail scene state data can be adapted to the simulation interaction training process of the intelligent driving model and directly used for iterative testing of the intelligent driving decision model. When training and evaluating the first generative world model based on the first long-tail scene state data, multi-dimensional evaluation indicators including physical dynamics compliance, visual fidelity, and long-tail scene scarcity are introduced to filter out low-quality generated data that does not conform to real physical rules. Simultaneously, the generation capability of the first generative world model is optimized in reverse, ultimately resulting in a teacher model with reliable data generation quality and high scene fidelity.

[0159] In one embodiment of this application, knowledge distillation is a model compression method that transfers the generalization ability and decision-making knowledge learned by a large-parameter teacher model to a small-parameter student model. It is particularly suitable for transferring decision logic in long-tail scenarios. Knowledge distillation preserves the decision-making accuracy of the teacher model in various long-tail scenarios while compressing the model's parameter count to a range suitable for vehicle-side edge computing. This ensures that the model's inference speed meets the real-time decision-making requirements of unmanned logistics vehicles, ultimately resulting in a first student model that balances performance and inference efficiency. This achieves an efficient collaborative closed loop between high-performance training of the cloud-based world model and real-time inference of the vehicle-side small model. The deployment of the first student model on the vehicle is achieved through OTA (Over-The-Air) technology.

[0160] This application introduces a generative world model and a long-tail scene generator to transform scarce long-tail scene data generation instructions into multimodal long-tail scene data. This reduces reliance on historical video data and generates long-tail scene data on demand, meeting data needs under extreme conditions. It effectively solves the problems of difficult and costly long-tail data acquisition and improves data coverage and diversity. The generative world model with a pre-defined architecture is trained using sample data composed of historical video data and a preset number of internet video data. The first generative world model is trained and evaluated using the first long-tail scene state data, avoiding the flooding of the first long-tail scene state data by massive sample data, thereby improving the accuracy of long-tail scene recognition. By constructing an evaluation system that includes physical dynamics compliance scoring, the physical consistency and realism of the teacher model's generated data are improved, narrowing the "virtual-real gap". By distilling the teacher model into a lightweight student model and deploying it on the vehicle, it is conducive to achieving an efficient closed loop of "cloud training - vehicle deployment", balancing model performance and vehicle real-time performance.

[0161] It should be noted that the world-model-based intelligent driving model training system for unmanned logistics vehicles provided in the above embodiments and the world-model-based intelligent driving model training method for unmanned logistics vehicles provided in the above embodiments belong to the same concept. The specific methods of execution of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the world-model-based intelligent driving model training system for unmanned logistics vehicles provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0162] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A training method for an intelligent driving model of an unmanned logistics vehicle based on a world model, characterized in that, The method includes: The system acquires historical video data collected by the unmanned logistics vehicle during its operation and generates long-tail scene data input by target personnel; the target personnel include: developers and operations personnel. The historical video data is merged with a preset number of internet video data to form sample data; and the generative world model with a preset architecture is trained using the sample data to obtain a first generative world model. Based on the long-tail scene generator, the semantic parsing of the long-tail scene data generation instructions is performed to obtain structured generation instructions, and the first generative world model is controlled to generate multimodal long-tail scene data with the structured generation instructions as input conditions. Based on the digital twin training engine, the multimodal long-tail scene data is injected into a pre-built simulation environment to obtain the first long-tail scene state data; the first generative world model is trained and evaluated using the first long-tail scene state data to obtain the teacher model. The teacher model is subjected to knowledge distillation to obtain a first student model, which is then deployed on the vehicle to make decisions and control the unmanned logistics vehicle during its operation.

2. The training method for the intelligent driving model of unmanned logistics vehicles based on a world model according to claim 1, characterized in that, The process of decision control using the first student model includes: The real-time driving status and perception information of the unmanned logistics vehicle, as well as the decision control results of the first student model, are obtained. If the real-time driving status, the decision control result, or the perception information meets the criteria for difficult case data, then the real-time driving status, the decision control result, and the perception information are taken as difficult case data. Based on the long-tail scene generator, semantic parsing is performed on the difficult example data to obtain structured scene fragments, and the first generative world model is controlled to generate long-tail scene augmentation data with the structured scene fragments as input conditions. The long-tail scene augmentation data is merged with the multimodal long-tail scene data to obtain merged data; based on the digital twin training engine, the merged data is injected into the pre-built simulation environment to obtain second long-tail scene state data; the teacher model is trained and evaluated using the second long-tail scene state data to obtain an adjusted teacher model. The adjusted teacher model is subjected to knowledge distillation to obtain a second student model, which is then deployed on the vehicle to make decisions during the operation of the unmanned logistics vehicle.

3. The training method for the intelligent driving model of unmanned logistics vehicles based on a world model according to claim 2, characterized in that, The process of determining whether the real-time driving status, the decision control result, or the perceived information meets the criteria for difficult data determination includes: If the decision control result includes an optimal planning trajectory and a suboptimal planning trajectory, and the cost difference between the cost of the optimal planning trajectory and the cost of the suboptimal planning trajectory is less than a preset difference threshold, then the decision control result is determined to meet the difficult case data determination condition. If the decision control result includes a prediction confidence level, and the prediction confidence level is less than a preset confidence threshold, then the decision control result is determined to meet the difficult case data determination condition. If the real-time driving status is an automatic emergency braking status or a forward collision warning status, then the real-time driving status is determined to meet the difficult case data determination conditions. If the real-time driving status is in a manually controlled state, then the real-time driving status is determined to meet the difficult case data determination conditions. If the perceived information includes the location information of the unmanned logistics vehicle, and the location difference between the location information and the location information in the preset precision map is greater than a preset distance threshold, then the perceived information is determined to meet the difficult case data determination condition.

4. The training method for an intelligent driving model of an unmanned logistics vehicle based on a world model according to any one of claims 1-3, characterized in that, The process of training a generative world model with a preset architecture using the sample data to obtain a first generative world model includes: The sample data is preprocessed to obtain preprocessed sample data; the preprocessing operations include: deduplication, noise reduction, and format unification. The pre-trained detection model automatically detects moving targets, static obstacles, and roads in the pre-processed sample data to obtain sample data with annotation information. The annotation information includes: the type, bounding box, and trajectory of the moving target; the type of the static obstacle; the type of the road; the weather type; and the light intensity. The format of the labeled sample data is converted to obtain the converted sample data. The transformed sample data is input into the generative world model of the preset architecture to obtain predicted video data; With the goal of minimizing the difference between the predicted video data and the transformed sample data, the generative world model of the preset architecture is iteratively trained to obtain the first generative world model.

5. The method for training an intelligent driving model for unmanned logistics vehicles based on a world model according to claim 4, characterized in that, If the difference between the predicted video data and the transformed sample data is represented by a first loss function, then the expression of the first loss function includes: ， in, Denotes the first loss function. This represents the pixel-level reconstruction loss weights. Indicates pixel-level reconstruction loss. Represents the weights of the perceptual feature loss. This represents the loss of perceptual features. Indicates the weight of the time-series consistency loss. This represents the loss of timing consistency. The expression for the pixel-level reconstruction loss includes: ， in, Indicates pixel-level reconstruction loss. The image representing the predicted video data. The image represents the transformed sample data, and the image of the predicted video data is aligned with the image of the transformed sample data in both time and space. Represents the L1 norm; The expression for the perceptual feature loss weights includes: ， in, This represents the loss of perceptual features. The image representing the predicted video data. An image representing the converted sample data. Indicates the first Each level of feature extractor Indicates the first Each level of feature extractor extracts image features from the predicted video data. Indicates the first Each level of feature extractor extracts image features from the transformed sample data. The L2 norm is represented; the feature extractor is a pre-trained neural network model. The expression for the timing consistency loss includes: ， in, Indicates the time-series consistency loss. Indicating that the predicted video data contains Images of moments Indicating that the predicted video data contains Images of moments This indicates that in the transformed sample data Images of moments This indicates that in the transformed sample data Images of moments This represents the L1 norm.

6. The method for training an intelligent driving model for unmanned logistics vehicles based on a world model according to any one of claims 1-3, characterized in that, If the first generative world model includes: a spatiotemporal encoder, a conditional injection module, a diffusion generation module, and a spatiotemporal decoder, then the process of controlling the first generative world model to generate multimodal long-tail scene data with the structured generation instructions as input conditions includes: Obtain the video data required for the structured generation instructions; The video data is compressed and encoded using the spatiotemporal encoder to obtain latent space features; and a first random noise tensor is constructed within the latent space; the dimension of the first random noise tensor is consistent with the dimension of the latent space features. The first random noise tensor is superimposed with the latent space features to obtain the latent space features after noise superposition. The structured generation instructions are text-encoded by the conditional injection module to obtain a text feature embedding vector; the text feature embedding vector and the latent space features after noise superposition are cross-attention calculated by the cross attention mechanism to obtain the first attention fusion feature. The diffusion generation module uses the text feature embedding vector as a guide to denoise the first attention fusion feature to obtain latent space video features. The spatiotemporal decoder decodes the latent space video features to obtain the multimodal long-tail scene data.

7. The training method for an intelligent driving model of an unmanned logistics vehicle based on a world model according to any one of claims 1-3, characterized in that, If the pre-built simulation environment is constructed based on a map of preset accuracy, and the pre-built simulation environment conforms to the dynamic constraints in the vehicle dynamics model and the traffic flow constraints in the traffic flow model, then the process of injecting the multimodal long-tail scene data into the pre-built simulation environment based on the digital twin training engine to obtain the first long-tail scene state data includes: The visual features of the multimodal long-tail scene data are compared with a pre-built precision map in the simulation environment to determine the target position of the long-tail scene in the pre-built simulation environment; the multimodal long-tail scene data is mapped to the target position through the distortion model, camera intrinsic matrix and camera extrinsic matrix in the sensor model to obtain aligned multimodal long-tail scene data. The aligned multimodal long-tail scene data is used to perform state calculations to obtain the environmental parameters of the long-tail scene and the motion state of the moving targets in the long-tail scene. The motion state of the moving target is corrected by the vehicle dynamics model to obtain the corrected motion target state; and the corrected motion target state and the environmental parameters are input into the traffic flow model to obtain the interactive response strategy of the moving target. The corrected motion target state, the environmental parameters, and the interaction response strategy are combined to obtain the first long-tail scene state data.

8. The method for training an intelligent driving model for unmanned logistics vehicles based on a world model according to any one of claims 1-3, characterized in that, The process of training and evaluating the first generative world model using the first long-tail scene state data to obtain the teacher model includes: The first generative world model is trained using preset expert data to obtain a second generative world model; the preset expert data includes driving rule data and driving experience data. The second generative world model is obtained by training the second generative world model using the first long-tail scene state data. The second generative world model is subjected to a visual fidelity evaluation to obtain a visual fidelity score; the second generative world model is subjected to a physical dynamics compliance evaluation to obtain a physical dynamics compliance score; and the second generative world model is subjected to a long-tail scene scarcity evaluation to obtain a long-tail scene scarcity score. The visual fidelity score, the physical dynamics compliance score, and the long-tail scene scarcity score are weighted and summed to obtain the comprehensive score of the second generative world model. Based on the comprehensive score, the second generative world model is selected to obtain the teacher model.

9. The method for training an intelligent driving model for unmanned logistics vehicles based on a world model according to claim 8, characterized in that, The formula for calculating the comprehensive score includes: ， in, This indicates the overall score. Indicates the weight of the visual fidelity score. Indicates visual fidelity score. Indicates the weight of the physical dynamics compliance score. Indicates the physical dynamics compliance score. This indicates the weighting of the scarcity score for long-tail scenarios. This indicates the scarcity score of long-tail scenarios; The formula for calculating the visual fidelity score includes: ， in, Indicates visual fidelity score. Indicates the weight of the distributional similarity score. The distributional similarity score is represented by the Fraser initial distance between the generated image and the real image. This indicates the weighting of perceived quality and aesthetic ratings. The perceived quality and aesthetic score are represented by the cosine similarity between the generated image and the real image. Indicates the perceived difference distance weight. The perceived difference distance is represented by the similarity of learned perceptual image patches between the generated image and the real image. Represents the normalization function; The formula for calculating the physical dynamics compliance score includes: ， in, Indicates the physical dynamics compliance score. Represents the physical error weights of the trajectory. The physical error of the trajectory is represented by the dynamic time-warped distance between the generated trajectory and the actual trajectory. This indicates the weighting of the collision and contact rationality scores. This represents the normalized collision and contact plausibility score. Indicates the weight of optical flow smoothness error. This represents the optical flow smoothness error. The optical flow smoothness error of the generated trajectory is used to measure the continuity of the motion field. Indicates the weight of the physical-logical consistency score. This represents the normalized physical-logical consistency score, used to characterize physical violations in the generated video; The formula for calculating the scarcity score of the long-tail scenario includes: ， in, This indicates the scarcity score of long-tail scenarios. Represents the marginal probability weights. Marginal probability represents the probability of a single feature appearing in the overall dataset. Indicates the scarcity of a single factor. Represents the conditional probability weights. Conditional probability represents the probability of a single element occurring given a pre-defined context. Used to characterize context scarcity This represents the Sigmoid function.

10. A training system for an intelligent driving model of an unmanned logistics vehicle based on a world model, characterized in that, include: The data acquisition module is used to acquire historical video data collected by the unmanned logistics vehicle during its operation and to generate long-tail scene data input by the target personnel. The target personnel include: developers and operations personnel; The first model training module is used to merge the historical video data with a preset number of internet video data as sample data; and to train a generative world model with a preset architecture using the sample data to obtain a first generative world model. The data generation module is used to perform semantic parsing on the long-tail scene data generation instructions based on the long-tail scene generator to obtain structured generation instructions, and control the first generative world model to generate multimodal long-tail scene data with the structured generation instructions as input conditions. The second model training module is used to inject the multimodal long-tail scene data into a pre-built simulation environment based on the digital twin training engine to obtain the first long-tail scene state data; and to train and evaluate the first generative world model using the first long-tail scene state data to obtain the teacher model. The model deployment module is used to perform knowledge distillation on the teacher model to obtain a first student model, and deploy the first student model on the vehicle to make decisions and control the vehicle during its operation.

Images