Intelligent driving method, device and equipment based on world model and storage medium

Abstract

The embodiment of the invention provides an intelligent driving method and device based on a world model, equipment and a storage medium, and belongs to the technical field of intelligent driving. The method comprises the following steps: performing feature coding processing on each 3D occupancy map of a target vehicle to obtain coding features, and performing axial pooling processing on the coding features along three coordinate axes to obtain plane features corresponding to three orthogonal planes; performing resampling processing on a plurality of plane features under the same orthogonal plane, predicting plane change information of the current orthogonal plane at a future moment, and integrating the plane change information of the three orthogonal planes to obtain predicted change information; and generating a query vector based on the plane feature corresponding to the current moment and the predicted change information, performing feature coding and decoding processing on the executed action sequence based on the query vector, and predicting and outputting a target execution action of the target vehicle at the future moment. According to the invention, the precision of the output result can be improved while the lightweight calculation of the world model is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to an intelligent driving method, device, equipment and storage medium based on a world model. Background Technology

[0002] World models are a key technology for advanced decision-making and planning, with their core task being to predict the future state of the environment based on historical and currently observed images. Among related technologies, the first approach involves predicting the trajectories of dynamic instances in a scene from a bird's-eye view perspective, or using diffusion models to generate pixel-level views of the future, to achieve high-precision motion planning. The second approach involves compressing real-time traffic images into ultra-low-dimensional features to achieve rapid motion planning.

[0003] However, the first approach involves a massive number of network parameters and incurs high computational costs, making it difficult to meet the real-time requirements of in-vehicle systems. The second approach, in pursuit of lightweight design, inevitably loses a significant amount of crucial three-dimensional spatial structural information. In other words, related technologies suffer from a trade-off between lightweight computation and high-precision output when using world models to achieve intelligent driving. Summary of the Invention

[0004] The main objective of this application is to propose an intelligent driving method, device, equipment, and storage medium based on a world model, aiming to improve the accuracy of the output results while enhancing the lightweight computation of the world model, so as to better realize intelligent driving assistance based on a high-quality world model.

[0005] To achieve the above objectives, a first aspect of this application proposes an intelligent driving method based on a world model, comprising:

[0006] Acquire multiple panoramic images of the target vehicle and the sequence of actions executed by the target vehicle, and generate a 3D occupancy map corresponding to each panoramic image, wherein the multiple panoramic images are from the current time and at least one historical time.

[0007] Multiple 3D occupancy maps are input into the trained world model. Feature encoding is performed on each 3D occupancy map to obtain encoded features. Axial pooling is then performed on the encoded features along the three coordinate axes to obtain planar features corresponding to the three orthogonal planes. The three coordinate axes are perpendicular to each other, and each orthogonal plane is defined by two different coordinate axes. A multi-scale Transformer is set in the world model.

[0008] By using a multi-scale Transformer, the planar change information of the current orthogonal plane in the future is predicted by resampling multiple planar features under the same orthogonal plane, and the planar change information of the three orthogonal planes is integrated to obtain the predicted change information.

[0009] A query vector is generated based on the planar features and predicted change information corresponding to the current moment. Based on the query vector, the sequence of executed actions is processed by feature encoding and decoding to predict the target vehicle's target action at a future moment.

[0010] In some embodiments, a multi-scale Transformer is used to predict the planar changes of the current orthogonal plane at future times by resampling multiple planar features under the same orthogonal plane, including:

[0011] Using a multi-scale Transformer, multiple planar features belonging to the same orthogonal plane are downsampled at different scales to obtain feature maps corresponding to each scale. The spatial resolution of the feature maps decreases from high to low.

[0012] At different scales, with a preset tracking category of objects as the target, feature extraction is performed on the feature map at the current scale to obtain object change information indicating the degree of change of the tracking category of objects at the current scale. The tracking category of objects is different at different scales.

[0013] Based on the object's change information, predict the planar change information of the current orthogonal plane at future times.

[0014] In some embodiments, at different scales, with a preset tracking category of objects as the target, feature extraction is performed on the feature map at the current scale to obtain object change information indicating the degree of change of the tracking category of objects at the current scale, including:

[0015] If the spatial resolution of the feature map at the current scale is the highest, feature extraction processing is performed on the feature map corresponding to the current scale to obtain object change information that indicates the degree of change of the tracked object category at the current scale.

[0016] If the spatial resolution of the feature map at the current scale is not the highest, the feature map obtained at the next scale adjacent to the current scale is determined as the feature map to be used, and the object change information obtained at the next scale adjacent to the current scale is determined as the object change information to be used.

[0017] The feature map to be used and the object change information to be used are superimposed to obtain the fusion information to be used. Based on the fusion information to be used, feature extraction processing is performed on the feature map corresponding to the current scale to obtain the object change information indicating the degree of change of the object of the tracking category at the current scale.

[0018] In some embodiments, a query vector is generated based on the planar features and predicted change information corresponding to the current time, including:

[0019] Obtain the initial query vector;

[0020] By integrating the planar features corresponding to the three orthogonal planes at the current moment, the current three-plane features are obtained. By superimposing the current three-plane features and the predicted change information, the future 3D occupancy map at the future moment is predicted.

[0021] The current three-plane features, predicted change information, and future 3D occupancy map are mapped to the shared latent space to obtain the latent features before change, the latent features after change, and the latent features after change.

[0022] Based on the preset first weight, the latent features before and after the change are multiplied element by element to obtain the first key feature. Based on the preset second weight, the latent features before and after the change are multiplied element by element to obtain the second key feature.

[0023] The first key feature, the variable potential feature, and the second key feature are averaged element by element to obtain the average potential feature. The initial query vector and the average potential feature are then superimposed to obtain the query vector.

[0024] In some embodiments, the sequence of executed actions includes multiple executed actions;

[0025] Based on the query vector, feature encoding and decoding of the executed action sequence is performed to predict the target vehicle's future execution actions, including:

[0026] Multiple executed actions are projected onto the same feature dimension as the query vector to obtain multiple first action projection features;

[0027] Multiple first action projection features are processed by feature encoding to obtain multiple first action encoded features. Based on the query vector, multiple first action encoded features are processed by feature decoding to predict the target vehicle's target action at future time.

[0028] In some embodiments, the method of performing feature encoding and decoding processing on the sequence of executed actions based on the query vector to predict the target vehicle's target execution action at a future time also includes:

[0029] Obtain the target driving path of the target vehicle, and adjust the query vector based on the target driving path to obtain the adjusted updated query vector;

[0030] Multiple executed actions are projected onto the same feature dimension as the updated query vector to obtain multiple second action projection features;

[0031] Multiple second action projection features are processed by feature encoding to obtain multiple second action encoded features. Based on the updated query vector, multiple action encoded features are processed by feature decoding to predict the target vehicle's target action at future time.

[0032] In some embodiments, the world model is trained through the following steps, including:

[0033] Acquire multiple panoramic images of the sample vehicle and the sequence of actions performed by the sample vehicle, and generate a sample 3D occupancy map corresponding to each panoramic image. The multiple panoramic images are from the current time and at least one historical time.

[0034] Multiple sample 3D occupancy maps are input into the initial world model. Feature encoding is performed on each sample 3D occupancy map to obtain sample encoded features. Axial pooling is then performed on the sample encoded features along the three coordinate axes to obtain sample plane features corresponding to the three sample orthogonal planes. The semantic loss value is determined based on the sample plane features. The three coordinate axes are mutually perpendicular, and each sample orthogonal plane is defined by two different coordinate axes. An initial multi-scale Transformer is set in the initial world model.

[0035] Using the initial multi-scale Transformer, the sample plane features of multiple sample planes under the same sample orthogonal plane are resampled to predict the sample plane change information of the current sample orthogonal plane at future time. The sample plane change information of the three sample orthogonal planes is integrated to obtain the sample prediction change information. The prediction loss value is determined based on the sample prediction change information.

[0036] A sample query vector is generated based on the sample plane features and sample prediction change information at the current moment. The sample query vector is then used to perform feature encoding and decoding on the sequence of actions already performed by the sample. The sample vehicle's actions at future moments are predicted and output. The action loss value is determined based on the sample actions.

[0037] Based on semantic loss, prediction loss, and action loss, the model parameters of the initial world model are adjusted until the preset training conditions are met, resulting in the trained world model.

[0038] In some embodiments, determining the prediction loss value based on sample prediction change information includes:

[0039] Obtain the verification prediction change information of the sample orthogonal plane, and the verification 3D occupancy map corresponding to the verification prediction change information;

[0040] Based on the sample prediction change information, determine the sample 3D occupancy map corresponding to the current moment. Based on the preset first trade-off coefficient, calculate the degree of difference between the sample prediction change information and the verification prediction change information to obtain the prediction change information loss value.

[0041] Based on the preset second tradeoff coefficient, the degree of difference between the sample 3D occupancy map and the validation 3D occupancy map is calculated to obtain the 3D occupancy map loss value.

[0042] The predicted loss value is obtained by superimposing the predicted change information loss value and the 3D occupancy map loss value.

[0043] To achieve the above objectives, a second aspect of this application proposes an intelligent driving device based on a world model, comprising:

[0044] The acquisition module is used to acquire multiple panoramic images of the target vehicle and the sequence of actions executed by the target vehicle, and generate a 3D occupancy map corresponding to each panoramic image. The multiple panoramic images are from the current time and at least one historical time.

[0045] The triorthogonal plane processing module is used to input multiple 3D occupancy maps into the trained world model. For each 3D occupancy map, feature encoding is performed to obtain encoded features. The encoded features are then axially pooled along the three coordinate axes to obtain the planar features corresponding to the three orthogonal planes. The three coordinate axes are perpendicular to each other, and each orthogonal plane is defined by two different coordinate axes. The world model is equipped with a multi-scale Transformer.

[0046] The resampling processing module is used to utilize a multi-scale Transformer to resample multiple planar features under the same orthogonal plane, predict the planar change information of the current orthogonal plane at future time, and integrate the planar change information of the three orthogonal planes to obtain the predicted change information.

[0047] The target prediction module is used to generate a query vector based on the planar features and predicted change information at the current moment, and to perform feature encoding and decoding processing on the executed action sequence based on the query vector to predict and output the target vehicle's target execution action at a future moment.

[0048] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the world-model-based intelligent driving method of the first aspect described above.

[0049] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the world-model-based intelligent driving method of the first aspect described above.

[0050] This application proposes a world model-based intelligent driving method, device, electronic device, and storage medium. It acquires multiple panoramic images of a target vehicle and a sequence of executed actions of the target vehicle, generating 3D occupancy maps corresponding to each panoramic image. The multiple panoramic images are from the current time and at least one historical time. The multiple 3D occupancy maps are input into a trained world model. Feature encoding is performed on each 3D occupancy map to obtain encoded features. Axial pooling is then performed along three coordinate axes to obtain planar features corresponding to three orthogonal planes. The three coordinate axes are mutually perpendicular, and each orthogonal plane is defined by two different coordinate axes. A multi-scale Transformer is incorporated into the world model. Using the multi-scale Transformer, multiple planar features under the same orthogonal plane are resampled to predict the planar change information of the current orthogonal plane at future time. The planar change information of the three orthogonal planes is integrated to obtain predicted change information. A query vector is generated based on the planar features corresponding to the current time and the predicted change information. Based on the query vector, feature encoding and decoding processing is performed on the sequence of executed actions to predict and output the target vehicle's target execution action at a future time. This application aims to improve the accuracy of the output results while enhancing the lightweight computation of the world model, so as to better realize intelligent driving assistance based on a high-quality world model. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of an optional implementation environment for the intelligent driving device based on a world model provided in this application embodiment;

[0052] Figure 2 This is an optional flowchart of the intelligent driving method based on a world model provided in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of an optional data processing flow for an intelligent driving method based on a world model provided in an embodiment of this application;

[0054] Figure 4 yes Figure 2 Step 103 is an optional implementation flowchart;

[0055] Figure 5 This is a schematic diagram of another optional data processing flow for the intelligent driving method based on a world model provided in the embodiments of this application;

[0056] Figure 6 yes Figure 4 Step 103.1.2 is an optional implementation flowchart;

[0057] Figure 7 yes Figure 2 Step 104 in the flowchart is an optional implementation.

[0058] Figure 8 This is a schematic diagram of another optional data processing flow for the intelligent driving method based on the world model provided in the embodiments of this application;

[0059] Figure 9 yes Figure 2 Another optional implementation flowchart for step 104 in the diagram;

[0060] Figure 10 yes Figure 2 Step 104 in the flowchart is another optional implementation.

[0061] Figure 11 This is an optional world model training flowchart of the intelligent driving method based on the world model provided in the embodiments of this application;

[0062] Figure 12 yes Figure 11 Step 203 is an optional implementation flowchart;

[0063] Figure 13 This is a schematic diagram of an optional device module of the intelligent driving device based on a world model provided in this application embodiment;

[0064] Figure 14 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] Intelligent driving refers to a modern driving method that utilizes advanced sensors, controllers, software algorithms, and communication technologies to enable vehicles to partially or fully autonomously perceive their surroundings, make decisions, and execute driving tasks. It aims to enhance road safety, comfort, and convenience by reducing human error, optimizing driving routes, and improving traffic efficiency. Intelligent driving encompasses multiple technological levels, from assisted driving (such as automatic emergency braking and lane keeping assist) to fully autonomous driving (safe driving without human intervention), and is applicable to various scenarios including private cars, public transportation vehicles, and logistics vehicles.

[0070] World models are a key technology for advanced decision-making and planning, with their core task being to predict the future state of the environment based on historical and currently observed images. Among related technologies, the first approach involves predicting the trajectories of dynamic instances in a scene from a bird's-eye view perspective, or using diffusion models to generate pixel-level views of the future, to achieve high-precision motion planning. The second approach involves compressing real-time traffic images into ultra-low-dimensional features to achieve rapid motion planning.

[0071] However, the first approach involves a massive number of network parameters and incurs high computational costs, making it difficult to meet the real-time requirements of in-vehicle systems. The second approach, in pursuit of lightweight design, inevitably loses a significant amount of crucial three-dimensional spatial structural information. In other words, related technologies suffer from a trade-off between lightweight computation and high-precision output when using world models to achieve intelligent driving.

[0072] Based on this, embodiments of this application provide an intelligent driving method, device, equipment, and storage medium based on a world model. This application introduces a triplane representation technique to efficiently compress a high-dimensional 3D occupancy map into three orthogonal two-dimensional plane features. Compared to the traditional method of compressing into a one-dimensional vector, this unique encoding mechanism significantly reduces data dimensionality while preserving the key three-dimensional spatial structure information of the scene to the maximum extent, laying a solid foundation for high-precision prediction. Furthermore, embodiments of this application utilize a multi-scale Transformer to predict the changes in future scenes rather than their complete states. This not only greatly reduces the learning burden of the model and the number of network parameters but also accurately captures the dynamic patterns of objects at different scales. Finally, embodiments of this application tightly couple the predicted scene change information with historical actions for motion planning. This significantly improves the accuracy of future scene prediction and trajectory planning while ensuring the model's lightweight nature and high inference speed, thereby achieving an effective balance between real-time performance and safety in intelligent driving systems making decisions in complex dynamic environments.

[0073] For example, such as Figure 1 As shown, Figure 1This is a schematic diagram of an optional implementation environment for the intelligent driving device based on a world model provided in this application embodiment. The implementation environment includes a camera 11 and a central processing unit 12 deployed on a target vehicle, wherein the intelligent driving device based on a world model (which may also be referred to as "device" for ease of description) is deployed on the central processing unit 12. The central processing unit 12 receives multiple panoramic images of the target vehicle captured by the camera 11, as well as a sequence of actions already performed by the target vehicle. The panoramic images are from the current moment and at least one historical moment. Next, the central processing unit 12 generates 3D occupancy maps corresponding to each panoramic image. These 3D occupancy maps are input into a trained world model. Feature encoding is performed on each 3D occupancy map to obtain encoded features. Axial pooling is then performed along the three coordinate axes to obtain planar features corresponding to three orthogonal planes. The three coordinate axes are mutually perpendicular, and each orthogonal plane is defined by two different coordinate axes. A multi-scale Transformer is configured in the world model. Then, using the multi-scale Transformer, multiple planar features under the same orthogonal plane are resampled to predict the planar change information of the current orthogonal plane at future moments. The planar change information of the three orthogonal planes is integrated to obtain predicted change information. Finally, a query vector is generated based on the planar features corresponding to the current moment and the predicted change information. Based on the query vector, feature encoding and decoding processing is performed on the sequence of actions already performed to predict and output the target vehicle's target actions at future moments. Thus, this application can improve the accuracy of the output results while enhancing the lightweight computation of the world model, so as to better realize intelligent driving assistance based on a high-quality world model.

[0074] It should be noted that in this application embodiment, when information related to user characteristics, such as basic user information or user identity, is required, the user's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. Additionally, when this application embodiment needs to obtain sensitive personal information of a user, the user's separate permission or consent will be obtained first. Only after obtaining the user's separate permission or consent will the necessary data for the normal operation of this application embodiment be obtained. For example, before obtaining a panoramic image of a target vehicle, the authorization or consent of the target vehicle owner will be obtained; otherwise, a panoramic image that cannot be used in this application embodiment will be obtained. Furthermore, all other relevant data obtained by the training device of this application are authorized data, and will not be elaborated upon here.

[0075] In this application embodiment, the description will be from the perspective of the device, such as... Figure 2 As shown, Figure 2This is an optional flowchart of the intelligent driving method based on a world model provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, the following steps 101 to 104. When the device executes the intelligent driving method based on the world model, the specific process is as follows. It should be noted first that this embodiment... Figure 2 The order of steps 101 to 104 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0076] Step 101: Acquire multiple panoramic images of the target vehicle and the sequence of actions executed by the target vehicle, and generate a 3D occupancy map corresponding to each panoramic image, wherein the multiple panoramic images are from the current time and at least one historical time.

[0077] Step 101 will be described in detail below.

[0078] In some embodiments, the device first acquires panoramic images of the target vehicle at the current moment and at least one historical moment. Each panoramic image can be captured and stitched together by multiple onboard cameras, providing 360-degree visual information about the target vehicle's surroundings. Additionally, the device simultaneously acquires the sequence of actions performed by the target vehicle at both historical and current moments, such as historical records of steering, acceleration, and braking. Subsequently, the device processes each acquired panoramic image to generate its corresponding 3D occupancy map. A 3D occupancy map is a data structure that divides three-dimensional space into fine grids (i.e., voxels) to represent the occupancy status of each spatial point (occupancy unit) in the environment surrounding the target vehicle. For example, an occupancy unit may be occupied (e.g., with obstacles present) or vacant. In reality, this corresponds to pedestrians walking on the road, vehicles driving, and large trees along the roadside. Thus, unstructured image information is transformed into a structured three-dimensional environment representation that can be directly processed by the world model.

[0079] In this application, the target vehicle refers to a specific car used for research, monitoring or analysis. All panoramic images are taken from the first-person perspective of the target vehicle. The first-person perspective refers to a form of observing and presenting the surrounding things from the perspective of the target vehicle.

[0080] In this system, one of the multiple panoramic images was captured at the current moment, while the remaining panoramic images were captured at historical moments, prior to the current moment. The apparatus employing the method proposed in this application aims to predict changes in the 3D occupancy map at future moments compared to the current moment, based on panoramic images captured at historical and current moments. This allows the apparatus to better understand the interaction between the target vehicle and its surrounding environment, thereby inferring the next action of each occupancy unit in the 3D occupancy map, and ultimately planning the target vehicle's target execution actions at future moments. This achieves the goal of assisting the target vehicle in safe driving or enabling the target vehicle to drive autonomously and efficiently.

[0081] Step 102: Input multiple 3D occupancy maps into the trained world model, perform feature encoding processing on each 3D occupancy map to obtain encoded features, and perform axial pooling processing on the encoded features along the three coordinate axes to obtain the planar features corresponding to the three orthogonal planes. The three coordinate axes are perpendicular to each other, and each orthogonal plane is defined by two different coordinate axes. The world model is equipped with a multi-scale Transformer.

[0082] Step 102 is described in detail below.

[0083] In some embodiments, such as Figure 3 As shown, Figure 3 This is a schematic diagram of an optional data processing flow for the intelligent driving method based on a world model provided in this application embodiment, showing the 3D occupancy map for each input. The encoder E within the world model first performs feature encoding, mapping it from the original voxel space to a compact and information-rich high-dimensional feature space, thus obtaining the encoded features. Furthermore, in order to achieve efficient compression without losing key spatial information, the device will then perform axial pooling on the encoded features along three mutually perpendicular coordinate axes (e.g., X, Y, and Z axes). This process essentially projects the high-dimensional encoded features onto three orthogonal two-dimensional planes and aggregates the information, thereby obtaining planar features corresponding to three orthogonal planes defined by two different coordinate axes. The combination of the three planar features (i.e., the three-plane representation) can compactly represent the complete three-dimensional scene structure with a lower data dimension.

[0084] Furthermore, in three-dimensional space, the three coordinate axes are the X-axis (horizontal), Y-axis (vertical), and Z-axis (height), which are perpendicular to each other, forming a standard three-dimensional Cartesian coordinate system. Each orthogonal plane is a two-dimensional plane formed by two of these coordinate axes. Specifically, the three orthogonal planes are:

[0085] ①XY plane: Composed of the X-axis and Y-axis, it represents the horizontal plane and can be understood as a top view.

[0086] ②YZ Plane: Composed of the Y-axis and Z-axis, it represents the side elevation and can be understood as the height and depth information when viewed from the side;

[0087] ③XZ plane: Composed of the X-axis and Z-axis, it represents another vertical plane, which can be understood as the perspective of observation from the front or back.

[0088] Furthermore, axial pooling is a downsampling operation performed on multidimensional data along specific coordinate axes to reduce data volume and extract key features. In this embodiment, axial pooling is performed along the X, Y, and Z axes for each encoded feature, meaning that pooling operations are performed independently on each orthogonal plane (XY, YZ, XZ) to compress information and focus attention on the most important features. This not only reduces computational complexity but also enhances the model's ability to capture key structural features, contributing to more efficient and accurate analysis of three-dimensional spatial data.

[0089] Furthermore, Figure 3 The 3D point cloud structure represents a 3D occupancy map at a certain moment. It meticulously depicts the driving environment around the target vehicle in voxel form, where different colors represent different semantic categories, such as purple for drivable roads, blue for vehicles, and green for vegetation. Of course, the semantic categories corresponding to different colors can be set according to actual conditions, and this application embodiment does not impose any limitations on this.

[0090] Furthermore, the world model also includes a multi-scale Transformer and a decoder. Since the planar features corresponding to the three orthogonal planes obtained after axial pooling need to undergo resampling and feature encoding / decoding before the output features can be decoded to obtain the target action, we will not introduce the specific operations of the decoder D here. Instead, we will first introduce how to process the planar features and output predicted change information according to the data processing order:

[0091] Step 103: Using a multi-scale Transformer, the planar change information of the current orthogonal plane is predicted in the future by resampling multiple planar features under the same orthogonal plane, and the planar change information of the three orthogonal planes is integrated to obtain the predicted change information.

[0092] Step 103 will be described in detail below.

[0093] In some embodiments, the world model resamples multiple planar features (from the current moment and at least one historical moment) under the same orthogonal plane to analyze the temporal evolution of features at multiple different feature processing scales. Furthermore, the multi-scale mechanism employed in this embodiment enables the world model to simultaneously focus on macroscopic scene changes (such as road structures) and microscopic object movements (such as vehicles and pedestrians), and accurately predict the planar changes of the current orthogonal plane in future moments. It is important to emphasize that the planar change information is not a complete feature set for future moments, but rather represents the increment or difference in features on that plane from the current moment to the future moment. Finally, by integrating the planar change information from three orthogonal planes, a predictive change information that comprehensively reflects the future dynamics of the overall 3D scene is formed.

[0094] Furthermore, this embodiment of the application resamples and temporally models planar features at different times under the same orthogonal plane using a multi-scale Transformer. This enables the simultaneous capture of macroscopic scene changes and microscopic object motions (e.g., at multiple feature scales), thereby accurately predicting future incremental changes corresponding to the plane. Compared to directly predicting complete future features, focusing only on the "changing part" improves the model's efficiency and relevance. Next, this embodiment of the application fuses the change information from three orthogonal planes to form a comprehensive prediction of the future dynamics of the entire three-dimensional space. This significantly enhances the world model's understanding and prediction capabilities of the spatiotemporal evolution of complex traffic scenarios, providing more reliable and fine-grained environmental perception support for applications such as intelligent driving.

[0095] In some embodiments, such as Figure 4 As shown, Figure 4 yes Figure 2 Step 103, an optional implementation flowchart, utilizes a multi-scale Transformer to predict the planar changes of the current orthogonal plane in the future by resampling multiple planar features under the same orthogonal plane, including the following steps:

[0096] 103.1.1 Using a multi-scale Transformer, multiple planar features belonging to the same orthogonal plane are downsampled at different scales to obtain feature maps corresponding to each scale. The spatial resolution of the feature maps decreases from high to low.

[0097] 103.1.2 At different scales, with a preset tracking category object as the target, feature extraction is performed on the feature map at the current scale to obtain object change information indicating the degree of change of the tracking category object at the current scale. The tracking category objects are different at different scales.

[0098] 103.1.3 Based on object change information, predict the planar change information of the current orthogonal plane at future times.

[0099] Steps 103.1.1 to 103.1.3 are described in detail below.

[0100] In some embodiments, such as Figure 5 As shown, Figure 5 This is a schematic diagram of another optional data processing flow for the intelligent driving method based on a world model provided in this application embodiment. Figure 5 On the left, the "Time" axis represents the temporal flow of data processing. The world model receives 3D occupancy maps from the current time and multiple historical time points as input. Each 3D occupancy map is processed by a shared encoder E (described in step 102). This process compresses the high-dimensional 3D occupancy map into three orthogonal planar features. Then, the three-plane representation of the current time (i.e., the last input time) is labeled as "Last triplane" so that the planar features corresponding to the current time, together with the planar change information determined by the multi-scale Transformer, can be used to predict the target action corresponding to the target vehicle.

[0101] Furthermore, Figure 5 The box at the top center illustrates the specific processing flow of the multi-scale Transformer module used to predict future scene changes. The data processing logic of the multi-scale Transformer is applied to all planes, meaning the same operation is performed on the xy, xz, and yz planes respectively. Historical and current three-plane features are input into this module and processed at multiple scales: the feature maps are downsampled step by step to form layers with different spatial resolutions.

[0102] In this embodiment of the application, the prediction model is deployed in the world model. Predict planar change information in different orthogonal planes using the prediction model. It consists of Transformers at multiple scales that share the same architecture but have different learnable parameters and different input sizes, depending on the individual orthogonal planes and scales. These are used to resample planar features from different orthogonal planes.

[0103] Furthermore, the multi-scale Transformer fuses information through U-Net-like resampling processes (including upsampling and downsampling) and skip connections to determine the planar changes of each orthogonal plane from historical moments to the current moment. Specifically, the multi-scale Transformer first targets multiple planar features belonging to the same orthogonal plane (these features come from the current moment and different historical moments); then, it performs downsampling processing on them at different scales. The downsampling process aims to gradually reduce the spatial dimension of the feature maps. After downsampling, the multi-scale Transformer obtains a set of feature maps containing multiple scales, with the spatial resolution of these feature maps decreasing from high to low, forming a hierarchical structure similar to an image pyramid.

[0104] Specifically, as shown in the diagram above, with Taking an example, we will explain the multiple planar features that belong to the same orthogonal plane. Through a UNet-like downsampling process, feature maps at V scales are obtained. The feature map at each scale v∈V is represented as follows: These feature maps will be flattened in the last two dimensions to form... Each token; then, the multi-scale Transformer generates spatiotemporal memory and uses a learnable query. (its dimensions and) (Similarly), through the cross-attention mechanism of multi-scale Transformers, token changes from the current time step to the next time step are generated based on memory; then, token changes at all scales are aggregated through U-Net-style upsampling to obtain the plane change information in the next time step. Finally, planar transformation information can also be obtained by convolution operation φ (1×1 convolution kernel) with the previous planar feature. By combining these data, the planar change information for the next moment can be predicted. This data processing process can be represented by the following formulas (1) to (3):

[0105]

[0106] Where i∈{xy, xz, yz}, i represents each plane, k∈{1, …, τ} f} represents the time steps for autoregressive prediction. It is a learnable query for the i-th plane at time step t+k. This represents the planar characteristics of the corresponding orthogonal plane at future times, determined based on planar change information; This represents the characteristics of all orthogonal planes at future moments obtained through integration.

[0107] Understandably, high-resolution feature maps retain rich detail, making them suitable for analyzing the dynamics of small objects; while low-resolution feature maps aggregate a wider receptive field, making them more suitable for capturing the overall changing trends of large objects or backgrounds. Based on this, the multi-scale Transformer extracts features from the feature map at the current scale, targeting pre-defined tracking categories. Tracking categories refer to the types or sizes of objects that are primarily of interest at each scale, based on the different resolutions of the feature maps obtained at different scales. For example, at scales with higher spatial resolution, tracking categories might be set to small, agile targets such as pedestrians and bicycles; while at scales with lower spatial resolution, they might be set to large, slow-moving targets such as trucks and buildings. In this way, the multi-scale Transformer can extract object change information from the corresponding feature maps that specifically indicates the degree of change in these tracking categories. Because different scales correspond to different tracking categories, the world model can decouple the kinematics of different objects, enabling more refined analysis of complex dynamic scenes.

[0108] It should be noted that the tracking categories of objects at different scales can be set according to actual conditions, and this application embodiment does not impose any limitations on this. For example, under high-resolution feature maps, the device focuses on small targets such as pedestrians and bicycles, and identifies a pedestrian accelerating from the roadside to the zebra crossing by using the changes in planar features at the current time and different historical times; under low-resolution feature maps, the device focuses on large static or slow-moving targets such as large trucks and buildings, and similarly identifies a large truck slowly entering the intersection by using the changes in planar features at the current time and different historical times.

[0109] Next, we will introduce step 103.1.2 (corresponding to...) Figure 5 Specifically, how does the feature extraction process (in the box section) obtain information about object changes?

[0110] In some embodiments, such as Figure 6 As shown, Figure 6 yes Figure 4 Step 103.1.2 of the flowchart is an optional implementation. At different scales, targeting objects of a preset tracking category, feature extraction is performed on the feature map at the current scale to obtain object change information indicating the degree of change of the tracking category object at the current scale. This includes the following steps:

[0111] A.1 If the spatial resolution of the feature map at the current scale is the highest, perform feature extraction processing on the feature map corresponding to the current scale to obtain object change information that indicates the degree of change of the tracked object at the current scale;

[0112] A.2 If the spatial resolution of the feature map at the current scale is not the highest, determine the feature map obtained at the next scale adjacent to the current scale as the feature map to be used, and determine the object change information obtained at the next scale adjacent to the current scale as the object change information to be used.

[0113] A.3 The feature map to be used and the object change information to be used are superimposed to obtain the fusion information to be used. Based on the fusion information to be used, the feature map corresponding to the current scale is processed for feature extraction to obtain the object change information indicating the degree of change of the object of the tracking category at the current scale.

[0114] Steps A.1 to A.3 are described in detail below.

[0115] In some embodiments, such as Figure 5 As shown, if the spatial resolution of the feature map at the current scale is the highest (corresponding to...) Figure 5 The processing path at the bottom of the mid-level structure will directly perform feature extraction processing on the feature map at this scale to obtain object change information indicating the degree of change of the tracked object category at the current scale (from...). Figure 5 (The box with the flame icon performs data processing). At this scale, the tracked category objects are typically large objects or background elements that match this low-resolution scale.

[0116] Furthermore, if the spatial resolution of the feature map at the current scale is not the highest, that is, when the upsampling process enters a lower resolution scale level, for example... Figure 5 In the middle or top layer, the device will determine the feature map obtained at the next adjacent scale (i.e., the layer with higher resolution) as the feature map to be used. Figure 5 In the diagram, the blue dashed arrows extending from the left-side feature map represent jump connections. Simultaneously, the device also determines the object change information obtained at the next adjacent scale as the object change information to be used. Then, the (⊕) object change information to be used is superimposed to obtain the fusion information to be used, which is represented in the diagram as an upward green arrow extending from the green prediction result at the next lower scale (lower layer).

[0117] Furthermore, based on this rich fusion information, the multi-scale Transformer performs feature extraction on the feature map corresponding to the current scale. At the current scale, the multi-scale Transformer can fully utilize the macroscopic dynamics predicted at lower levels as prior knowledge, thus focusing more on finer dynamic analysis at this scale, ultimately obtaining object change information at the current scale that indicates the degree of change of the corresponding smaller-sized tracking category object.

[0118] Understandably, this application's embodiments cleverly decompose the complex task of full-scene dynamic prediction into a hierarchical processing flow. Specifically, by first grasping the motion trends of the entire scene and large objects at a low-resolution scale, and then progressively moving upwards, the prediction results of lower levels are used to guide and constrain higher-level predictions, ultimately predicting and outputting object change information at various scales. This design, through effective decoupling of motion patterns of objects of different scales, allows each level to focus on the dynamic details of its corresponding scale, avoiding the huge computational resources generated by processing at a single scale, thereby greatly improving the accuracy and robustness of the overall world model prediction.

[0119] Step 104: Generate a query vector based on the planar features and predicted change information corresponding to the current moment, and perform feature encoding and decoding processing on the executed action sequence based on the query vector to predict and output the target vehicle's target execution action at a future moment.

[0120] Step 104 will be described in detail below.

[0121] In some embodiments, the device generates a query vector based on the planar features corresponding to the current moment (i.e., the current scene state) and the predicted change information obtained in the previous step (i.e., the expected future scene dynamics). The query vector can be understood as a dynamically generated "planning hint" or "guidance information" based on the current scene state and expected future changes. Subsequently, the device uses this information-rich query vector to perform feature encoding and decoding processing on the target vehicle's executed action sequence. Specifically, the encoder first processes the historical action sequence to understand the target vehicle's historical operations, and then the decoder uses the query vector as key guidance to predict and output the target action that the target vehicle should execute in the future, thus completing a complete perception-prediction-planning closed loop.

[0122] It is understood that, while significantly reducing data dimensionality, the embodiments of this application retain key three-dimensional spatial structure information to the maximum extent, effectively avoiding the information loss problem caused by excessive compression in traditional solutions. Secondly, the embodiments of this application use multi-scale Transformers to predict future planar change information, rather than complete future states, which greatly reduces the learning burden and computational complexity of the model. Combining these two aspects, the world model of the embodiments of this application can produce high-precision future scene prediction results while maintaining lightweight and high computational efficiency. Finally, the query vector generated by the world model based on this high-precision prediction provides accurate guidance for action planning, thereby ensuring that the output target execution action is both safe and reasonable, perfectly solving the problem of the inability to balance lightweight computation and high-precision result output of the world model in related technologies.

[0123] Next, we will explain how the query vector is generated:

[0124] In some embodiments, such as Figure 7 As shown, Figure 7 yes Figure 2 Step 104, an optional implementation flowchart, generates a query vector based on the planar features and predicted change information at the current time, including the following steps:

[0125] 104.1.1 Obtain the initial query vector;

[0126] 104.1.2 Integrate the planar features corresponding to the three orthogonal planes at the current time to obtain the current three-plane features, and predict the future 3D occupancy map at the future time by superimposing the current three-plane features and the predicted change information;

[0127] 104.1.3 The current three-plane features, predicted change information, and future 3D occupancy map are mapped to the shared latent space to obtain the latent features before change, the latent features after change, and the latent features after change;

[0128] 104.1.4 Based on the preset first weight, the latent features before and after the change are multiplied element by element to obtain the first key feature, and based on the preset second weight, the latent features before and after the change are multiplied element by element to obtain the second key feature.

[0129] 104.1.5 The first key feature, the variable potential feature and the second key feature are averaged element by element to obtain the average potential feature. The initial query vector and the average potential feature are then superimposed to obtain the query vector.

[0130] Steps 104.1.1 to 104.1.5 are described in detail below.

[0131] In some embodiments, such as Figure 8 As shown, Figure 8 This is a schematic diagram of another optional data processing flow for the intelligent driving method based on a world model provided in this application embodiment. In this diagram, the Future query is the obtained initial query vector, which is typically a learnable parameter. It serves as a general, scenario-independent planning starting point, providing a foundation for subsequently generating query vectors specific to the current context. Further, the device integrates the planar features corresponding to the three orthogonal planes at the current moment to obtain... Figure 8The current triplane feature, represented by the gray triplane icon at the top, is called the Last triplane. Then, by overlaying this current triplane feature with the previously predicted change information (i.e., the Triplane changes represented by the gray noisy triplane icon on the right in the attached figure), the future 3D occupancy map at future time points is predicted.

[0132] Furthermore, information from each stage related to scene changes is mapped to a unified feature space (shared latent space) to obtain consistent information representations across different states, thereby enabling subsequent comprehensive analysis, comparison, and prediction operations. Specifically, the system maps the current three-plane features... Predicting change information And the prediction of future 3D occupancy map The inputs are fed into three different mapping modules, which are represented by θ respectively. p θ Δ and θ f They map their respective inputs to a shared latent space, thereby obtaining the latent features before the change, the latent features after the change, and the latent features after the change, respectively.

[0133] Furthermore, learnable weights are used to modulate and extract information most relevant to the object's motion planning. Specifically, based on a preset first weight (denoted by the green ζ in the diagram)... p (Block representation), performing element-wise multiplication of the latent features before the change. The system processes the data to obtain the first key feature; simultaneously, it applies a preset second weight (denoted by the green ζ in the diagram). f (Represented by blocks), the potential features after the change are multiplied element-wise to obtain the second key feature. Through an attention mechanism, the world model is allowed to dynamically determine whether to focus more on the state features before or after the scene change during planning.

[0134] Furthermore, the world model performs element-wise averaging (Mean) on the first key feature, the changing latent feature, and the second key feature, the output of which is the average latent feature. Subsequently, the world model superimposes the obtained initial query vector with this average latent feature to obtain the final result. Figure 8 The middle mark is The query vector.

[0135] After determining the query vector, continue to combine Figure 8 This explains how to output the target vehicle's corresponding target action at a future time based on the query vector:

[0136] In some embodiments, such as Figure 9 As shown, Figure 9 yes Figure 2 Another optional implementation flowchart for step 104 involves performing feature encoding and decoding processing on the executed action sequence based on the query vector to predict the target vehicle's target action at a future time, including the following steps:

[0137] 104.2.1 Project multiple executed actions onto the same feature dimension as the query vector to obtain multiple first action projection features;

[0138] 104.2.2 Perform feature encoding processing on multiple first action projection features to obtain multiple first action encoded features, and perform feature decoding processing on multiple first action encoded features based on the query vector to predict and output the target vehicle's target execution action at future time.

[0139] Steps 104.2.1 to 104.2.2 are described in detail below.

[0140] In some embodiments, to enable historical action information to interact with scene-aware query vectors in a unified feature space, the executed action sequence is first feature-encoded using a Transformer Encoder, and the resulting first action projection features are input into a Transformer Decoder. The executed action sequence includes multiple executed actions. The executed actions can be historical control commands such as steering and acceleration, and this application embodiment does not limit this.

[0141] Furthermore, the query vector is also input into the Transformer Decoder; then, guided by the query vector, the Transformer Decoder performs feature decoding on multiple first action encoding features, thereby predicting the target's action at future time steps.

[0142] It is understood that the embodiments of this application do not use a static or simple query to plan actions. Instead, they dynamically integrate the potential features before the change, the potential features after the change, and the potential features after the change, and use learnable weights for intelligent weighing to construct a query vector with a deep understanding of the future. This query vector is then used to provide precise guidance for the subsequent Transformer Decoder, enabling it to not only refer to historical action habits but also proactively adapt to predicted future environmental changes. Ultimately, it outputs a safe and efficient target execution action, thereby solving the problem of the disconnect between perception and prediction and decision-making planning in traditional planning methods.

[0143] In some embodiments, such as Figure 10 As shown, Figure 10 yes Figure 2 Step 104 in the flowchart is another optional implementation. Based on the query vector, the executed action sequence is processed by feature encoding and decoding to predict the target vehicle's target action at a future time. It also includes the following steps:

[0144] 104.3.1 Obtain the target driving path of the target vehicle, and adjust the query vector based on the target driving path to obtain the adjusted updated query vector;

[0145] 104.3.2 Project multiple executed actions onto the same feature dimension as the updated query vector to obtain multiple second action projection features;

[0146] 104.3.3 Feature encoding is performed on multiple second action projection features to obtain multiple second action encoded features. Based on the update query vector, feature decoding is performed on multiple action encoded features to predict the target vehicle's target execution action at future time.

[0147] Steps 104.3.1 to 104.3.3 are described in detail below.

[0148] In some embodiments, to align the vehicle's short-term decisions with long-term mission objectives, the world model acquires the target vehicle's target driving path before determining the target action. The target driving path is typically generated by a global path planning module such as an in-vehicle navigation system, and includes a series of key waypoints or trajectories guiding the vehicle from its current location to its final destination. Subsequently, the world model adjusts the query vector generated in the preceding steps, which primarily reflects short-term environmental predictions and obstacle avoidance requirements, based on this target driving path. One specific implementation involves characterizing the future key waypoints on the target driving path, and then fusing their information into the original query vector through concatenation, addition, or attention mechanisms, thereby obtaining an adjusted updated query vector that incorporates both short-term safety considerations and long-term driving intentions.

[0149] For example, updating the query vector not only guides the Transformer Decoder to focus on short-term obstacle avoidance (such as predicted pedestrians ahead), but also provides it with a higher-level task constraint of "finally needing to cross the intersection ahead", thereby guiding the decoder to predict and output a more macroscopic target execution action that is more in line with the overall task requirements.

[0150] Furthermore, in order to perform subsequent encoding and decoding processing within a unified feature space, multiple executed actions are projected onto the same feature dimension as the update query vector obtained in the previous step. The specific implementation method is similar to steps 104.2.1 to 104.2.2, but the key difference is that the alignment reference of the feature dimension is changed to the update query vector that incorporates the long-term driving target, thereby obtaining multiple second action projection features. This ensures that in subsequent processing, the historical action information and the guidance information containing the global planning intent are fully compatible in terms of dimension.

[0151] Furthermore, the system performs a final action prediction that integrates long-term goals. The system first performs feature encoding on multiple second action projection features, using a transform encoder to capture the temporal dependencies of historical actions, resulting in multiple second action encoded features. The core aspect is that the system then performs feature decoding on these multiple action encoded features based on the update query vector containing long-term driving intentions. At this point,

[0152] In this way, the target vehicle can achieve intelligent driving functions with a better user experience, making this intelligent response no longer isolated or localized. For example, when faced with a suddenly appearing pedestrian, the target vehicle no longer simply performs an "emergency stop" as a pure obstacle avoidance maneuver. Instead, guided by the global goal of "ultimately passing through the intersection along the original path," it generates more advanced, human-centered, and logically consistent complex actions, such as "smoothly decelerating to allow the pedestrian to cross, and then accelerating again to resume driving." This greatly improves the planning quality and smoothness of the intelligent driving experience, upgrading vehicle behavior from simple "obstacle avoidance" to targeted "navigation," ensuring safety while also considering driving efficiency and task completion.

[0153] Back Figure 5 The Motion Planning module outputs the target's execution action; the changes in the three future planes of the multi-scale Transformer are fed into the decoder D of the world model to recover the complete 3D occupancy map at the future moment, which is used for supervision and evaluation during model training; then, the target's execution action and the complete 3D occupancy map at the future moment will work together to enable intelligent driving of the target vehicle at the future moment, achieving efficient and high-precision intelligent driving decisions.

[0154] In some embodiments, such as Figure 11 As shown, Figure 11 This is an optional world model training flowchart of the intelligent driving method based on a world model provided in this application embodiment. The world model is trained through the following steps:

[0155] Step 201: Obtain multiple panoramic images of the sample vehicle and the sequence of actions executed by the sample vehicle, and generate a sample 3D occupancy map corresponding to each panoramic image. The multiple panoramic images are from the current time and at least one historical time.

[0156] Step 202: Input multiple sample 3D occupancy maps into the initial world model. Perform feature encoding processing on each sample 3D occupancy map to obtain sample encoded features. Perform axial pooling processing on the sample encoded features along the three coordinate axes to obtain sample plane features corresponding to the three sample orthogonal planes. Determine the semantic loss value based on the sample plane features. The three coordinate axes are perpendicular to each other. Each sample orthogonal plane is defined by two different coordinate axes. An initial multi-scale Transformer is set in the initial world model.

[0157] Step 203: Using the initial multi-scale Transformer, the sample plane features of multiple sample planes under the same sample orthogonal plane are resampled to predict the sample plane change information of the current sample orthogonal plane at future time. The sample plane change information of the three sample orthogonal planes is integrated to obtain the sample prediction change information. The prediction loss value is determined based on the sample prediction change information.

[0158] Step 204: Generate a sample query vector based on the sample plane features and sample prediction change information corresponding to the current time, and perform feature encoding and decoding processing on the sequence of actions already executed by the sample based on the sample query vector, predict and output the sample execution actions of the sample vehicle at future time, and determine the action loss value based on the sample execution actions.

[0159] Step 205: Based on the semantic loss value, prediction loss value, and action loss value, adjust the model parameters of the initial world model until the preset training conditions are met to obtain the trained world model.

[0160] Steps 201 and 205 are described in detail below.

[0161] In some embodiments, the core process of this training method is consistent with the aforementioned inference method (steps 101 to 104) in architecture, and will not be repeated here. Unlike the inference method, after each key output in the training process, the system calculates the difference between it and the true label to obtain a series of loss values ​​to guide the model's learning.

[0162] Furthermore, in this embodiment, after obtaining the sample plane features, a decoder is used to reconstruct the 3D occupancy map. The semantic loss value is determined based on the difference between this reconstruction result and the actual sample 3D occupancy map. As shown in the figure, to decode and reconstruct the original 3D occupancy map, for each point pos = (x, y, z) in the scene, features are extracted from the three planes and summed. These features are then concatenated with the position encoding of pos and input to the decoder Φ. dec This is used to predict the semantic category corresponding to pos, so as to determine the semantic loss value based on the semantic category. The semantic loss value (e.g., calculated through cross-entropy loss) directly measures the model's ability to encode and decode static 3D scenes, i.e., the fidelity of scene reconstruction. The specific process can be represented by the following formula (4):

[0163]

[0164] in, Indicates each moment, Represents the loss function. and These are the cross-entropy loss and the Lovasz-softmax loss, respectively. For the 3D occupancy map of the future moment obtained from the reconstruction; S t For the corresponding truth value label.

[0165] Furthermore, the text will intersperse explanations of how the predicted loss value is determined:

[0166] In some embodiments, such as Figure 12 As shown, Figure 12 yes Figure 11 Step 203, an optional implementation flowchart, determines the prediction loss value based on sample prediction change information, including the following steps:

[0167] 203.1.1 Obtain the verification prediction change information of the sample orthogonal plane, and the verification 3D occupancy map corresponding to the verification prediction change information;

[0168] 203.1.2 Based on the sample prediction change information, determine the sample 3D occupancy map corresponding to the current time. Based on the preset first trade-off coefficient, calculate the degree of difference between the sample prediction change information and the verification prediction change information to obtain the prediction change information loss value.

[0169] 203.1.3 Based on the preset second trade-off coefficient, calculate the degree of difference between the sample 3D occupancy map and the validation 3D occupancy map to obtain the 3D occupancy map loss value;

[0170] 203.1.4 The predicted loss value is obtained by superimposing the predicted change information loss value and the 3D occupancy map loss value.

[0171] Steps 203.1.1 to 203.1.4 are described in detail below.

[0172] In some embodiments, the prediction loss value is determined by comparing the prediction result with the true future sample plane features. The specific process can be represented by the following formula (5):

[0173]

[0174] in, To verify the predicted changes, To verify the 3D occupancy map corresponding to the predicted change information; For predicting changes in the sample, S t+k 3D occupancy map of the sample; It is the weighted sum of L1 and L2 losses, defined as γ is the first trade-off coefficient, and ξ is the second trade-off coefficient. The specific values ​​of the first and second trade-off coefficients can be set according to the actual situation, and this application does not limit them.

[0175] Next, we will continue to explain how to determine the action loss value:

[0176] In this process, after predicting the sample actions of the output sample vehicle at future times, the world model compares these actions with the actual future actions in the sample data to determine the action loss value. The specific process can be represented by the following formula (6):

[0177]

[0178] in, Used to measure the L2 difference between the predicted trajectory and the actual trajectory; a t+k Perform actions for the sample. To actually perform the action.

[0179] Furthermore, based on the semantic loss value, the prediction loss value, and the action loss value calculated in the previous steps, the model parameters of the initial world model are adjusted using optimization algorithms such as gradient descent. This adjustment process is iterated repeatedly until the total loss value converges or preset training conditions such as the preset number of training rounds are met. Once the training conditions are met, this fully learned and optimized initial world model becomes a trained world model that can be used for practical deployment and has high-precision prediction and planning capabilities.

[0180] It is understood that this application's embodiments achieve precise supervision of different modules of the model by designing a multi-task, multi-stage joint loss function composed of semantic loss, prediction loss, and action loss. Semantic loss ensures the model possesses a basic understanding and reconstruction capability of the 3D world; prediction loss directly drives the model to learn the dynamic evolution laws of the physical world; and action loss tightly couples perception and prediction with the final planning decision, achieving end-to-end optimization. This clearly defined yet interconnected supervision mechanism enables the model to achieve high levels of performance in static scene understanding, dynamic change prediction, and downstream task planning, ultimately training a high-performance world model that balances lightweight computation and high-precision results in complex dynamic environments.

[0181] To enable readers to better understand the beneficial effects of the embodiments of this application, the embodiments of this application are compared with conventional methods below:

[0182] (1) In terms of network design, this application features a lighter-weight occupancy graph coding and prediction network structure, as shown in the table below:

[0183]

[0184] Compared with traditional methods, the world model of this application has a smaller feature dimension in the latent space, the fewest parameters, and the highest accuracy (mIoU and IoU) in latent feature recovery scene.

[0185] Regarding the prediction mechanism, the world model in this application only needs to learn the changes, thus the learning pressure is very low and the network converges quickly, as shown in the table below:

[0186] Model Total number of parameters (M) FPS OccWorld 72 18 This application 55 26

[0187] In this application's world model, the total number of parameters decreased by 23%, and the FPS also increased to 26 FPS.

[0188] (3) In terms of performance, the future scenarios predicted by this application are more accurate, the future motion planning is more accurate, and the collision rate is lower.

[0189]

[0190] Among them, mIoU and IoU measure the accuracy of the scene. We can see that at 0s, which is the current moment, the scene recovery accuracy is the highest. At the same time, at 1s, 2s, and 3s, which represent the prediction accuracy of the future scene, our accuracy is also the highest, and the cumulative error is also the smallest.

[0191]

[0192] Among them, L2 measures the accuracy of future trajectory planning, and Collisionrate measures the collision rate of future trajectories. It can be seen that the future trajectory predicted in this application has the highest accuracy and the lowest average collision rate.

[0193] like Figure 13 As shown, Figure 13 This is a schematic diagram of an optional device module of the intelligent driving device based on a world model provided in this application embodiment. The intelligent driving device based on a world model may include the following modules 301 to 304:

[0194] The acquisition module 301 is used to acquire multiple panoramic images of the target vehicle and the sequence of actions executed by the target vehicle, and generate a 3D occupancy map corresponding to each panoramic image. The multiple panoramic images are from the current time and at least one historical time.

[0195] The triorthogonal plane processing module 302 is used to input multiple 3D occupancy maps into the trained world model, perform feature encoding processing on each 3D occupancy map to obtain encoded features, and perform axial pooling processing on the encoded features along the three coordinate axes to obtain the planar features corresponding to the three orthogonal planes. The three coordinate axes are perpendicular to each other, and each orthogonal plane is defined by two different coordinate axes. The world model is equipped with a multi-scale Transformer.

[0196] The resampling processing module 303 is used to use a multi-scale Transformer to resample multiple planar features under the same orthogonal plane, predict the planar change information of the current orthogonal plane at future time, and integrate the planar change information of the three orthogonal planes to obtain the predicted change information.

[0197] The target prediction module 304 is used to generate a query vector based on the planar features and predicted change information at the current time, and to perform feature encoding and decoding processing on the executed action sequence based on the query vector, and predict and output the target vehicle's target execution action at a future time.

[0198] The specific implementation of this world-model-based intelligent driving device is basically the same as the specific implementation of the world-model-based intelligent driving method described above, and will not be repeated here.

[0199] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned intelligent driving method based on a world model. This electronic device can be any intelligent terminal, including tablet computers, in-vehicle computers, etc.

[0200] like Figure 14 As shown, Figure 14This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0201] The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0202] The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and called and executed by the processor 401 to execute the intelligent driving method based on the world model of the embodiments of this application.

[0203] Input / output interface 403 is used to implement information input and output;

[0204] The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0205] Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0206] The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0207] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described intelligent driving method based on a world model.

[0208] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0209] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0210] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0211] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0213] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0214] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0216] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0218] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An intelligent driving method based on a world model, characterized in that, include: Acquire multiple panoramic images of the target vehicle and the sequence of actions performed by the target vehicle, and generate a 3D occupancy map corresponding to each panoramic image, wherein the multiple panoramic images are from the current time and at least one historical time. Multiple 3D occupancy maps are input into a trained world model. Feature encoding is performed on each 3D occupancy map to obtain encoded features. Axial pooling is then performed on the encoded features along the three coordinate axes to obtain planar features corresponding to the three orthogonal planes. The three coordinate axes are perpendicular to each other, and each orthogonal plane is defined by two different coordinate axes. A multi-scale Transformer is set in the world model. Using the multi-scale Transformer, the planar change information of the current orthogonal plane at future times is predicted by resampling multiple planar features under the same orthogonal plane, and the planar change information of the three orthogonal planes is integrated to obtain the predicted change information. A query vector is generated based on the planar features and the predicted change information corresponding to the current moment. The executed action sequence is then processed by feature encoding and decoding based on the query vector to predict and output the target action to be executed by the target vehicle in the future.

2. The intelligent driving method based on a world model according to claim 1, characterized in that, The method of using the multi-scale Transformer to predict the planar change information of the current orthogonal plane in future time moments by resampling multiple planar features under the same orthogonal plane includes: Using the multi-scale Transformer, multiple planar features belonging to the same orthogonal plane are downsampled at different scales to obtain feature maps corresponding to each scale, and the spatial resolution of the feature maps decreases from high to low. At different scales, with a preset tracking category object as the target, feature extraction is performed on the feature map at the current scale to obtain object change information indicating the degree of change of the tracking category object at the current scale. The tracking category object is different at different scales. Based on the object change information, predict the planar change information of the current orthogonal plane at future times.

3. The intelligent driving method based on a world model according to claim 2, characterized in that, At different scales, with a preset tracking category of objects as the target, feature extraction is performed on the feature map at the current scale to obtain object change information indicating the degree of change of the tracking category of objects at the current scale, including: If the spatial resolution of the feature map at the current scale is the highest, feature extraction processing is performed on the feature map corresponding to the current scale to obtain the object change information indicating the degree of change of the tracked category object at the current scale; If the spatial resolution of the feature map at the current scale is not the highest, the feature map obtained at the next scale adjacent to the current scale is determined as the feature map to be used, and the object change information obtained at the next scale adjacent to the current scale is determined as the object change information to be used. The feature map to be used and the object change information to be used are superimposed to obtain the fusion information to be used. Based on the fusion information to be used, feature extraction processing is performed on the feature map corresponding to the current scale to obtain the object change information indicating the degree of change of the tracked category object at the current scale.

4. The intelligent driving method based on a world model according to claim 1, characterized in that, The step of generating a query vector based on the planar features and the predicted change information at the current time includes: Obtain the initial query vector; By integrating the planar features corresponding to the three orthogonal planes at the current moment, the current three-plane features are obtained. By superimposing the current three-plane features and the predicted change information, the future 3D occupancy map at the future moment is predicted. The current three-plane features, the predicted change information, and the future 3D occupancy map are respectively mapped to a shared latent space to obtain the latent features before the change, the latent features after the change, and the latent features after the change. Based on a preset first weight, the potential features before the change and the potential features after the change are multiplied element by element to obtain a first key feature. Based on a preset second weight, the potential features before the change and the potential features after the change are multiplied element by element to obtain a second key feature. The first key feature, the variable potential feature, and the second key feature are averaged element by element to obtain the average potential feature. The initial query vector and the average potential feature are then superimposed to obtain the query vector.

5. The intelligent driving method based on a world model according to claim 1, characterized in that, The sequence of executed actions includes multiple executed actions; The step of performing feature encoding and decoding processing on the executed action sequence based on the query vector to predict and output the target action to be executed by the target vehicle at a future time includes: Multiple executed actions are projected onto the same feature dimension as the query vector to obtain multiple first action projection features; Multiple first action projection features are processed by feature encoding to obtain multiple first action encoding features. Based on the query vector, multiple first action encoding features are processed by feature decoding to predict and output the target vehicle's target action at a future time.

6. The intelligent driving method based on a world model according to claim 1, characterized in that, The step of performing feature encoding and decoding processing on the executed action sequence based on the query vector to predict and output the target action to be executed by the target vehicle at a future time also includes: Obtain the target driving path of the target vehicle, and adjust the query vector based on the target driving path to obtain the adjusted updated query vector; Multiple executed actions are projected onto the same feature dimension as the updated query vector to obtain multiple second action projection features; Multiple second action projection features are subjected to feature encoding processing to obtain multiple second action encoding features. Based on the update query vector, multiple action encoding features are subjected to feature decoding processing to predict and output the target vehicle's target action at a future time.

7. The intelligent driving method based on a world model according to claim 1, characterized in that, The world model is trained through the following steps: Acquire multiple panoramic images of the sample vehicle and the sequence of actions performed by the sample vehicle, and generate a sample 3D occupancy map corresponding to each of the panoramic images. The multiple panoramic images are from the current time and at least one historical time. Multiple sample 3D occupancy maps are input into the initial world model. Feature encoding is performed on each sample 3D occupancy map to obtain sample encoded features. Axial pooling is then performed on the sample encoded features along the three coordinate axes to obtain sample plane features corresponding to the three sample orthogonal planes. The semantic loss value is determined based on the sample plane features. The three coordinate axes are perpendicular to each other, and each sample orthogonal plane is defined by two different coordinate axes. An initial multi-scale Transformer is set in the initial world model. Using the initial multi-scale Transformer, the sample plane features of multiple sample planes under the same sample orthogonal plane are resampled to predict the sample plane change information of the current sample orthogonal plane at future time. The sample plane change information of the three sample orthogonal planes is integrated to obtain the sample prediction change information. The prediction loss value is determined based on the sample prediction change information. A sample query vector is generated based on the sample plane features and the sample prediction change information corresponding to the current time. The sample query vector is used to perform feature encoding and decoding on the sequence of actions already performed by the sample. The sample vehicle's sample execution actions in the future are predicted and output. The action loss value is determined based on the sample execution actions. Based on the semantic loss value, the prediction loss value, and the action loss value, the model parameters of the initial world model are adjusted until the preset training conditions are met, thus obtaining the trained world model.

8. The intelligent driving method based on a world model according to claim 7, characterized in that, Determining the prediction loss value based on the sample prediction change information includes: Obtain the verification prediction change information of the sample orthogonal plane, and the verification 3D occupancy map corresponding to the verification prediction change information; Based on the sample prediction change information, determine the sample 3D occupancy map corresponding to the current moment. Based on the preset first trade-off coefficient, calculate the degree of difference between the sample prediction change information and the verification prediction change information to obtain the prediction change information loss value. Based on a preset second tradeoff coefficient, the degree of difference between the sample 3D occupancy map and the verification 3D occupancy map is calculated to obtain the 3D occupancy map loss value. The predicted loss value is obtained by superimposing the predicted change information loss value and the 3D occupancy map loss value.

9. An intelligent driving device based on a world model, characterized in that, include: The acquisition module is used to acquire multiple panoramic images of the target vehicle and the sequence of actions performed by the target vehicle, and generate a 3D occupancy map corresponding to each panoramic image. The multiple panoramic images are from the current time and at least one historical time. The three orthogonal plane processing module is used to input multiple 3D occupancy maps into a trained world model, perform feature encoding processing on each 3D occupancy map to obtain encoded features, and perform axial pooling processing on the encoded features along the three coordinate axes to obtain the planar features corresponding to the three orthogonal planes. The three coordinate axes are perpendicular to each other, and each orthogonal plane is defined by two different coordinate axes. The world model is equipped with a multi-scale Transformer. The resampling processing module is used to utilize the multi-scale Transformer to resample multiple planar features under the same orthogonal plane, predict the planar change information of the current orthogonal plane at future times, and integrate the planar change information of the three orthogonal planes to obtain the predicted change information. The target prediction module is used to generate a query vector based on the planar features and the predicted change information at the current time, and to perform feature encoding and decoding processing on the executed action sequence based on the query vector, and to predict and output the target action to be executed by the target vehicle at a future time.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the intelligent driving method based on a world model as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the intelligent driving method based on the world model as described in any one of claims 1 to 8.

Images