Model training method and device, automatic driving method and system, mobile device

Abstract

This disclosure provides a model training method and apparatus, an autonomous driving method and system, and a mobile device. The model training method includes: acquiring a first BEV feature obtained by a spatial perception module based on a multi-view image and a second BEV feature encoding at a first time moment; performing 3D reconstruction based on the first BEV feature to obtain a first 3D reconstruction result; iteratively training the spatial perception module based on a first training sample including the first 3D reconstruction result and a first reference point cloud; acquiring at least one fifth BEV feature corresponding to at least one fifth time moment predicted by a future decoding module based on a third BEV feature and motion information of the second mobile device at at least one fifth time moment; performing 3D reconstruction based on the at least one fifth BEV feature to obtain at least one second 3D reconstruction result; and iteratively training the future decoding module based on a second training sample including at least one second 3D reconstruction result and at least one second reference point cloud.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology and autonomous driving technology, and more particularly to a model training method and apparatus, an autonomous driving method and system, and a mobile device. Background Technology

[0002] Data-driven learning has greatly promoted the development of autonomous driving technology, enhancing core tasks such as perception, decision-making, and planning. With the rapid development of autonomous driving technology, accurately understanding and predicting complex and ever-changing driving environments to provide accurate information support for subsequent perception, planning, and control decisions, thereby improving robustness and safety in various autonomous driving tasks, has become a critical issue that urgently needs to be addressed. Summary of the Invention

[0003] This disclosure provides a model training method and apparatus, an autonomous driving method and system, a mobile device, an electronic device, a computer program product, and a computer-readable storage medium to accurately understand and predict the driving environment.

[0004] One aspect of this disclosure provides a model training method applied to a world model, the world model including a spatial perception module and a future decoding module, the method comprising:

[0005] The spatial perception module encodes a first BEV feature based on the multi-view images at the first moment and the second bird's-eye view BEV feature. The multi-view images include images of the first scene collected by multiple visual sensors on the first mobile device in different orientations. The second bird's-eye view BEV feature is the BEV feature encoded by the spatial perception module that corresponds to the moment before the first moment.

[0006] Based on the first BEV feature, a three-dimensional reconstruction is performed to obtain the first three-dimensional reconstruction result;

[0007] The spatial perception module is iteratively trained based on at least one first training sample; wherein, one of the at least one first training samples includes the first three-dimensional reconstruction result and a first reference point cloud, and the first reference point cloud is a three-dimensional point cloud generated by scanning the first scene using the lidar sensor on the first mobile device at the first time.

[0008] Based on the third BEV feature and the motion information of the second mobile device at at least a fifth moment, the future decoding module predicts at least one fifth BEV feature corresponding to the at least one fifth moment; wherein, the third BEV feature is the BEV feature corresponding to the third moment encoded by the trained spatial perception module, and the at least one fifth moment is at least one different moment after the third moment.

[0009] Three-dimensional reconstruction is performed based on the at least one fifth BEV feature to obtain at least one second three-dimensional reconstruction result;

[0010] The future decoding module is iteratively trained based on at least one second training sample; wherein, one of the at least one second training samples includes at least one second 3D reconstruction result and at least one second reference point cloud, and each second reference point cloud is a 3D point cloud generated by scanning the second scene using the lidar sensor on the second mobile device at each of the fifth time moments.

[0011] Another aspect of this disclosure provides an autonomous driving method, comprising:

[0012] Using a world model, based on multi-view images of the current moment and previous BEV features corresponding to the previous moment, at least one future BEV feature corresponding to at least one future moment is predicted; wherein, the multi-view images include images collected by multiple visual sensors on the mobile device facing different directions regarding the external environment of the mobile device, and the previous moment is a moment before the current moment; the world model includes a spatial perception module and a future decoding module, and the world model is trained by the model training method described in any embodiment of this disclosure;

[0013] Using the autonomous driving model, tasks corresponding to the autonomous driving model are processed based on at least one future BEV feature to obtain at least one task processing result, so as to perform autonomous driving control on the mobile device based on the at least one task processing result.

[0014] Another aspect of this disclosure provides a model training apparatus for use with a world model, the world model including a spatial perception module and a future decoding module, the apparatus comprising:

[0015] The first acquisition module is configured to: input the multi-view image and the second bird's-eye view BEV feature at the first moment into the spatial perception module, and acquire the first BEV feature encoded by the spatial perception module; wherein, the multi-view image includes images of the first scene collected by multiple visual sensors in different orientations on the first mobile device, and the second bird's-eye view BEV feature is the BEV feature encoded by the spatial perception module corresponding to the moment before the first moment.

[0016] The 3D reconstruction module is configured to perform 3D reconstruction based on the first BEV feature to obtain a first 3D reconstruction result.

[0017] The first training module is configured to iteratively train the spatial perception module based on at least one first training sample; wherein, one of the at least one first training samples includes the first three-dimensional reconstruction result and a first reference point cloud, and the first reference point cloud is a three-dimensional point cloud generated by scanning the first scene using a lidar sensor on the first mobile device at the first time.

[0018] The second acquisition module is configured to: input the third BEV feature and the motion information of the second mobile device at at least a fifth moment into the future decoding module, and acquire at least one fifth BEV feature corresponding to the at least one fifth moment predicted by the future decoding module; wherein, the third BEV feature is the BEV feature corresponding to the third moment encoded by the trained spatial perception module, and the at least one fifth moment is at least one different moment after the third moment.

[0019] The three-dimensional reconstruction module is further configured to: perform three-dimensional reconstruction based on the at least one fifth BEV feature to obtain at least one second three-dimensional reconstruction result;

[0020] The second training module is configured to iteratively train the future decoding module based on at least one second training sample; wherein, one of the at least one second training samples includes the at least one second 3D reconstruction result and at least one second reference point cloud, and each second reference point cloud is a 3D point cloud generated by scanning the second scene using a lidar sensor on the second mobile device at each of the fifth time moments.

[0021] In another aspect of the present disclosure, a world model is provided, including a spatial perception module and a future decoding module, wherein the world model is trained by the model training method described in any embodiment of the present disclosure.

[0022] In another aspect of the present disclosure, an autonomous driving system is provided, including a world model and an autonomous driving model; the world model includes a spatial perception module and a future decoding module, and the world model is trained by the model training method described in any embodiment of the present disclosure.

[0023] The world model is configured to: predict at least one future BEV feature corresponding to at least one future moment based on the multi-view image at the current moment and the previous BEV feature corresponding to the previous moment; wherein, the multi-view image includes images collected by multiple visual sensors on the mobile device in different orientations for the external environment of the mobile device, and the previous moment is a moment before the current moment;

[0024] The autonomous driving model is configured to perform task processing corresponding to the autonomous driving model based on the at least one future BEV feature, and obtain at least one task processing result, so as to perform autonomous driving control on the mobile device based on the at least one task processing result.

[0025] In another aspect of the present disclosure, a mobile device is provided, wherein a plurality of visual sensors facing different directions and an autonomous driving system as described in any embodiment of the present disclosure are deployed on the mobile device;

[0026] The plurality of visual sensors are configured to: acquire images at preset time intervals and sequentially input the multi-view images acquired in each acquisition into the world model of the autonomous driving system, so as to perform autonomous driving control of the mobile device based on at least one task processing result output by the autonomous driving model in the autonomous driving system.

[0027] In another aspect, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above embodiments of the present disclosure.

[0028] In another aspect of this disclosure, an electronic device is provided, the electronic device including a memory and a processor:

[0029] The memory is used to store the processor-executable instructions;

[0030] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the above embodiments.

[0031] In another aspect of this disclosure, a computer program product is provided that, when an instruction processor in the computer program product is executed, performs the method described in any of the above embodiments.

[0032] Based on the embodiments of this disclosure, a novel and scalable world model is provided for an autonomous driving system. This world model includes a spatial perception module and a future decoding module, decoupling spatial encoding and future decoding. The learning process of the world model is decomposed into two independent stages: spatial perception representation learning and latent space future decoding. In the spatial perception representation learning stage, the spatial perception module encodes first BEV features based on multi-view images at a first time moment and second BEV features corresponding to the time moment preceding the first time moment. Then, 3D reconstruction is performed based on these first BEV features to obtain a first 3D reconstruction result. Subsequently, the spatial perception module is iteratively trained based on at least one first training sample, where each first training sample includes the first 3D reconstruction result and a first reference point cloud. The first reference point cloud is generated using a LiDAR on a first mobile device. The sensor generates a 3D point cloud by scanning the first scene at the first moment. In the future decoding stage, the future decoding module predicts at least one fifth BEV feature corresponding to at least one fifth moment based on the third BEV feature corresponding to the third moment encoded by the trained spatial perception module and the motion information of the second mobile device at at least one fifth moment. 3D reconstruction is then performed based on the at least one fifth BEV feature to obtain at least one second 3D reconstruction result. Then, the future decoding module is iteratively trained based on at least one second training sample, where each second training sample includes at least one second 3D reconstruction result and at least one second reference point cloud. Each second reference point cloud is a 3D point cloud generated by scanning the second scene at each fifth moment using the lidar sensor on the second mobile device.Therefore, this embodiment employs a decoupled training method, performing spatial encoding and future decoding separately. This allows the world model to focus on learning the spatial perception representation and temporal dynamics of the physical world during the spatial encoding stage. It can learn more robust and universal world features while maintaining spatial structure information. In the future decoding stage, this spatial structure information is used to achieve spatiotemporally consistent future predictions, improving the accuracy of future prediction results. This not only reduces the learning difficulty and improves the training efficiency of the world model but also enables it to achieve accurate physical world state encoding and future state decoding, learning knowledge applicable to various downstream tasks and improving its generalization ability when applied to various tasks. Furthermore, spatial encoding compresses the physical world state... Modeling is performed within the BEV implicit space, where each grid corresponds to a spatial region in the physical world. This approach preserves the natural spatial structure attributes of the physical world while compressing high-dimensional information. In the latent space future decoding, spatial structure attribute information is used for future prediction, which helps maintain spatiotemporal consistency (i.e., consistency in both time and spatial dimensions). Furthermore, point clouds generated by LiDAR sensors for the same scene at the same time are used as supervision, and the world model is trained in an unsupervised manner. This enables scalable learning, and the entire learning process does not require manually labeled training datasets. This avoids the impact of manually labeled training datasets on model training performance, reduces training costs, and improves training effectiveness and efficiency.

[0033] The world model trained in this embodiment decouples spatial encoding and future decoding. During spatial encoding, it focuses on learning the spatial perception representation and temporal dynamics of the physical world, learning more robust and universal world features while preserving spatial structure information. In the future decoding stage, this spatial structure information is used to achieve spatiotemporally consistent future predictions, improving the accuracy of future prediction results. Furthermore, by compressing the state of the physical world into the BEV implicit space and modeling within it, where each grid in the BEV corresponds to a spatial region in the physical world, both the natural spatial structure attributes of the physical world are preserved, and high-dimensional information compression is achieved, enabling spatiotemporally consistent future predictions. Additionally, since the world model learns world knowledge universally applicable to various downstream tasks rather than specific tasks, it possesses generalization capabilities for various tasks, flexibly supporting different types of downstream applications, such as scene understanding and trajectory planning.

[0034] When the world model trained in the embodiments of this disclosure is applied to an autonomous driving system, it can learn and understand the operating rules of the real physical world from multi-view images, accurately achieve spatiotemporally consistent future prediction, accurately understand and predict future scene states, such as the movement trajectories of other vehicles and pedestrians, changes in the road environment, etc., and obtain general future BEV features. This provides accurate information support for subsequent tasks such as perception, planning and control decisions. When the autonomous driving model of downstream tasks performs task processing based on these future BEV features, it can effectively improve the accuracy of task processing results, thereby improving the safety of mobile device driving. Since the world model learns world knowledge that is universally applicable to various downstream tasks and is not specific to any particular task, it has the ability to generalize when applied to various tasks and can flexibly support different types of downstream applications, such as scene understanding and trajectory planning. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of a world model provided in an exemplary embodiment of this disclosure.

[0036] Figure 2 This is a schematic flowchart of a model training method provided in an exemplary embodiment of this disclosure.

[0037] Figure 3 This is an exemplary flowchart illustrating the encoding of the first BEV feature in an embodiment of the model training method disclosed herein.

[0038] Figure 4 This is an exemplary flowchart illustrating the iterative training of the spatial perception module in an embodiment of the model training method disclosed herein.

[0039] Figure 5 This is an exemplary flowchart illustrating how a mobile device predicts at least one fifth BEV feature in an embodiment of the model training method disclosed herein.

[0040] Figure 6 This is another exemplary flowchart illustrating the process by which a mobile device predicts at least one fifth BEV feature in an embodiment of the model training method disclosed herein.

[0041] Figure 7 This is a schematic diagram of the structure of a world model provided in another exemplary embodiment of this disclosure.

[0042] Figure 8 This is a flowchart illustrating a model training method provided in another exemplary embodiment of this disclosure.

[0043] Figure 9 This is a flowchart illustrating an exemplary embodiment of the autonomous driving method provided in this disclosure.

[0044] Figure 10This is an exemplary flowchart illustrating how at least one future BEV feature is predicted using a world model in an embodiment of the autonomous driving method disclosed herein.

[0045] Figure 11 This is an exemplary flowchart illustrating how at least one future BEV feature is predicted by a future decoding module in an embodiment of the autonomous driving method disclosed herein.

[0046] Figure 12 This is a flowchart illustrating an autonomous driving method provided in another exemplary embodiment of this disclosure.

[0047] Figure 13 This is a flowchart illustrating an autonomous driving method provided in yet another exemplary embodiment of this disclosure.

[0048] Figure 14 This is a schematic diagram of the structure of a model training apparatus provided in an exemplary embodiment of the present disclosure.

[0049] Figure 15 This is a schematic diagram of the structure of a model training apparatus provided in another exemplary embodiment of this disclosure.

[0050] Figure 16 This is a schematic diagram of the structure of an autonomous driving system provided in an exemplary embodiment of the present disclosure.

[0051] Figure 17 This is a schematic diagram of the structure of an autonomous driving system provided in another exemplary embodiment of this disclosure.

[0052] Figure 18 This is a schematic diagram of the structure of a mobile device provided in an exemplary embodiment of this disclosure.

[0053] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0054] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0055] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0056] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0057] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0058] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0059] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0060] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0061] This disclosure outlines

[0062] With the rapid development of autonomous driving technology, accurately understanding and predicting complex and ever-changing driving environments to provide accurate information support for subsequent perception, planning, and control decisions has become a key challenge.

[0063] World models are deep learning models that can model and predict the real world, and are currently widely used in key aspects of autonomous driving such as perception, decision-making, and planning.

[0064] In related technologies, when world models are applied to the field of autonomous driving, they are usually pre-trained first. After pre-training, the world model is fine-tuned using manually labeled training datasets for the specific task to be applied. The fine-tuned world model is then used for processing that specific task.

[0065] In implementing the embodiments of this disclosure, the inventors discovered through research that, since fine-tuning the world model for a specific task inevitably changes the model parameters, the world knowledge learned by the world model during pre-training is gradually lost. This makes it difficult for the model to fully utilize the scene understanding capabilities gained during the pre-training stage, and the trained world model can only be used for processing specific tasks, but cannot be applied to other tasks. In addition, since fine-tuning the world model using manually labeled training datasets may affect the training effect of the world model due to the limited training data, it also leads to high training costs and low training efficiency.

[0066] Exemplary methods

[0067] Figure 1 This is a schematic diagram of the structure of a world model provided in an exemplary embodiment of this disclosure. For example... Figure 1 As shown, the world model includes a spatial perception module 102 and a future decoding module 104. This world model can be trained using the model training method of any embodiment of this disclosure. The world model of this disclosure embodiment will be described below in conjunction with the model training method of this disclosure embodiment.

[0068] Figure 2 This is a schematic flowchart of a model training method provided in an exemplary embodiment of this disclosure, which is applied to... Figure 1 The world model shown. (As shown) Figure 2 As shown, the model training method in this embodiment includes:

[0069] 202. The spatial perception module encodes the first BEV feature based on the multi-view image and the second bird's eye view (BEV) feature at the first moment.

[0070] In any embodiment of this disclosure, the features in the BEV feature latent space, where the latent space is a compressed feature space in a neural network, are used to represent the essential features of high-dimensional input data in a low-dimensional form. A BEV feature can be a feature map of size H*W*C, where H, W, and C represent the width, height, and number of channels of the feature map, respectively. A BEV feature includes multiple grids, each corresponding to a spatial region in the physical world.

[0071] The aforementioned multi-view images include images captured by multiple visual sensors (e.g., cameras) on a first mobile device (e.g., a vehicle, an intelligent robot, etc.) facing different directions, targeting a first scene. The second BEV feature is the BEV feature encoded by the spatial perception module corresponding to the moment preceding the first moment. The moment preceding the first moment can be an adjacent moment preceding the first moment, or it can be an adjacent moment with a preset number of moments (e.g., 2) preceding the first moment. This disclosure does not impose any limitations on this.

[0072] The time in any embodiment of this disclosure is the time when the corresponding visual sensor performs each image acquisition according to a preset time interval. The preset time interval can be determined according to actual needs, such as 0.001 seconds (S), 0.03 seconds, etc., and this disclosure does not limit it.

[0073] 204. Based on the first BEV features, perform three-dimensional (3D) reconstruction to obtain the first three-dimensional reconstruction result.

[0074] 206. The spatial perception module is iteratively trained based on at least one first training sample until the first preset training completion condition is met, and a trained spatial perception module is obtained.

[0075] Among them, at least one of the first training samples includes a first 3D reconstruction result and a first reference point cloud, which is a 3D point cloud generated by scanning a first scene using a lidar sensor on a first mobile device at a first moment.

[0076] In an alternative example, a lidar sensor can be deployed on a first mobile device to scan the external scene of the first mobile device at a corresponding scanning angle. The distance between the device and objects in the external scene is measured by emitting laser pulses and receiving reflected signals. The time required for the laser pulse to travel from emission to return is calculated and converted into distance. Combined with other relevant information (such as scanning angle, position and attitude data of the lidar sensor), a three-dimensional point cloud is generated. Each point in the three-dimensional point cloud is discretely distributed in three-dimensional space and carries three-dimensional spatial coordinates (x, y, z) representing its physical position, which can be used to describe the geometry of three-dimensional objects or scenes.

[0077] 208. Through the future decoding module, based on the third BEV feature and the motion information of the second mobile device at at least a fifth time moment, at least one fifth BEV feature corresponding to at least a fifth time moment is predicted.

[0078] The third BEV feature is the BEV feature corresponding to the third time point obtained by encoding the trained spatial perception module. The at least one fifth time point is at least one different time point after the third time point, that is, each fifth time point is a future time point relative to the third time point. The at least one fifth time point can be at least one consecutive time point adjacent to the third time point, or it can be at least one time point selected after the third time point according to the method of selecting one time point at each preset number of time points (e.g., 2 time points). This embodiment of the present disclosure does not limit this.

[0079] The motion information in the embodiments of this disclosure may include, for example, at least one of the following: motion speed, acceleration, etc.

[0080] 210, perform three-dimensional reconstruction based on at least one fifth BEV feature to obtain at least one second three-dimensional reconstruction result.

[0081] 212, the future decoding module is iteratively trained based on at least one second training sample until the second preset training completion condition is met, and the trained future decoding module is obtained.

[0082] Among them, at least one of the second training samples includes at least one second 3D reconstruction result and at least one corresponding second reference point cloud. Each second reference point cloud is a 3D point cloud generated by scanning the second scene at each fifth time using a lidar sensor on a second mobile device (e.g., a vehicle, an intelligent robot, etc.). The specific acquisition method of each second reference point cloud can be referred to the implementation example in operation 206 above, and will not be repeated here.

[0083] Based on this embodiment, a novel and scalable world model is provided for autonomous driving systems. This world model includes a spatial perception module and a future decoding module, decoupling spatial encoding and future decoding. The learning process of the world model is decomposed into two independent stages: spatial perception representation learning and latent space future decoding. This allows the world model to focus on learning the spatial perception representation and temporal dynamics of the physical world during the spatial encoding stage, enabling it to learn more robust and universal world features while maintaining spatial structure information. In the future decoding stage, this spatial structure information is used to achieve spatiotemporally consistent future predictions, improving the accuracy of future prediction results. This not only reduces the learning difficulty and improves the training efficiency of the world model but also enables it to achieve accurate physical world state encoding and future state decoding, learning knowledge applicable to various downstream tasks and improving the overall performance of the world model. The model demonstrates strong generalization ability across various tasks. Furthermore, by spatially encoding the state of the physical world into the BEV implicit space, modeling is performed within this space. Each grid in the BEV corresponds to a spatial region in the physical world, preserving the natural spatial structure attributes of the physical world while achieving high-dimensional information compression. In the future decoding of the implicit space, spatial structure attribute information is used for future prediction, which helps maintain spatiotemporal consistency (i.e., consistency in both time and spatial dimensions). In addition, point clouds generated by LiDAR sensors for the same scene at the same time are used as supervision, and the world model is trained in an unsupervised manner, achieving scalable learning. The entire learning process does not require manually labeled training datasets, avoiding the impact of manually labeled training datasets on model training performance, reducing training costs, and improving training effectiveness and efficiency.

[0084] Based on the decoupling of spatial encoding and future decoding in this embodiment, the trained world model can focus on learning the spatial perception representation and temporal dynamics of the physical world during the spatial encoding stage. This allows it to learn more robust and universal world features while preserving spatial structure information. In the future decoding stage, this spatial structure information is used to achieve spatiotemporally consistent future predictions, improving the accuracy of future prediction results. Furthermore, by compressing the state of the physical world into the BEV implicit space and modeling within it, where each grid in the BEV corresponds to a spatial region in the physical world, both the natural spatial structure attributes of the physical world are preserved, and high-dimensional information compression is achieved, enabling spatiotemporally consistent future predictions. Additionally, since the world model learns world knowledge universally applicable to various downstream tasks rather than specific tasks, it possesses generalization capabilities applicable to various tasks, flexibly supporting different types of downstream applications, such as scene understanding and trajectory planning.

[0085] Figure 3 This is an exemplary flowchart illustrating the encoding of the first BEV feature in an embodiment of the model training method disclosed herein. Figure 3 As shown, in Figure 2 In some implementations of the illustrated embodiments, in operation 202, the spatial perception module can perform the following operations:

[0086] In 2022, for example, a two-dimensional (2D) convolutional neural network (CNN) was used to extract features from each image in the multi-view images at the first time step, resulting in multiple image features.

[0087] In 2024, for example, a two-dimensional to three-dimensional transformation network was used to transform multiple image features from two-dimensional space to three-dimensional space to obtain multiple three-dimensional features.

[0088] In 2026, for example, a temporal cross-attention module is used to perform temporal cross-attention processing on the above multiple three-dimensional features and the second BEV feature to obtain the first BEV feature.

[0089] Based on this embodiment, BEV features of the latent space can be generated based on multi-view images to represent the spatial structure information of the physical world.

[0090] Figure 4 This is an exemplary flowchart illustrating the iterative training of the spatial perception module in an embodiment of the model training method disclosed herein. Figure 4 As shown, in Figures 2-3 In some implementations of any of the illustrated embodiments, operation 206 may include:

[0091] 2062, for each of the first training samples in at least one of the above-mentioned first training samples, obtain the first difference between the first three-dimensional reconstruction result and the first reference point cloud in the first training sample.

[0092] 2064. Based on at least one difference corresponding to at least one first training sample, the spatial perception module is iteratively trained until the first preset training completion condition is met, and a trained spatial perception module is obtained.

[0093] In some implementations of the embodiments of this disclosure, the first preset training completion condition may include at least one of the following: the number of iterations of training reaches a corresponding preset number (e.g., 1000 times), at least one difference corresponding to at least one first training sample is less than a first preset threshold, etc., and the embodiments of this disclosure do not limit this.

[0094] Based on this embodiment, for each first training sample, the first reference point cloud can be used as supervision to obtain the first difference (which can be called reconstruction loss) between the first 3D reconstruction result and the first reference point cloud. The spatial perception module is iteratively trained through at least one difference corresponding to at least one first training sample, so that the trained spatial perception module can accurately learn the spatial perception representation and temporal dynamics of the physical world. While maintaining spatial structure information, it learns more robust and universal world features to improve the accuracy of future prediction results.

[0095] Optionally, in some implementations of the embodiments of this disclosure, the operator 204 may transform the first BEV feature into a three-dimensional density field. In this embodiment, the first three-dimensional reconstruction result includes the three-dimensional density field.

[0096] For example, in an alternative example of the above implementation, the first BEV feature Bt can be transformed into a three-dimensional density field Dt via a channel to the height module in the following manner:

[0097] Dt=Conv(Bt,C,Z) Formula (1)

[0098] In formula (1), Conv(Bt, C, Z) represents a 2D convolutional layer (input, input channel, output channel), which converts the input (i.e. the first BEV feature) from the number of channels C to the number of channels Z of the three-dimensional density field Dt, and obtains the three-dimensional density field Dt. The size of the three-dimensional density field Dt is H*W*Z.

[0099] Accordingly, in the above operation 2062, the first difference between the first 3D reconstruction result and the first reference point cloud in the first training sample can be obtained in the following way:

[0100] A1, obtain at least one sampling point corresponding to each of the multiple first reference points.

[0101] The first reference point refers to a point in the corresponding first reference point cloud. Multiple first reference points can be sampled from the first reference point cloud based on a preset sampling method, such as random sampling or sampling according to a preset sampling ratio. This embodiment does not limit the specific sampling method used. At least one of the above sampling points is obtained by sampling a three-dimensional ray from a lidar sensor along a parameterized corresponding reference point.

[0102] For example, in an optional example, based on a preset sampling method, Np points are obtained from the first reference point cloud, whose set P = {pi | i = 1, 2, ..., Np}, where pi ∈ R. 3 The vector representing the coordinates of reference point Pi, where Np is an integer greater than 0, is used. These reference points are then reparameterized into 3D rays from the LiDAR sensor. For each reference point Pi, n points are sampled along its 3D ray to obtain n sampling points xi (also called path points), where n is an integer greater than 1. The 3D coordinates of the n sampling points are as follows:

[0103]

[0104] In formula (2), j represents the index of the sampling point, o represents the coordinates of the lidar sensor, and λ = {λ1, λ2, ..., λ} n} represents a predefined distance, |r i | Indicates the true depth of the reference point.

[0105] A2, perform three-dimensional interpolation on the sampling points corresponding to the above three-dimensional density field and multiple first reference points to obtain the first occupancy value.

[0106] A3, based on the first occupancy value, adopts a classification-based discrete preset method to determine the depth values ​​of multiple first reference points, thereby obtaining multiple first predicted depth values.

[0107] For example, in one optional example, a classification-based discrete preset method is used to discretize the first occupancy value from a continuous depth value into multiple depth segments for each first reference point, and to determine the sampling point g closest to the reference point from the n sampling points corresponding to the reference point in the multiple depth segments. i (Referred to as the target sampling point), as shown below:

[0108]

[0109] Using the target sampling point g iThe depth value is used as the predicted depth value (i.e., the first predicted depth value) of the reference point. The median, minimum, or maximum depth value of the depth segment corresponding to the target sampling point can be preset as the depth value of the target sampling point; this embodiment does not impose any limitations on this.

[0110] A4, based on the first occupancy value, adopts an integral-based continuous rendering method to determine the depth values ​​of multiple first reference points, thereby obtaining multiple second predicted depth values.

[0111] For example, in one alternative example, using an integral-based continuous rendering method, the depth value of each first reference point is directly calculated based on the first occupancy value to obtain the second predicted depth value d. i :

[0112]

[0113] In formula (4), δ j =λ j -λ j-1 (λ0=o) represents the distance between two adjacent sampling points. This represents the occupancy value corresponding to sampling point j.

[0114] A5. Based on the first difference between multiple reference depth values ​​and multiple first predicted depth values ​​corresponding to multiple first reference points in the first reference point cloud, the second difference between multiple reference depth values ​​and multiple second predicted depth values, and the balance coefficient between the first difference and the second difference, a first loss value corresponding to the first training sample is determined. In this embodiment, the aforementioned first difference includes the first loss value.

[0115] For example, in an optional example, the first loss value corresponding to each first training sample can be determined in the following way:

[0116] In formula (5), This represents a discrete preset method based on classification, where the sampling point g is determined by the above formula (3). i The depth value, Indicates the first difference, |||r i |-d i || represents the second difference, ω represents the balance coefficient between the first difference and the second difference, and ω takes a value greater than 0. The specific value of ω can be set according to actual needs, and this embodiment does not limit it.

[0117] Based on the embodiments of this disclosure, at least one point is obtained from the first reference point cloud as a first reference point, and a first difference is obtained between the first 3D reconstruction result in the first training sample and the first reference point cloud. Voxel-based downsampling is used to reduce point redundancy, thereby reducing computational workload and improving computational efficiency. When obtaining the first difference between the first 3D reconstruction result in the first training sample and the first reference point cloud, a classification-based discrete preset method is used to supervise the 3D reconstruction of BEV features, which can help the spatial perception module learn clear and sharp geometric structures in the physical world. An integral-based continuous rendering method is used to estimate continuous depth values, and distance is used to measure the depth prediction error to ensure a smooth transition in depth prediction. This can ensure that the features learned by the spatial perception module are not interfered with by noisy point clouds. The embodiments of this disclosure use detailed and accurate geometric information of the first reference point cloud as supervision, and combine the classification-based discrete preset method and the integral-based continuous rendering method to help the spatial perception module learn how to extract world latent space codes with 3D structures from images, improve the spatial perception module's comprehensive understanding of the physical world, and thus improve the accuracy and richness of spatial representation.

[0118] Alternatively, in an optional example of the above implementation, the multiple first reference points can be sampled from the first reference point cloud based on the following hierarchical point sampling method:

[0119] B1. For the first reference point cloud, sparse point sampling is performed according to a first sampling ratio (e.g., 100:1, i.e., 1 point is sampled for every 100 points) to obtain the first sampling point, which includes at least one point obtained from the sampling.

[0120] In operation B1, since point sampling is performed on the entire first reference point cloud according to the first sampling ratio, this can be referred to as uniform sampling. In operation B2, dense point sampling is performed on the regions of interest (ROI) in the first reference point cloud, for example, according to the second sampling ratio (e.g., 20:1, i.e., 1 point is sampled for every 20 points), to obtain the second sampling points, which include multiple points.

[0121] In a practical implementation, a detector can be used to detect objects of a preset category (such as moving targets like vehicles, pedestrians, and animals) in the first reference point cloud to obtain regions of interest with high confidence scores.

[0122] In operation B2, since point sampling is performed only on the region of interest in the first reference point cloud, it can be called target-guided sampling.

[0123] B3, based on the first sampling point and the second sampling point, yields multiple first reference points.

[0124] For example, the first sampling point can be merged with the second sampling point, and duplicates of the same sampling point can be removed to obtain multiple first reference points.

[0125] The inventors of this disclosure have discovered through research that in real-world scenarios, since most points are static background elements (such as buildings and trees along the roadside), uniform sampling ratios during model training may lead to the neglect of significant moving targets. This disclosure proposes a hierarchical point sampling method. In the uniform sampling stage, voxel-based downsampling is used to reduce point redundancy, thereby reducing subsequent computational workload and improving training efficiency. In the target-guided sampling stage, only regions of interest with high confidence scores in the entire first reference point cloud are sampled. This preserves points within these regions of interest, prioritizing areas of significant motion. Therefore, when training the spatial perception module based on multiple first reference points obtained from the first and second sampling points, the world model's understanding of world dynamics and temporal dynamics is enhanced, leading to more accurate spatiotemporally consistent future predictions.

[0126] Figure 5 This is an exemplary flowchart illustrating how the model training method of this disclosure predicts at least one fifth BEV feature. Figure 5 As shown, in Figures 2-4 In some implementations of any of the illustrated embodiments, the future decoding module in operation 208 may perform the following operations:

[0127] 2082, for example, by using a feature update module, based on the third BEV feature and the motion information of at least one fifth time moment, a feature corresponding to at least one fifth time moment is predicted to obtain at least one intermediate BEV feature.

[0128] 2084, for example, through a motion flow prediction module, the future position of each grid at each fifth time step is determined based on the current position of each network in the third BEV feature and the motion information at each fifth time step.

[0129] 2086, for example, by using an interpolation processing module, interpolation processing is performed on the intermediate BEV features corresponding to each fifth time step based on the future position of each grid at each fifth time step, to obtain at least one fifth BEV feature.

[0130] For example, in one optional example, for each grid cell, based on its future position at a fifth time step, multiple neighboring grid cells can be selected from the intermediate BEV features corresponding to that fifth time step. The features of these selected neighboring grid cells are then aggregated using an inverse distance-weighted summation method to obtain the feature of that grid cell at that fifth time step. Optionally, a two-dimensional convolutional backbone network can be used to refine the features of each grid cell at each fifth time step. Based on the features of all grid cells in the third BEV feature at a fifth time step, a fifth BEV feature can be obtained.

[0131] Figure 6 This is another exemplary flowchart illustrating the process of predicting at least one fifth BEV feature in an embodiment of the model training method disclosed herein. Figure 6 As shown, in Figure 5 In some optional examples of the implementation shown, operation 2082 may include:

[0132] 302, for example, by using a multi-layer perceptron (MLP) to encode motion information at least at the fifth time step, motion features are obtained.

[0133] 304, for example, by using an addition module, the third BEV feature is enhanced based on motion features to obtain the first conditionally enhanced feature.

[0134] For example, in an optional example, motion features can be added to the third BEV feature by adding units, thereby obtaining a first conditional enhancement feature that includes motion information of the second mobile device at at least a fifth time.

[0135] 306, for example, by using a feature update module to perform self-attention processing based on the first conditional enhanced features, at least one intermediate BEV feature is obtained.

[0136] Optionally, see also Figure 6 ,exist Figure 5 In some optional examples of the implementation shown, operation 2084 may include:

[0137] 402, for example, by using the motion head unit in the motion flow prediction module, motion flow prediction is performed based on conditional enhancement features to obtain the flow field.

[0138] 404, for example, by using the summing unit in the motion flow prediction module, the current position and flow field of each network are used to determine the position of each grid in the third coordinate system corresponding to at least one fifth time moment, thus obtaining the position of each grid pair.

[0139] It should be at least one reference position at at least the fifth moment.

[0140] The third coordinate system is the coordinate system corresponding to the second mobile device at the third moment.

[0141] For example, in an optional example, the current position of each grid and the flow field can be added together by an additive unit to obtain the position of each grid in the third coordinate system corresponding to at least one fifth time moment (i.e., at least one reference position mentioned above).

[0142] 406, for example, by using the transformation unit in the motion flow prediction module, based on the transformation matrix between the third coordinate system and each fifth coordinate system, at least one reference position of each grid is transformed into a position in each fifth coordinate system, so as to obtain at least one future position of each grid at at least a fifth time.

[0143] Among them, each fifth coordinate system is the coordinate system corresponding to the second mobile device at each fifth moment.

[0144] For example, in an optional example, a reference position g of a grid can be set as follows: f Transform the grid into its position in each fifth coordinate system to obtain at least one future position g′ of each grid at at least at a fifth time. f :

[0145]

[0146] In formula (6), This represents the transformation matrix from the third coordinate system to the fifth coordinate system corresponding to the fifth time t+k.

[0147] A schematic diagram of the structure of a world model provided in another exemplary embodiment. For example... Figure 7 As shown, in Figure 1 In some implementations of the illustrated embodiments, the spatial perception module 102 may include: a convolutional neural network 1022, a two-dimensional to three-dimensional conversion network 1024, and a temporal cross-attention module 1026. Figure 1 In other implementations of the illustrated embodiments, the future decoding module 104 may include a feature update module 1042, a motion flow prediction module 1044, and an interpolation processing module 1046. It may also optionally include a multilayer perceptron 1048 and an addition module 1050. In some optional examples, the motion flow prediction module 1044 may include a motion head unit, an addition unit, and a transformation unit, and the information processing functions of each module and unit can be combined with those described above. Figures 5-6 The illustrated embodiments are described below and will not be repeated here.

[0148] Figure 8 This is a flowchart illustrating a model training method provided in another exemplary embodiment of this disclosure. For example... Figure 8 As shown, in Figures 2-6Based on any of the illustrated embodiments, this embodiment may further include, prior to operation 208:

[0149] 502. The trained spatial perception module encodes the third BEV feature based on the multi-view image at the third time step and the fourth BEV feature.

[0150] Among them, the fourth BEV feature is the BEV feature corresponding to the time before the third time step, which is encoded by the trained spatial perception module.

[0151] The specific implementation of operation 502 can be found in the implementation of operation 202, and will not be repeated here.

[0152] Optionally, see also Figure 8 In yet another exemplary embodiment of the model training method, after operation 502, it may further include:

[0153] 504, Obtain at least one reference BEV feature.

[0154] Each reference BEV feature is encoded by the trained spatial perception module based on the multi-view images at each fifth time step and the corresponding sixth BEV feature. Each multi-view image at the fifth time step corresponds to a sixth BEV feature, which is the BEV feature encoded by the trained spatial perception module at the time step preceding the fifth time step. The specific implementation of the spatial perception module encoding the corresponding reference BEV feature based on the multi-view images at each fifth time step and the corresponding sixth BEV feature can be found in the implementation of operation 202, and will not be elaborated here.

[0155] Accordingly, the aforementioned second training sample in operation 212 further includes at least one fifth BEV feature and at least one reference BEV feature, and operation 212 may include:

[0156] 2122, for each of the at least one second training samples in the second training samples, obtain the third difference (which can be called reconstruction loss) between at least one second 3D reconstruction result and at least one second reference point cloud in the second training samples, and obtain the fourth difference (which can be called latent space alignment loss) between at least one fifth BEV feature and at least one reference BEV feature.

[0157] The specific implementation of obtaining the third difference between at least one second 3D reconstruction result and at least one second reference point cloud in the second training sample can refer to the implementation of obtaining the first difference between the first 3D reconstruction result and the first reference point cloud in the first training sample in operation 2062 above, and will not be repeated here.

[0158] For example, in an alternative example, suppose at least one fifth time step includes F fifth time steps, where F is an integer greater than 0, and the fourth difference between each fifth BEV feature (i.e., the k-th fifth BEV feature, k = 1, 2, ..., F) and the corresponding reference BEV feature. It can be represented as:

[0159]

[0160] In formula (7), B t+k This represents the k-th reference BEV feature. This represents the k-th fifth BEV feature.

[0161] 2124. Based on the third and fourth differences corresponding to the second training sample, obtain the comprehensive difference corresponding to the second training sample.

[0162] For example, in an optional example, the comprehensive difference corresponding to the second training sample can be obtained in the following way.

[0163]

[0164] In formula (8), F represents the number of times included in at least one fifth time (i.e., the number of future times relative to the third time). and ω represents the third and fourth differences corresponding to the k-th fifth BEV feature, respectively. l and ω p ω represents the balance coefficient between the third and fourth differences, respectively. l and ω p The values ​​of ω are all greater than 0, and ω l and ω p The specific value can be set according to actual needs, and this embodiment does not limit it.

[0165] 2126. Based on the comprehensive differences corresponding to at least one second training sample, the future decoding module is iteratively trained until the second preset training completion condition is met, and the trained future decoding module is obtained.

[0166] Optionally, in some implementations of the embodiments of this disclosure, the second preset training completion condition may include at least one of the following: the number of iterations of training reaches a corresponding preset number (e.g., 800 times), the comprehensive difference corresponding to at least one second training sample is less than a second preset threshold, etc., and the embodiments of this disclosure do not limit this.

[0167] Based on this embodiment, the detailed and accurate geometric information of the second reference point cloud can be used as supervision, and the reference BEV features encoded by the trained spatial perception module can be used to optimize the future decoding module and improve the accuracy of the future decoding module's prediction results for the future.

[0168] Figure 9 This is a flowchart illustrating an exemplary embodiment of the autonomous driving method provided in this disclosure. Figure 9 As shown, the model training method in this embodiment includes:

[0169] 602. Using a world model, based on the multi-view image at the current moment and the previous BEV feature corresponding to the previous moment, at least one future BEV feature corresponding to at least one future moment is predicted.

[0170] The multi-view images include images captured by multiple visual sensors on the mobile device facing different directions, targeting the external environment of the mobile device. The "previous moment" refers to a moment preceding the current moment; specifically, it can be an adjacent moment preceding the current moment, or an adjacent moment at a preset interval (e.g., 2 moments) preceding the current moment. This disclosure does not impose any limitations on this. The aforementioned "at least one future moment" refers to at least one different moment after the current moment. The "at least one future moment" can be at least one consecutive moment following the current moment, or it can be at least one moment selected after the current moment at preset intervals (e.g., 2 moments). This disclosure does not impose any limitations on this.

[0171] The time in any embodiment of this disclosure is the time when the above-mentioned multiple visual sensors perform each image acquisition at a preset time interval. The preset time interval can be determined according to actual needs, such as 0.001 seconds (S), 0.03 seconds, etc., and this disclosure does not limit it.

[0172] The world model in this embodiment includes a spatial perception module and a future decoding module, and the world model can be trained by the model training method of any embodiment of this disclosure.

[0173] 604. Using an autonomous driving model, perform task processing corresponding to the autonomous driving model based on at least one future BEV feature to obtain at least one task processing result corresponding to at least one future moment, so as to perform autonomous driving control of the mobile device (e.g., vehicle) based on the at least one task processing result, such as maintaining the current motion state of the mobile device, adjusting the motion speed and direction of the mobile device, etc.

[0174] Optionally, in some implementations of this disclosure, different types of downstream tasks can be processed by an autonomous driving model trained based on training samples corresponding to that type of downstream task. These different types of downstream tasks may include, but are not limited to, tasks in the perception, decision-making, and planning stages such as 3D object detection, occupancy prediction, and trajectory planning. The corresponding autonomous driving model may be, for example, a detection model, an occupancy prediction model, or an end-to-end model. The corresponding task processing results may be, for example, 3D detection boxes, occupancy prediction results, planned trajectories, etc. This disclosure does not limit the specific downstream tasks and autonomous driving models used in the application.

[0175] The autonomous driving method based on this embodiment, by employing a world model trained using the model training method of any embodiment of this disclosure, can learn and understand the operating rules of the real physical world from multi-view images, accurately achieve spatiotemporally consistent future prediction, accurately understand and predict future scene states, such as the movement trajectories of other vehicles and pedestrians, changes in the road environment, etc., and obtain general future BEV features. This provides accurate information support for subsequent downstream tasks such as perception, planning, and control decisions. When the autonomous driving model of downstream tasks performs task processing based on these future BEV features, it can effectively improve the accuracy of task processing results and adjust the behavior of the mobile device at the current moment, thereby improving the safety of mobile device driving. Since the world model learns world knowledge that is universally applicable to various downstream tasks and is not specific to any particular task, it has the ability to generalize when applied to various tasks and can flexibly support different types of downstream applications, such as scene understanding and trajectory planning.

[0176] Figure 10 This is an exemplary flowchart illustrating how at least one future BEV feature is predicted using a world model in an embodiment of the autonomous driving method disclosed herein. Figure 10 As shown, in Figure 9 In some implementations of the illustrated embodiments, operation 602 may include:

[0177] 6022, through the spatial perception module, extract the features of each image in the multi-view image to obtain multiple image features; transform each second image feature from two-dimensional space to three-dimensional space to obtain multiple three-dimensional features; perform temporal cross-attention processing on the multiple three-dimensional features and the previous BEV feature to obtain the current BEV feature corresponding to the current time.

[0178] For a detailed implementation of operation 6022, please refer to [link / reference]. Figure 3 The implementation method of the illustrated embodiment will not be described in detail here.

[0179] 6024. Through the future decoding module, at least one future BEV feature is predicted based on the current BEV features and the motion information of the mobile device at at least one future moment.

[0180] Based on this embodiment, the spatial perception module can learn the spatial perception representation and temporal dynamics of the physical world. While maintaining spatial structure information, it can learn more robust and universal world features. The future decoding module can use this spatial structure information to achieve spatiotemporally consistent future prediction, thereby improving the accuracy of future prediction results. Furthermore, by compressing the state of the physical world into the BEV implicit space and modeling in the BEV implicit space, where each grid in the BEV corresponds to a spatial region in the physical world, the natural spatial structure attributes of the physical world are preserved, and high-dimensional information compression is achieved, enabling spatiotemporally consistent future prediction.

[0181] Figure 11 This is an exemplary flowchart illustrating how at least one future BEV feature is predicted by a future decoding module in an embodiment of the autonomous driving method disclosed herein. Figure 11 As shown, in Figure 10 In some implementations of the illustrated embodiments, the decoding module in operation 6024 may perform the following operations:

[0182] 702. Based on the current BEV features and motion information at least one future moment, predict the features corresponding to at least one future moment to obtain at least one intermediate BEV feature.

[0183] 704. Based on the current position of each network in the current BEV feature and the motion information of each future time, determine the future position of each grid at each future time.

[0184] 706. Based on the future position of each grid at each future time, interpolation is performed on the intermediate BEV features corresponding to each future time to obtain at least one future BEV feature.

[0185] For a detailed implementation of operations 702-706 in this approach, please refer to [link / reference]. Figure 5 The implementation method of the illustrated embodiment will not be described in detail here.

[0186] Based on this embodiment, it is possible to accurately predict at least one future BEV feature based on the current BEV features and the motion information of the mobile device at at least one future moment.

[0187] Optionally, in Figure 11 In some alternative examples of the implementation shown, operation 702 may include:

[0188] For example, motion features can be obtained by encoding motion information at least one future moment using a multilayer perceptron.

[0189] For example, by using an addition module, the current BEV features can be enhanced based on motion features to obtain conditionally enhanced features;

[0190] For example, by using a feature update module, self-attention processing is performed based on conditionally enhanced features to obtain at least one intermediate BEV feature.

[0191] For a detailed implementation of operation 702 in this optional example, please refer to [link / reference]. Figure 6 The implementation of operations 302-306 in the illustrated embodiment will not be described in detail here.

[0192] Based on this embodiment, motion information at least one future moment can be added to the current BEV feature to update the current BEV feature at at least one future moment.

[0193] Optionally, in Figure 11 In some alternative examples of the implementation shown, operation 704 may include:

[0194] For example, the motion head unit in the motion flow prediction module can be used to predict the motion flow based on conditional enhancement features to obtain the flow field;

[0195] For example, by using the summation unit in the motion flow prediction module, the current position and flow field of each network are used to determine the position of each grid in the current coordinate system corresponding to at least one future time, and obtain at least one reference position of each grid; where the current coordinate system is the coordinate system corresponding to the mobile device at the current time, that is, the coordinate system established with the center point of the mobile device at the current time. When the mobile device is a vehicle, it is the vehicle coordinate system at the current time.

[0196] For example, through the transformation unit in the motion flow prediction module, based on the transformation matrix between the current coordinate system and each future coordinate system, at least one reference position of each grid is transformed into a position in each future coordinate system, so as to obtain at least one future position of each grid at at least one future time; wherein, each future coordinate system is the coordinate system corresponding to the mobile device at each future time.

[0197] For a specific implementation of operation 704 in this optional example, please refer to [link / reference]. Figure 6 The implementation of operations 402-406 in the illustrated embodiment will not be described in detail here.

[0198] Figure 12 This is a flowchart illustrating an autonomous driving method provided in another exemplary embodiment of this disclosure. Figure 12 As shown, in Figures 10-11 Based on any of the illustrated embodiments, this embodiment may further include, prior to operation 6024:

[0199] 6023, using an autonomous driving model, predicts the motion information of a mobile device at at least one future moment based on the current motion state and historical trajectory of the mobile device.

[0200] In this embodiment of the disclosure, the current motion state of the mobile device may include, but is not limited to, at least one of the following: the current motion speed, acceleration, steering wheel angle, braking information, and other relevant information that affects the possible motion state and trajectory of the mobile device. The historical trajectory is the motion trajectory of the mobile device within a preset time period (e.g., 30 seconds, 100 seconds, etc.) before the current moment. This embodiment of the disclosure does not limit the length of the preset time period.

[0201] Based on this embodiment, an autonomous driving model can be used to determine the current motion state of the mobile device.

[0202] Figure 13 This is a flowchart illustrating an autonomous driving method provided in yet another exemplary embodiment of this disclosure. Figure 13 As shown above, in the above Figures 9-12 Based on any of the illustrated embodiments, after operation 602, this embodiment may further include:

[0203] 603. Through the Future Spatial Attention (FSA) network, feature fusion processing is performed on at least one future BEV feature to obtain at least one attention feature.

[0204] The future spatial attention network and the autonomous driving model are pre-trained together for the corresponding task. The trained future spatial attention network can perform feature fusion processing adapted to the corresponding task for the input features, dynamically select and aggregate the prediction features related to the corresponding task, eliminate redundancy, and obtain a richer and more comprehensive representation of the corresponding task, thus obtaining the enhanced features of the corresponding task. When the autonomous driving model performs corresponding task processing based on the enhanced features of the corresponding task, it can improve the accuracy and precision of the task processing results.

[0205] In some optional examples, future spatial attention networks can be implemented using a cross-attention mechanism, which can effectively align and focus on contexts from different sources, thereby better capturing the correlation between information.

[0206] Accordingly, in this embodiment, operation 604 may include:

[0207] 6042. Using an autonomous driving model, perform task processing corresponding to the autonomous driving model based on at least one attention feature to obtain at least one task processing result.

[0208] Based on this embodiment, a unified future spatial attention paradigm is proposed. The future spatial attention network is used as a unified interface for the interaction between the world model and various downstream tasks. It adaptively captures changes in the future environment and integrates the general future BEV features predicted by the world model into various downstream tasks. By performing feature fusion processing on the future BEV features to adapt to the corresponding tasks, the prediction features related to the corresponding tasks are dynamically selected and aggregated and output to the autonomous driving models corresponding to various downstream tasks. This can effectively improve the accuracy and precision of task processing results.

[0209] Exemplary devices and systems

[0210] Figure 14 This is a schematic diagram of the structure of a model training apparatus provided in an exemplary embodiment of this disclosure. The model training apparatus of this embodiment can be used to implement the model training methods described above in this disclosure. The model training apparatus is applied to a world model, which includes a spatial perception module and a future decoding module. Figure 14 As shown, the device model training in this embodiment includes: a first acquisition module 802, a 3D reconstruction module 804, a first training module 806, a second acquisition module 808, a 3D reconstruction module 810, and a second training module 812. Wherein:

[0211] The first acquisition module 802 is configured to: input the multi-view image and the second bird's-eye view BEV feature at the first moment into the spatial perception module, and acquire the first BEV feature encoded by the spatial perception module. The multi-view image includes images of the first scene acquired using multiple visual sensors on the first mobile device facing different directions, and the second BEV feature is the BEV feature encoded by the spatial perception module corresponding to the moment preceding the first moment.

[0212] The 3D reconstruction module 804 is configured to perform 3D reconstruction based on the first BEV feature to obtain the first 3D reconstruction result.

[0213] The first training module 806 is configured to iteratively train the spatial perception module based on at least one first training sample. One of the at least one first training samples includes a first 3D reconstruction result and a first reference point cloud, wherein the first reference point cloud is a 3D point cloud generated by scanning a first scene using a LiDAR sensor on a first mobile device at a first moment.

[0214] The second acquisition module 808 is configured to: input the third BEV feature and the motion information of the second mobile device at at least a fifth moment into the future decoding module, and acquire at least one fifth BEV feature corresponding to the at least one fifth moment predicted by the future decoding module. The third BEV feature is the BEV feature corresponding to the third moment encoded by the trained spatial perception module, and the at least one fifth moment is at least one different moment after the third moment.

[0215] The 3D reconstruction module 810 is also configured to perform 3D reconstruction based on at least one fifth BEV feature to obtain at least one second 3D reconstruction result.

[0216] The second training module 812 is configured to iteratively train the future decoding module based on at least one second training sample. One of the at least one second training sample includes at least one second 3D reconstruction result and at least one second reference point cloud, where each second reference point cloud is a 3D point cloud generated by scanning the second scene using a lidar sensor on a second mobile device at each fifth time step.

[0217] Optionally, in some implementations of the embodiments of this disclosure, the first training module 806 is further configured to: obtain a first difference between a first three-dimensional reconstruction result and a first reference point cloud in each of the at least one first training samples, and perform iterative training on the spatial perception module based on at least one difference corresponding to at least one first training sample.

[0218] Optionally, in some other implementations of the present disclosure, when the three-dimensional reconstruction module 804 performs three-dimensional reconstruction based on the first BEV feature and obtains the first three-dimensional reconstruction result, it is further configured to: transform the first BEV feature into a three-dimensional density field, wherein the first three-dimensional reconstruction result includes the three-dimensional density field. Accordingly, when the first training module 806 acquires the first difference between the first 3D reconstruction result and the first reference point cloud in the first training sample, it is further configured to: acquire at least one sampling point corresponding to each of the multiple first reference points, wherein the first reference point is a point in the corresponding first reference point cloud, and the at least one sampling point is obtained by sampling a 3D ray from a lidar sensor obtained by parameterizing the corresponding reference point; perform 3D interpolation on the 3D density field and the sampling points corresponding to the multiple first reference points to obtain a first occupancy value; based on the first occupancy value, use a classification-based discrete preset method to determine the depth values ​​of the multiple first reference points respectively to obtain multiple first predicted depth values; based on the first occupancy value, use an integral-based continuous rendering method to determine the depth values ​​of the multiple first reference points respectively to obtain multiple second predicted depth values; and determine the first loss value corresponding to the first training sample based on the first difference between the multiple reference depth values ​​corresponding to the multiple first reference points in the first reference point cloud and the multiple first predicted depth values, the second difference between the multiple reference depth values ​​and the multiple second predicted depth values, and the balance coefficient between the first difference and the second difference, wherein the first difference includes the first loss value.

[0219] Figure 15 This is a schematic diagram of the structure of a model training apparatus provided in another exemplary embodiment of this disclosure. For example... Figure 15 As shown, in Figure 14 Based on the illustrated embodiment, this embodiment may further include: a sampling module 814, configured to: perform sparse point sampling on the first reference point cloud to obtain a first sampling point; perform dense point sampling on the region of interest in the first reference point cloud to obtain a second sampling point; and obtain a plurality of first reference points based on the first sampling point and the second sampling point.

[0220] Optionally, in some other implementations of the embodiments of this disclosure, the first acquisition module 802 is further configured to: input the multi-view image at the third time and the fourth BEV feature into the trained spatial perception module, and acquire the third BEV feature encoded by the spatial perception module, wherein the fourth BEV feature is the BEV feature corresponding to the time before the third time, encoded by the trained spatial perception module.

[0221] Optionally, see also Figure 15In another exemplary embodiment of this disclosure, the model training apparatus may further include: a third acquisition module 816, configured to: acquire at least one reference BEV feature, wherein each reference BEV feature is encoded by a trained spatial perception module based on the multi-view image at each fifth time point and the corresponding sixth BEV feature, and the corresponding sixth BEV feature is the BEV feature corresponding to a time point prior to each fifth time point, encoded by the trained spatial perception module. Accordingly, one of the at least one second training samples further includes the at least one fifth BEV feature and the at least one reference BEV feature. The second training module 812 is further configured to: acquire, for each of the at least one second training samples, a third difference between at least one second 3D reconstruction result and at least one second reference point cloud in the second training sample, and a fourth difference between at least one fifth BEV feature and at least one reference BEV feature; acquire a comprehensive difference corresponding to the second training sample based on the third and fourth differences corresponding to the second training sample; and iteratively train the future decoding module based on the comprehensive difference corresponding to at least one second training sample.

[0222] Figure 16 This is a schematic diagram of the structure of an autonomous driving system provided in an exemplary embodiment of this disclosure. Figure 16 As shown, the autonomous driving system of this embodiment includes a world model 10 and an autonomous driving model 20. The world model 10 includes a spatial perception module 102 and a future decoding module 104, and the world model 10 is trained using the model training method of any embodiment of this disclosure.

[0223] in:

[0224] World Model 10 is configured to predict at least one future BEV feature corresponding to at least one future moment based on the multi-view image of the current moment and the previous BEV feature corresponding to the previous moment; wherein, the multi-view image includes images of the external environment of the mobile device collected by multiple visual sensors on different orientations of the mobile device, and the previous moment is the moment before the current moment.

[0225] The autonomous driving model 20 is configured to perform task processing corresponding to the autonomous driving model 20 based on at least one of the aforementioned future BEV features, and obtain at least one task processing result, so as to perform autonomous driving control of the mobile device based on the at least one task processing result.

[0226] Figure 17 This is a schematic diagram of the structure of an autonomous driving system provided in another exemplary embodiment of this disclosure. (See diagram below.) Figure 17 As shown, in Figure 16Based on the illustrated embodiment, the autonomous driving system of this embodiment may further include: a future spatial attention network 30, configured to: perform feature fusion processing on at least one future BEV feature predicted by the world model 10 to obtain at least one attention feature; wherein, the future spatial attention network and the autonomous driving model are jointly trained in advance for the corresponding task. Accordingly, in this embodiment, the autonomous driving model 20 is further configured to: perform task processing corresponding to the autonomous driving model based on the at least one attention feature to obtain at least one task processing result.

[0227] The exemplary embodiments of the model training method and apparatus, autonomous driving method and apparatus, world model, etc. disclosed herein correspond to each other, and can be referenced and cited in their respective implementation methods. The beneficial technical effects of each exemplary embodiment can also be referenced in turn, and will not be described in detail here.

[0228] Exemplary device

[0229] Figure 18 This is a schematic diagram of the structure of a mobile device provided in an exemplary embodiment of this disclosure. Figure 18 As shown, the mobile device in this embodiment is equipped with multiple visual sensors 1 facing different directions and an autonomous driving system 2 as described in any embodiment of this disclosure. The multiple visual sensors 1 are configured to: acquire images at preset time intervals and sequentially input the acquired multi-view images into the world model 10 in the autonomous driving system 2, so as to perform autonomous driving control of the mobile device based on at least one task processing result output by the autonomous driving model 20 in the autonomous driving system 2.

[0230] The specific implementation of the autonomous driving system 2, world model 10, and autonomous driving model 20, as well as their information processing, can be found in the descriptions of the corresponding embodiments above, and will not be repeated here.

[0231] In addition, according to actual needs, at least one lidar sensor with different orientations can be deployed on the mobile device in this embodiment of the present disclosure so as to scan the external scene of the mobile device with a corresponding scanning angle. The distance between the device and objects in the external scene is measured by emitting laser pulses and receiving reflected signals. The time required for the laser pulse to return from emission is calculated and converted into distance. Combined with other relevant information (such as scanning angle, position and attitude data of lidar sensor) to generate a three-dimensional point cloud. Each point in the three-dimensional point cloud is discretely distributed in three-dimensional space and carries three-dimensional coordinates (x, y, z) representing its physical position.

[0232] Optionally, in some implementations of the embodiments of this disclosure, the mobile device may be, for example, a manned device such as a vehicle, ship, or flying equipment (e.g., an airplane, an aircraft), or any autonomous driving device such as a robot. The embodiments of this disclosure do not limit the specific type of mobile device.

[0233] The mobile device disclosed herein corresponds to the exemplary embodiments described above, such as the model training method and apparatus, the autonomous driving method and apparatus, and the world model. They can be referenced and cited in their respective implementations, and their respective beneficial technical effects can also be referenced in each other, which will not be elaborated further.

[0234] In addition, this disclosure also provides an electronic device, which includes a memory and a processor. The memory stores processor-executable instructions; the processor reads the executable instructions from the memory and executes the instructions to implement the model training method or autonomous driving method described in any embodiment of this disclosure.

[0235] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure. For example... Figure 19 As shown, the electronic device of this embodiment includes at least one processor 902 and a memory 904. The processor 902 may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. The memory 904 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 902 may execute one or more computer program instructions to implement the model training method or autonomous driving method of the various embodiments of this disclosure above, and / or other desired functions.

[0236] In one example, the electronic device may further include an input device 906 and an output device 908, these components being interconnected via a bus system and / or other forms of connection mechanism (not shown). The input device 906 may also include, for example, a keyboard, a mouse, etc. The output device 908 may output various information to the outside, and may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0237] Of course, for the sake of simplicity, Figure 19Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0238] Exemplary computer program products and computer-readable storage media

[0239] Additionally, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform steps in the model training method or autonomous driving method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0240] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0241] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform steps in the model training method or autonomous driving method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0242] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0243] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0244] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A model training method applied to a world model, the world model comprising a spatial awareness module and a future decoding module, the method comprising: The spatial perception module encodes a first BEV feature based on the multi-view images at the first moment and the second bird's-eye view BEV feature. The multi-view images include images of the first scene collected by multiple visual sensors on the first mobile device in different orientations. The second bird's-eye view BEV feature is the BEV feature encoded by the spatial perception module that corresponds to the moment before the first moment. Based on the first BEV feature, a three-dimensional reconstruction is performed to obtain the first three-dimensional reconstruction result; The spatial perception module is iteratively trained based on at least one first training sample; wherein, one of the at least one first training samples includes the first three-dimensional reconstruction result and a first reference point cloud, and the first reference point cloud is a three-dimensional point cloud generated by scanning the first scene using the lidar sensor on the first mobile device at the first time. Based on the third BEV feature and the motion information of the second mobile device at at least a fifth moment, the future decoding module predicts at least one fifth BEV feature corresponding to the at least one fifth moment; wherein, the third BEV feature is the BEV feature corresponding to the third moment encoded by the trained spatial perception module, and the at least one fifth moment is at least one different moment after the third moment. Three-dimensional reconstruction is performed based on the at least one fifth BEV feature to obtain at least one second three-dimensional reconstruction result; The future decoding module is iteratively trained based on at least one second training sample; wherein, one of the at least one second training samples includes at least one second 3D reconstruction result and at least one second reference point cloud, and each second reference point cloud is a 3D point cloud generated by scanning the second scene using the lidar sensor on the second mobile device at each of the fifth time moments.

2. The method according to claim 1, wherein, Iterative training of the spatial perception module based on at least one first training sample includes: For each of the first training samples in at least one first training sample, obtain the first difference between the first 3D reconstruction result and the first reference point cloud in the first training sample; The spatial perception module is iteratively trained based on at least one difference corresponding to at least one first training sample.

3. The method according to claim 2, wherein, Based on the first BEV feature, a three-dimensional reconstruction is performed to obtain a first three-dimensional reconstruction result, including: The first BEV feature is transformed into a three-dimensional density field, and the first three-dimensional reconstruction result includes the three-dimensional density field. Obtaining the first difference between the first 3D reconstruction result and the first reference point cloud in the first training sample includes: At least one sampling point corresponding to each of a plurality of first reference points is obtained; wherein, the first reference point is a point in the corresponding first reference point cloud, and the at least one sampling point is obtained by sampling a three-dimensional ray from the lidar sensor obtained by parameterizing the corresponding reference point; Three-dimensional interpolation is performed on the sampling points corresponding to the three-dimensional density field and the plurality of first reference points to obtain the first occupancy value; Based on the first occupancy value, a classification-based discrete preset method is used to determine the depth values ​​of the plurality of first reference points respectively, thereby obtaining a plurality of first predicted depth values; Based on the first occupancy value, an integral-based continuous rendering method is used to determine the depth values ​​of the plurality of first reference points respectively, thereby obtaining a plurality of second predicted depth values; Based on the first difference between the multiple reference depth values ​​corresponding to the multiple first reference points in the first reference point cloud and the multiple first predicted depth values, the second difference between the multiple reference depth values ​​and the multiple second predicted depth values, and the balance coefficient between the first difference and the second difference, a first loss value corresponding to the first training sample is determined, wherein the first difference includes the first loss value.

4. The method according to claim 3, wherein, Before obtaining at least one sampling point corresponding to each of the multiple first reference points, the method further includes: Sparse point sampling is performed on the first reference point cloud to obtain the first sampling point; Dense point sampling is performed on the region of interest in the first reference point cloud to obtain the second sampling point; Based on the first sampling point and the second sampling point, the plurality of first reference points are obtained.

5. The method according to any one of claims 1-4, wherein, Before predicting at least one fifth BEV feature corresponding to the at least one fifth time moment based on the third BEV feature and the motion information of the second mobile device at at least one fifth time moment, the method further includes: The trained spatial perception module encodes the third BEV feature based on the multi-view image at the third time point and the fourth BEV feature; wherein, the fourth BEV feature is the BEV feature corresponding to the time point preceding the third time point, which is encoded by the trained spatial perception module.

6. The method according to any one of claims 1-5, wherein, After encoding the third BEV feature based on the multi-view image at the third time point and the fourth BEV feature, the method further includes: At least one reference BEV feature is obtained; wherein each reference BEV feature is encoded by the trained spatial perception module based on the multi-view image of each fifth time point and the corresponding sixth BEV feature, and the corresponding sixth BEV feature is the BEV feature corresponding to the time point before each fifth time point, which is encoded by the trained spatial perception module. The second training sample also includes the at least one fifth BEV feature and the at least one reference BEV feature; Training the future decoding module based on at least one second training sample includes: For each of the at least one second training samples, a third difference is obtained between the at least one second 3D reconstruction result and the at least one second reference point cloud in the second training sample, and a fourth difference is obtained between the at least one fifth BEV feature and the at least one reference BEV feature. Based on the third and fourth differences corresponding to the second training sample, obtain the comprehensive difference corresponding to the second training sample; The future decoding module is iteratively trained based on the comprehensive differences corresponding to the at least one second training sample.

7. An autonomous driving method, comprising: Using a world model, based on multi-view images of the current moment and previous BEV features corresponding to the previous moment, at least one future BEV feature corresponding to at least one future moment is predicted; wherein, the multi-view images include images collected by multiple visual sensors on the mobile device facing different directions regarding the external environment of the mobile device, and the previous moment is a moment before the current moment; the world model includes a spatial perception module and a future decoding module, and the world model is trained by any of the model training methods described in claims 1-6; Using the autonomous driving model, tasks corresponding to the autonomous driving model are processed based on at least one future BEV feature to obtain at least one task processing result, so as to perform autonomous driving control on the mobile device based on the at least one task processing result.

8. The method according to claim 7, wherein, Using a world model, based on the multi-view images of the current moment and the previous BEV features corresponding to the previous moment, at least one future BEV feature corresponding to at least one future moment is predicted, including: The spatial perception module extracts features from each image in the multi-view image to obtain multiple image features; each second image feature is transformed from two-dimensional space to three-dimensional space to obtain multiple three-dimensional features; the multiple three-dimensional features and the previous BEV feature are subjected to temporal cross-attention processing to obtain the current BEV feature corresponding to the current time. The future decoding module predicts at least one future BEV feature based on the current BEV feature and the motion information of the mobile device at at least one future time.

9. The method according to claim 8, wherein, Based on the current BEV characteristics and the motion information of the mobile device at at least one future time, the at least one future BEV characteristic is predicted, including: Based on the current BEV features and the motion information of the at least one future moment, predict the features corresponding to the at least one future moment to obtain at least one intermediate BEV feature; Based on the current position of each network in the current BEV feature and the motion information at each future time, the future position of each grid at each future time is determined. Based on the future position of each grid at each future time, interpolation processing is performed on the intermediate BEV features corresponding to each future time to obtain the at least one future BEV feature.

10. The method according to claim 9, wherein, Based on the current BEV features and the motion information of the at least one future time, predict the features corresponding to the at least one future time to obtain at least one intermediate BEV feature, including: Encode the motion information at at least one future moment to obtain motion features; The current BEV feature is enhanced based on the motion feature to obtain conditional enhancement feature; Self-attention processing is performed based on the conditional enhancement features to obtain at least one intermediate BEV feature.

11. The method according to claim 10, wherein, Based on the current position of each network in the current BEV features and the motion information at each future time, the future position of each grid at each future time is determined, including: Based on the aforementioned conditional enhancement features, motion flow prediction is performed to obtain the flow field; Based on the current position of each network and the flow field, determine the position of each grid in the current coordinate system corresponding to at least one future time, and obtain at least one reference position for each grid; wherein, the current coordinate system is the coordinate system corresponding to the mobile device at the current time; Based on the transformation matrix between the current coordinate system and each future coordinate system, the at least one reference position of each grid is transformed into its position in each future coordinate system, thereby obtaining at least one future position of each grid at the at least one future time; wherein, each future coordinate system is the coordinate system corresponding to the mobile device at each future time.

12. The method according to any one of claims 8-11, further comprising: The autonomous driving model predicts the motion information of the mobile device at at least one future moment based on the current motion state and historical trajectory of the mobile device; wherein the historical trajectory is the motion trajectory of the mobile device within a preset time period before the current moment.

13. The method according to any one of claims 7-12, wherein, After predicting at least one future BEV feature corresponding to at least one future time point, the process also includes: The at least one future BEV feature is fused using a future spatial attention network to obtain at least one attention feature; wherein the future spatial attention network and the autonomous driving model are jointly trained in advance for the corresponding task. Using the autonomous driving model, task processing corresponding to the autonomous driving model is performed based on at least one future BEV feature to obtain at least one task processing result, including: Using the autonomous driving model, task processing corresponding to the autonomous driving model is performed based on the at least one attention feature to obtain the at least one task processing result.

14. A model training device applied to a world model, the world model including a spatial perception module and a future decoding module, the device comprising: The first acquisition module is configured to: input the multi-view image and the second bird's-eye view BEV feature at the first moment into the spatial perception module, and acquire the first BEV feature encoded by the spatial perception module; wherein, the multi-view image includes images of the first scene collected by multiple visual sensors in different orientations on the first mobile device, and the second bird's-eye view BEV feature is the BEV feature encoded by the spatial perception module corresponding to the moment before the first moment. The 3D reconstruction module is configured to perform 3D reconstruction based on the first BEV feature to obtain a first 3D reconstruction result. The first training module is configured to iteratively train the spatial perception module based on at least one first training sample; wherein, one of the at least one first training samples includes the first three-dimensional reconstruction result and a first reference point cloud, and the first reference point cloud is a three-dimensional point cloud generated by scanning the first scene using a lidar sensor on the first mobile device at the first time. The second acquisition module is configured to: input the third BEV feature and the motion information of the second mobile device at at least a fifth moment into the future decoding module, and acquire at least one fifth BEV feature corresponding to the at least one fifth moment predicted by the future decoding module; wherein, the third BEV feature is the BEV feature corresponding to the third moment encoded by the trained spatial perception module, and the at least one fifth moment is at least one different moment after the third moment. The three-dimensional reconstruction module is further configured to: perform three-dimensional reconstruction based on the at least one fifth BEV feature to obtain at least one second three-dimensional reconstruction result; The second training module is configured to iteratively train the future decoding module based on at least one second training sample; wherein, one of the at least one second training samples includes the at least one second 3D reconstruction result and at least one second reference point cloud, and each second reference point cloud is a 3D point cloud generated by scanning the second scene using a lidar sensor on the second mobile device at each of the fifth time moments.

15. A world model comprising a spatial perception module and a future decoding module, wherein the world model is trained by any one of the model training methods described in claims 1-6.

16. An autonomous driving system, comprising a world model and an autonomous driving model; the world model comprising a spatial perception module and a future decoding module, wherein the world model is trained by any one of the model training methods described in claims 1-6; The world model is configured to: predict at least one future BEV feature corresponding to at least one future time based on the multi-view image at the current moment and the previous BEV feature corresponding to the previous moment; wherein... The multi-view image includes images captured by multiple visual sensors on the mobile device facing different directions, targeting the external environment of the mobile device; the previous moment is the moment before the current moment. The autonomous driving model is configured to perform task processing corresponding to the autonomous driving model based on the at least one future BEV feature, and obtain at least one task processing result, so as to perform autonomous driving control on the mobile device based on the at least one task processing result.

17. The system of claim 16, further comprising: The future spatial attention network is configured to: perform feature fusion processing on the at least one future BEV feature to obtain at least one attention feature; wherein the future spatial attention network and the autonomous driving model are jointly trained in advance for the corresponding task; The autonomous driving model is further configured to: perform task processing corresponding to the autonomous driving model based on the at least one attention feature, and obtain the at least one task processing result.

18. A mobile device having a plurality of visual sensors facing different directions and an autonomous driving system as described in any one of claims 16-17 deployed thereon; The plurality of visual sensors are configured to: acquire images at preset time intervals and sequentially input the multi-view images acquired in each acquisition into the world model of the autonomous driving system, so as to perform autonomous driving control of the mobile device based on at least one task processing result output by the autonomous driving model in the autonomous driving system.

19. A computer-readable storage medium storing a computer program that, when executed, performs the method according to any one of claims 1-13.

20. An electronic device, the electronic device comprising a memory and a processor: The memory is used to store the processor-executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-13.

21. A computer program product comprising computer program instructions that, when executed by a processor, implement the method of any one of claims 1-13.

Images