Streaming three-dimensional semantic occupancy prediction method based on Gaussian world model

Abstract

The invention provides a streaming three-dimensional semantic occupancy prediction method based on a Gaussian world model, and the method comprises the steps: employing explicit three-dimensional Gaussian representation, and explicitly modeling three decomposition factors of scene evolution in a three-dimensional Gaussian space; through a unified evolution layer module, modeling evolution of historical Gaussian and perception of newly complemented Gaussian at the same time, interacting Gaussian representation and visual input by adopting a three-dimensional convolution operation and a deformable attention mechanism respectively, and adding time sequence characteristic attributes to model historical information of Gaussian representation; and finally, predicting scene evolution by using all optimized Gaussian representations, performing three-dimensional semantic occupancy prediction of the current scene, and constraining the model through cross entropy loss and lovasz loss of a three-dimensional semantic occupancy prediction task. Compared with a traditional time sequence fusion method, the method has the advantages that scene evolution is explicitly modeled in the three-dimensional Gaussian space, the static consistency of scene evolution and dynamic object movement can be modeled, and the efficiency and effectiveness of model three-dimensional occupancy prediction are improved.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision and machine learning, and particularly relates to a streaming three-dimensional semantic occupancy prediction method based on the Gaussian world model. Background Art

[0002] The vision-centered three-dimensional semantic occupancy prediction task has recently received extensive attention due to its key applications in autonomous driving. This task aims to predict the occupancy state and semantic label of each voxel in the three-dimensional environment based on visual inputs, enabling a comprehensive perception of the surrounding environment and providing fine-grained structural and semantic information.

[0003] Utilizing time-series inputs is important for three-dimensional semantic occupancy prediction as it provides sufficient historical context for understanding scene evolution. Most existing methods follow a traditional perception, transformation, and fusion process. Given a sequence of inputs, the perception module first independently obtains the scene representation of each frame, such as bird's-eye view (BEV) features or voxel features. Then the transformation module aligns the multi-frame features according to the ego-vehicle trajectory, and finally the fusion module fuses the aligned multi-frame representations to infer the three-dimensional semantic occupancy of the current environment.

[0004] However, these methods do not take into account the inherent continuity and simplicity of driving scene evolution. The driving scene representations in adjacent frames should be closely related because scene evolution usually only stems from the movement of the ego-vehicle and other dynamic objects. Directly fusing multi-frame representations ignores this strong prior information provided by three-dimensional scene evolution, namely the consistency of static objects and the movement of dynamic objects, thus limiting the performance of the model. In addition, this design increases the complexity and computational cost of temporal modeling, reducing the efficiency and effectiveness of model inference. Summary of the Invention

[0005] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0006] To this end, the first objective of the present application is to propose a streaming three-dimensional semantic occupancy prediction method based on the Gaussian world model.

[0007] The second objective of the present application is to propose a streaming three-dimensional semantic occupancy prediction device based on the Gaussian world model.

[0008] The third objective of the present application is to propose an electronic device.

[0009] The fourth objective of the present application is to propose a computer-readable storage medium.

[0010] The fifth objective of the present application is to propose a computer program product.

[0011] To achieve the above object, an embodiment of the first aspect of the present application provides a streaming three-dimensional semantic occupancy prediction method based on a Gaussian world model, including:

[0012] Explicitly model the evolution of the driving scene based on a three-dimensional Gaussian space, decompose the evolution of the driving scene into the ego-vehicle motion alignment of the static scene, the local motion of dynamic objects, and the scene completion of the newly observed area, and describe the driving scene using a sparse three-dimensional Gaussian representation;

[0013] Use the alignment module to perform a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the ego-vehicle trajectory, align the positions of the three-dimensional Gaussians, and divide the aligned three-dimensional Gaussians into a dynamic Gaussian set and a static Gaussian set according to the semantic probability;

[0014] For the dynamic Gaussian set, combine the motion layer module and the visual input of the current frame to model the local motion of the dynamic Gaussians and update the position information of the dynamic Gaussians; for the static Gaussian set, keep its position information unchanged;

[0015] For the newly observed area not covered in the current frame, generate a new three-dimensional Gaussian representation by random initialization, and complete its attributes through the perception layer module, and merge the dynamic Gaussians, static Gaussians, and the completed three-dimensional Gaussians into the complete three-dimensional Gaussian representation of the current frame;

[0016] Use a unified evolution layer module to iteratively adjust the complete three-dimensional Gaussian representation, gradually optimize the attributes of the three-dimensional Gaussians by stacking multiple evolution layers, and combine the fine-tuning layer to adjust the global attributes of the Gaussian representation;

[0017] Based on all optimized three-dimensional Gaussian representations, predict the three-dimensional semantic occupancy of the current scene through a three-dimensional semantic occupancy prediction task, and use the cross-entropy loss and Lovasz loss to constrain the Gaussian world model to output the three-dimensional semantic occupancy prediction result of the current scene.

[0018] Optionally, using a sparse three-dimensional Gaussian representation to describe the driving scene includes:

[0019] Use a set of sparse three-dimensional Gaussian representations g = {p, s, r, c, f} to describe the driving scene, where p, s, r, c, f represent the position, scale, rotation, semantic probability, and temporal feature of the three-dimensional Gaussian, respectively.

[0020] Optionally, using the alignment module to perform a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the ego-vehicle trajectory, adjust the positions of the three-dimensional Gaussians, and divide the adjusted three-dimensional Gaussians into a dynamic Gaussian set and a static Gaussian set includes:

[0021] Use the alignment module A lignGenerate an affine transformation matrix M based on the ego vehicle's trajectory ego , and perform global alignment on the position attribute p of the three-dimensional Gaussian representation g of the previous frame T-1 . After alignment, the three-dimensional Gaussian is represented as:

[0022]

[0023] where A ttr (g; p) represents the attribute p of the three-dimensional Gaussian g, and R ef (g; n; p) represents updating the attribute p of the Gaussian g with n;

[0024] According to the semantic probability c, divide the aligned three-dimensional Gaussian to form a dynamic Gaussian set {g D} and a static Gaussian set {g S}.

[0025] Optionally, for the dynamic Gaussian set, combine the motion layer module and the visual input of the current frame to model the local motion of the dynamic Gaussian and update the position information of the dynamic Gaussian; for the static Gaussian set, keep its position information unchanged, including:

[0026] Through the motion layer module M ove Based on the aligned three-dimensional Gaussian and the visual input x of the current frame T Learn the local motion of the dynamic Gaussian set {g D} and update its position information. The update process is achieved through the following formula:

[0027]

[0028] where is the encoder, which is used to perform interactive modeling between the three-dimensional Gaussian and the visual input; I(·) represents the indicator function;

[0029] For the static Gaussian set {g S}, keep its position attribute p unchanged.

[0030] Optionally, for the newly observed area not covered in the current frame, use random initialization to generate a new three-dimensional Gaussian representation, and complete its attributes through the perception layer module. Combine the dynamic Gaussian, static Gaussian, and the completed three-dimensional Gaussian into a complete three-dimensional Gaussian representation of the current frame, including:

[0031] Uniformly sample the same number of randomly initialized three-dimensional Gaussians in the newly observed area

[0032] Through the perception layer module P er Based on the visual input x of the current frameT Complete all attributes of the newly initialized three-dimensional Gaussian , and the completion process is achieved through the following formula:

[0033]

[0034] where E nc and M ove use the same model structure and parameters in the encoder;

[0035] Merge the dynamic Gaussian set {g D}, the static Gaussian set {g S} and the completed three-dimensional Gaussian set to form a complete three-dimensional Gaussian representation of the current frame

[0036] Optionally, use a unified evolution layer module to iteratively adjust the complete three-dimensional Gaussian representation, and gradually optimize the three-dimensional Gaussian attributes by stacking multiple evolution layers, and combine the fine-tuning layer to adjust the global attributes of the Gaussian representation, including:

[0037] Iteratively adjust the complete three-dimensional Gaussian representation g vol through the unified evolution layer module E T , where each evolution layer uses a unified structure to update the attributes {p, s, r, c, f} of the three-dimensional Gaussian representation, and the formula is:

[0038]

[0039] where represents the three-dimensional Gaussian representation in the current evolution layer;

[0040] Introduce n r fine-tuning layers to adjust the global attributes of the three-dimensional Gaussian representation, and the formula is:

[0041]

[0042] where R efine is the fine-tuning layer.

[0043] Optionally, the training process of the Gaussian world model includes:

[0044] Train using cross-entropy loss and lovasz loss, and pre-train the Gaussian world model on a single-frame prediction task;

[0045] Use a streaming training strategy to fine-tune the model, input the sequence images of the scene into the model in turn, and perform three-dimensional semantic occupancy prediction by combining the current frame image and the three-dimensional Gaussian representation predicted by the previous frame;

[0046] Among them, during the streaming training process, the current frame image and the three-dimensional Gaussian representation predicted in the previous frame are input, and the prediction result of the current frame is passed to the next iteration to form continuous streaming training;

[0047] Among them, at the initial stage of streaming training, short sequence images are input, and the length of the sequence is gradually increased during the entire training process. At the same time, the three-dimensional Gaussian representation of the previous frame is randomly discarded with probability p to actively terminate the streaming prediction of the sequence; the value of p is gradually reduced to 0 during the training process to complete the adaptation to the prediction of longer sequences.

[0048] To achieve the above object, the second aspect embodiment of the present application proposes a streaming three-dimensional semantic occupancy prediction device based on a Gaussian world model, including:

[0049] A modeling unit, which is used to explicitly model the evolution of the driving scene based on a three-dimensional Gaussian space, decompose the evolution of the driving scene into the ego-vehicle motion alignment of the static scene, the local motion of dynamic objects, and the scene completion of the newly observed area, and describe the driving scene using a sparse three-dimensional Gaussian representation;

[0050] An alignment unit, which is used to perform a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the ego-vehicle trajectory through an alignment module, align the positions of the three-dimensional Gaussians, and divide the aligned three-dimensional Gaussians into a dynamic Gaussian set and a static Gaussian set according to the semantic probability;

[0051] A motion modeling unit, which is used to model the local motion of dynamic Gaussians for the dynamic Gaussian set by combining a motion layer module and the visual input of the current frame, and update the position information of the dynamic Gaussians; for the static Gaussian set, keep its position information unchanged;

[0052] A completion unit, which is used to generate a new three-dimensional Gaussian representation for the newly observed area not covered in the current frame by random initialization, and complete its attributes through a perception layer module, and merge the dynamic Gaussians, static Gaussians, and the completed three-dimensional Gaussian into a complete three-dimensional Gaussian representation of the current frame;

[0053] An evolution adjustment unit, which is used to iteratively adjust the complete three-dimensional Gaussian representation using a unified evolution layer module, gradually optimize the attributes of the three-dimensional Gaussians by stacking multiple evolution layers, and combine a fine-tuning layer to adjust the global attributes of the Gaussian representation;

[0054] A prediction unit, which is used to predict the three-dimensional semantic occupancy of the current scene through a three-dimensional semantic occupancy prediction task based on all optimized three-dimensional Gaussian representations, and use cross-entropy loss and Lovasz loss to constrain the Gaussian world model, and output the three-dimensional semantic occupancy prediction result of the current scene.

[0055] To achieve the above object, an embodiment of the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0056] The memory stores computer-executable instructions;

[0057] The processor executes the computer-executable instructions stored in the memory to implement the method described in any one of the first aspect.

[0058] To achieve the above object, an embodiment of the fourth aspect of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any one of the first aspect.

[0059] To achieve the above object, an embodiment of the fifth aspect of the present application provides a computer program product, which implements the method described in any one of the first aspect when executed by a processor.

[0060] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0061] By explicitly modeling the evolution of the driving scene in the three-dimensional Gaussian space, the changes in the driving scene can be decomposed into the ego-vehicle motion alignment of the static scene, the local motion of dynamic objects, and the scene completion of the newly observed area, thereby effectively capturing the dynamic logic and temporal consistency of the scene evolution; using a sparse three-dimensional Gaussian representation to model the driving scene improves the computational efficiency while reducing redundant information, providing guarantees for the real-time performance and applicability of the model; through the alignment module, a global affine transformation is performed on the three-dimensional Gaussian representation based on the ego-vehicle trajectory to accurately correct the global displacement in the scene, and the dynamic and static Gaussian sets are divided by combining semantic probabilities, which helps to more accurately capture the local motion of dynamic objects; for the newly observed area, three-dimensional Gaussians are randomly initialized and their attributes are completed through the perception layer module, and a complete three-dimensional scene description is constructed by combining the dynamic and static Gaussian representations, significantly improving the integrity and accuracy of the three-dimensional semantic occupancy prediction; through the unified evolution layer module, the three-dimensional Gaussian representation is iteratively adjusted, and the global attributes are optimized by combining the fine-tuning layer, further enhancing the model's ability to model scene evolution; by using the cross-entropy loss and Lovasz loss to constrain the three-dimensional semantic occupancy prediction task, the semantic classification error and boundary accuracy are optimized, effectively improving the prediction performance of the model.

[0062] In summary, compared with the traditional temporal fusion method, the present application can model the static consistency of scene evolution and the motion of dynamic objects by explicitly modeling the evolution of the scene in the three-dimensional Gaussian space, improving the efficiency and effectiveness of the model's three-dimensional occupancy prediction.

[0063] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0064] The above-mentioned and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the drawings, where:

[0065] Figure 1 is a schematic flowchart of a method for streaming three-dimensional semantic occupancy prediction based on a Gaussian world model provided by an embodiment of the present application;

[0066] Figure 2 is a schematic architecture diagram of a method for streaming three-dimensional semantic occupancy prediction based on a Gaussian world model provided by an embodiment of the present application;

[0067] Figure 3 is a schematic architecture diagram of a unified evolution layer provided by an embodiment of the present application;

[0068] Figure 4 is a schematic diagram of the training process of a Gaussian world model provided by an embodiment of the present application;

[0069] Figure 5 is a schematic structural diagram of a device for streaming three-dimensional semantic occupancy prediction based on a Gaussian world model provided by an embodiment of the present application. Detailed Embodiments

[0070] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0071] To solve the problems existing in the prior art, the present application proposes a method for streaming three-dimensional semantic occupancy prediction based on a Gaussian world model, namely, the Gaussian world model, which improves the performance of perception tasks by modeling scene evolution. The present application uses an explicit three-dimensional Gaussian representation as the basis for scene modeling, rather than the traditional implicit bird's-eye view or voxel feature representation, so as to be able to explicitly and continuously describe the dynamic changes of objects in the scene. The three-dimensional Gaussian representation allows the model to learn the scene evolution process with real-world dynamic characteristics, where the position offset of the Gaussian directly reflects the movement trajectory of dynamic objects, significantly enhancing the interpretability of the model.

[0072] In order to meet the needs of perception tasks, this application redefines 3D semantic occupancy prediction as a 4D semantic occupancy prediction problem based on current observation input. Given the 3D Gaussian representation of the historical frames and the visual input of the current frame, the Gaussian world model aims to predict the evolution of the scene and output the 3D semantic occupancy result of the current frame.

[0073] To achieve the above goals, this application decomposes scene evolution into the following three key factors: ego-vehicle motion alignment of static scenes, local motion of dynamic objects, and scene completion of new observation areas. In response to these factors, this application first aligns the 3D Gaussian representation of historical frames to the current frame based on the ego-vehicle trajectory. For new observation areas, this application generates random Gaussians and completes them, so that the model can effectively model and perceive new areas.

[0074] In addition, this application proposes a unified evolution layer for simultaneously modeling the evolution process of historical Gaussians and the perception process of newly completed Gaussians. By combining three-dimensional convolution operations and deformable attention mechanisms, this application realizes interactive modeling of three-dimensional Gaussian representation and visual input, and characterizes the historical information of three-dimensional Gaussian representation by introducing temporal feature attributes. On this basis, the present invention uses the optimized three-dimensional Gaussian representation to predict the evolution process of the scene and outputs the three-dimensional semantic occupancy prediction result of the current scene.

[0075] Compared with traditional temporal fusion methods, this application can better capture the static consistency of scene evolution and the motion characteristics of dynamic objects by explicitly modeling the evolution of the scene in three-dimensional Gaussian space, thereby significantly improving the efficiency and accuracy of the model in three-dimensional semantic occupancy prediction tasks.

[0076] Reference Figure 1 and Figure 2 , the method specifically comprises the following steps:

[0077] Step 101, based on the three-dimensional Gaussian space, explicitly modeling the evolution of the driving scene, decomposing the evolution of the driving scene into the self-vehicle motion alignment of the static scene, the local motion of the dynamic object and the scene completion of the new observation area, and using a sparse three-dimensional Gaussian representation to describe the driving scene.

[0078] In driving scenarios, the evolution process is usually manifested as simple and continuous dynamic changes, mainly caused by the motion of dynamic objects. In a scene representation centered on the ego vehicle and limited to a certain range, this evolution can be decomposed into three key factors: alignment of the ego vehicle motion with static scenes, local motion of dynamic objects, and scene completion of new observation areas. By modeling these factors, the world model can effectively learn and simulate the evolution of the scene.

[0079] The embodiments of this application use a set of sparse three-dimensional Gaussian representations \(g = \{p, s, r, c, f\}\) to describe the driving scenario, where \(p\), \(s\), \(r\), \(c\), and \(f\) represent the position, scale, rotation, semantic probability, and temporal characteristics of the three-dimensional Gaussian, respectively, and are used to determine the specific coordinates of the Gaussian in space, describe its coverage range, describe its direction information, describe the semantic characteristics of the area, and depict the historical evolution information of the three-dimensional Gaussian, respectively.

[0080] It can be understood that the explicit representation of the three-dimensional Gaussian has significant advantages. By simply moving the position attribute of the Gaussian, the motion process of the object can be continuously and explicitly simulated, thus being closer to the dynamic evolution logic of the real world. Compared with traditional implicit representation methods (such as bird's-eye view or voxel feature representation), the three-dimensional Gaussian representation can explicitly model the evolution dynamics of the scene while maintaining high computational efficiency, improving the interpretability and performance of the model.

[0081] Step 102: Use the alignment module to perform a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the ego-vehicle trajectory, align the positions of the three-dimensional Gaussians, and divide the aligned three-dimensional Gaussians into a dynamic Gaussian set and a static Gaussian set according to the semantic probability.

[0082] The embodiments of this application further propose a three-dimensional Gaussian world model to perform perception tasks by modeling scene evolution. The three-dimensional Gaussian world model predicts the three-dimensional Gaussian representation of the current frame based on the three-dimensional Gaussian representation of the previous frame and the current frame visual input. Next, how the three-dimensional Gaussian world model models the above-decomposed scene evolution factors in the three-dimensional Gaussian space will be introduced.

[0083] The goal of the three-dimensional Gaussian world model is to predict the three-dimensional Gaussian representation of the current frame based on the three-dimensional Gaussian representation of the previous frame. The three-dimensional Gaussian representation of each frame is centered on the ego-vehicle position of the corresponding frame and describes the scene information within a certain range, while the motion of the ego-vehicle will cause a relative displacement of the global scene.

[0084] Therefore, in this step, the embodiments of this application propose a method to perform a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the ego-vehicle trajectory through the alignment module, align the positions of the three-dimensional Gaussians, and divide the aligned three-dimensional Gaussians into a dynamic Gaussian set and a static Gaussian set according to the semantic probability.

[0085] Specifically, this application uses the alignment module \(A\) lign to generate an affine transformation matrix \(M\) according to the ego-vehicle trajectory ego , and perform global alignment on the position attribute \(p\) of the three-dimensional Gaussian representation \(g\) of the previous frame T-1 . The aligned three-dimensional Gaussian is represented as:

[0086]

[0087] Among them, A ttr (g; p) represents the property p of the three-dimensional Gaussian g, and R ef (g; n; p) represents updating the property p of the Gaussian g with n.

[0088] In addition, the embodiments of the present application also consider the local motion of dynamic objects in the scene evolution and achieve this by updating the positions of the dynamic Gaussians.

[0089] After the alignment is completed, the present application divides the aligned three-dimensional Gaussians according to the semantic probability c to form two non-overlapping sets: the dynamic Gaussian set {g D} and the static Gaussian set {g S}. The dynamic Gaussian set contains the three-dimensional Gaussian representations corresponding to the moving objects in the scene, and the static Gaussian set contains the three-dimensional Gaussian representations related to the fixed scene elements.

[0090] Through the alignment and division in this step, the three-dimensional Gaussian world model can effectively handle the global displacement of the scene caused by the motion of the ego vehicle, and at the same time provides a data basis for the separation modeling of dynamic objects and static scenes.

[0091] Step 103: For the dynamic Gaussian set, model the local motion of the dynamic Gaussians by combining the motion layer module and the visual input of the current frame, and update the position information of the dynamic Gaussians; for the static Gaussian set, keep its position information unchanged.

[0092] After dividing the aligned three-dimensional Gaussians, the embodiments of the present application further propose a motion modeling method for the dynamic Gaussian set and the static Gaussian set, model the local motion of the dynamic Gaussians and update their position information, while keeping the position information of the static Gaussians unchanged.

[0093] Specifically, the embodiments of the present application use the motion layer module M ove Based on the aligned three-dimensional Gaussians and the visual input x of the current frame T learn the local motion of the dynamic Gaussian set {g D} and update its position information. The update process is achieved through the following formula:

[0094]

[0095] Among them, is the encoder, which is used to interactively model between the three-dimensional Gaussians and the visual input; I(·) represents the indicator function;

[0096] As for the static Gaussian set {g S}, keep its position property p unchanged and do not update its position.

[0097] Through this step, the present embodiment effectively models the local motion characteristics of the dynamic Gaussian, ensuring accurate prediction of the motion trajectories of dynamic objects while keeping the positions of the static Gaussians unchanged, thereby further enhancing the model's ability to model scene evolution.

[0098] Step 104: For the newly observed regions not covered in the current frame, generate new three-dimensional Gaussian representations by random initialization, and complete their attributes through the perception layer module. Combine the dynamic Gaussians, static Gaussians, and the completed three-dimensional Gaussians into a complete three-dimensional Gaussian representation of the current frame.

[0099] When the ego vehicle moves to a new position, some scene regions may go beyond the current field of view, while some new regions enter the observable range. To update the current observable region at all times, the present application directly discards the three-dimensional Gaussians that go beyond the boundary and uses randomly initialized three-dimensional Gaussians to complete these newly observed regions. To keep the number of three-dimensional Gaussians constant, the present application uniformly samples an equal number of three-dimensional Gaussians in the newly observed regions.

[0100] Subsequently, the embodiment of the present application passes through the perception layer module P er Based on the visual input x of the current frame T For the newly initialized three-dimensional Gaussians Complete all attributes, including position, scale, rotation, semantic probability, and temporal features. The completion process is achieved through the following formula:

[0101]

[0102] where E nc and M ove in use the same model structure and parameters for the encoder.

[0103] After completing the completion of the new regions, the embodiment of the present application further combines the dynamic Gaussian set {g D}, the static Gaussian set {g S}, and the completed three-dimensional Gaussian set to form a complete three-dimensional Gaussian representation of the current frame

[0104] Through the above method, the embodiment of the present application can effectively update the observable region information of the current frame, ensuring the integrity of scene evolution modeling. This step enhances the model's temporal perception ability for dynamic and static scenes while ensuring the continuity of scene description.

[0105] Step 105: Use a unified evolution layer module to iteratively adjust the complete three-dimensional Gaussian representation, gradually optimize the three-dimensional Gaussian attributes by stacking multiple evolution layers, and combine a fine-tuning layer to adjust the global attributes of the Gaussian representation.

[0106] Based on the above modeling of the scene evolution decomposition factors, an embodiment of the present application proposes a Gaussian world model to model scene evolution and promote temporal perception. This model starts from the three-dimensional Gaussian representation g of the previous frame T-1 and first obtains the aligned three-dimensional Gaussian representation of the current frame through the alignment module A lign In the new observation area, randomly sample additional three-dimensional Gaussians and merge them with to form the complete three-dimensional Gaussian representation g of the current frame to jointly describe the current scene. T

[0107] To further optimize the scene modeling effect, referring to Figure 3 after completing alignment, dynamic object modeling, and filling in the new observation area, an embodiment of the present application uses a unified evolution layer module E vol to iteratively adjust the complete three-dimensional Gaussian representation g T so that it can more accurately model the dynamic change process of the scene. Each evolution layer uses a unified structure to update the attributes {p, s, r, c, f} of the three-dimensional Gaussian representation, and the formula is:

[0108]

[0109] where represents the three-dimensional Gaussian representation in the current evolution layer.

[0110] In each layer, the evolution layer module can dynamically select to use the perception layer module P er or the motion layer module M ove to update the attributes of the three-dimensional Gaussian representation. By stacking n e evolution layers, the model can iteratively adjust the three-dimensional Gaussian representation multiple times, thereby endowing the model with a powerful scene evolution modeling ability.

[0111] In addition, to solve the possible misalignment problem between the three-dimensional Gaussian representation and the real-world scene, the present application introduces n r fine-tuning layers R efine to optimize the global attributes of the three-dimensional Gaussian. The adjustment process formula of the fine-tuning layer is:

[0112]

[0113] where R efine is the fine-tuning layer.

[0114] By stacking multiple evolution layers, the implementation of the present application can iteratively adjust the dynamic attributes of the three-dimensional Gaussian and learn the internal laws of scene evolution. At the same time, the fine-tuning layer optimizes the global attributes of the three-dimensional Gaussian, further enhancing the model's ability to model real scenes. This process not only maintains the efficient computational characteristics of the model but also enhances the accuracy and consistency of scene evolution modeling.

[0115] Step 106: Based on all optimized three-dimensional Gaussian representations, predict the three-dimensional semantic occupancy of the current scene through a three-dimensional semantic occupancy prediction task, and use the cross-entropy loss and Lovasz loss to constrain the Gaussian world model to output the three-dimensional semantic occupancy prediction result of the current scene.

[0116] In the embodiment of the present application, a method for predicting the three-dimensional semantic occupancy of the current scene through a three-dimensional semantic occupancy prediction task based on all optimized three-dimensional Gaussian representations is proposed, and the cross-entropy loss and Lovasz loss are used to constrain the Gaussian world model to output the three-dimensional semantic occupancy prediction result of the current scene.

[0117] During the training process, the embodiment of the present application uses the cross-entropy loss and Lovasz loss to optimize the Gaussian world model. The cross-entropy loss is used to constrain the semantic classification error, and the Lovasz loss is used to optimize the semantic segmentation boundary.

[0118] Refer to Figure 4 , the training process includes two stages:

[0119] First, pre-train the Gaussian world model on the single-frame prediction task. Taking the single-frame image and the corresponding three-dimensional Gaussian representation as inputs, the model mainly learns the basic semantic occupancy prediction ability in this stage to improve the modeling effect of the single-frame scene.

[0120] Subsequently, use a streaming training strategy to fine-tune the model. During the streaming training process, input the sequence images of the scene into the model in sequence, and combine the current frame image and the three-dimensional Gaussian representation predicted by the previous frame to perform three-dimensional semantic occupancy prediction. The specific operation is as follows: the prediction result g of the current frame T is passed to the next iteration to construct a continuous streaming training process; gradually improve the model's ability to model multi-frame dynamic scenes as the training process progresses.

[0121] It should be noted that in the early stage of streaming training, the model is not yet good at predicting scene evolution, resulting in a huge cumulative streaming prediction error. To enhance the stability of streaming training, the embodiment of the present application inputs short sequence images at the initial stage of training and gradually increases the sequence length.

[0122] Specifically, the embodiment of the present application introduces a probability modeling method, and randomly discards the three-dimensional Gaussian representation of the previous frame with probability p. The formula is:

[0123]

[0124] where t is the current iteration number, and T is the maximum iteration number.

[0125] Thus, in each iteration, the three-dimensional Gaussian representation of the previous frame is randomly discarded with probability p to actively terminate the streaming prediction of the sequence and reduce the influence of cumulative errors. As the training progresses, the value of p is gradually reduced to 0, enabling the model to gradually adapt to the prediction task of longer sequences.

[0126] Through the above training strategy, the Gaussian world model can fully learn the dynamic logic of scene evolution, show higher accuracy and robustness in the three-dimensional semantic occupancy prediction task of the current scene, and finally output a reliable three-dimensional semantic occupancy prediction result, providing strong support for downstream tasks.

[0127] To implement the above embodiment, the present application also proposes a streaming three-dimensional semantic occupancy prediction device based on the Gaussian world model. Figure 5 It is a schematic structural diagram of a streaming three-dimensional semantic occupancy prediction device 10 provided by an embodiment of the present application. As Figure 5 shown, the device includes:

[0128] A modeling unit 100, configured to explicitly model the evolution of the driving scene based on a three-dimensional Gaussian space, decompose the evolution of the driving scene into the ego-vehicle motion alignment of the static scene, the local motion of dynamic objects, and the scene completion of the newly observed area, and describe the driving scene using a sparse three-dimensional Gaussian representation;

[0129] An alignment unit 200, configured to perform a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the ego-vehicle trajectory through an alignment module, align the positions of the three-dimensional Gaussians, and divide the aligned three-dimensional Gaussians into a dynamic Gaussian set and a static Gaussian set according to the semantic probability;

[0130] A motion modeling unit 300, configured to, for the dynamic Gaussian set, model the local motion of the dynamic Gaussians in combination with the motion layer module and the visual input of the current frame, and update the position information of the dynamic Gaussians; for the static Gaussian set, keep its position information unchanged;

[0131] A completion unit 400, configured to generate a new three-dimensional Gaussian representation for the newly observed area not covered in the current frame by random initialization, and complete its attributes through the perception layer module, and merge the dynamic Gaussians, static Gaussians, and the completed three-dimensional Gaussian into a complete three-dimensional Gaussian representation of the current frame;

[0132] An evolution adjustment unit 500 is used to iteratively adjust the complete three-dimensional Gaussian representation by using a unified evolution layer module, gradually optimize the three-dimensional Gaussian attributes by stacking multiple evolution layers, and combine a fine-tuning layer to adjust the global attributes of the Gaussian representation.

[0133] A prediction unit 600 is used to predict the three-dimensional semantic occupancy of the current scene through a three-dimensional semantic occupancy prediction task based on all optimized three-dimensional Gaussian representations, and use cross-entropy loss and Lovasz loss to constrain the Gaussian world model, and output the three-dimensional semantic occupancy prediction result of the current scene.

[0134] To implement the above embodiments, the present application also proposes an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0135] To implement the above embodiments, the present application also proposes a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are used to implement the method provided in the foregoing embodiments when executed by a processor.

[0136] To implement the above embodiments, the present application also proposes a computer program product including a computer program, and the computer program implements the method provided in the foregoing embodiments when executed by a processor.

[0137] The collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved in the present application and other processing are all in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0138] It should be noted that personal information from users should be collected for legal and reasonable purposes and not shared or sold outside of these legal uses. In addition, such collection / sharing should be carried out after obtaining the informed consent of the user, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization including authorizing relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0139] The present application anticipates providing embodiments that allow users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.

[0140] In the description of the foregoing embodiments, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0141] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0142] Any process or method description in the flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions may be performed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0143] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0144] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0145] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0146] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0147] The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present application.

[0148] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in the present application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present application can be achieved, and no limitation is made herein.

[0149] The above specific implementation manners do not constitute a limitation to the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A streaming three-dimensional semantic occupancy prediction method based on a Gaussian world model, characterized in that: The following steps are involved: The evolution of driving scenes is explicitly modeled based on three-dimensional Gaussian space. The evolution of driving scenes is decomposed into the alignment of ego-vehicle motion in static scenes, the local motion of dynamic objects, and the scene completion of new observation areas. The driving scenes are described using sparse three-dimensional Gaussian representation. The alignment module performs a global affine transformation on the 3D Gaussian representation of the previous frame based on the vehicle trajectory, aligns the position of the 3D Gaussian, and divides the aligned 3D Gaussian into a dynamic Gaussian set and a static Gaussian set according to the semantic probability. For the dynamic Gaussian set, the local motion of the dynamic Gaussian is modeled by combining the motion layer module and the visual input of the current frame, and the position information of the dynamic Gaussian is updated; for the static Gaussian set, its position information is kept unchanged; For the new observation area not covered in the current frame, a new 3D Gaussian representation is generated by random initialization, and its attributes are completed through the perception layer module. The dynamic Gaussian, static Gaussian and completed 3D Gaussian are merged into a complete 3D Gaussian representation of the current frame. A unified evolution layer module is used to iteratively adjust the complete 3D Gaussian representation. Multiple evolution layers are stacked to gradually optimize the 3D Gaussian properties. The global properties of the Gaussian representation are adjusted in combination with the fine-tuning layer. Based on all optimized 3D Gaussian representations, the 3D semantic occupancy of the current scene is predicted through the 3D semantic occupancy prediction task, and the Gaussian world model is constrained using cross entropy loss and Lovasz loss to output the 3D semantic occupancy prediction result of the current scene.

2. The method according to claim 1, characterized in that The driving scene is described using a sparse three-dimensional Gaussian representation, including: A set of sparse three-dimensional Gaussian representations g = {p, s, r, c, f} are used to describe the driving scene, where p, s, r, c, f represent the position, scale, rotation, semantic probability and temporal features of the three-dimensional Gaussian, respectively.

3. The method according to claim 2, characterized in that The alignment module performs a global affine transformation on the 3D Gaussian representation of the previous frame based on the vehicle trajectory, adjusts the position of the 3D Gaussian, and divides the adjusted 3D Gaussian into a dynamic Gaussian set and a static Gaussian set according to the semantic probability, including: Use the alignment module A lign Generate the affine transformation matrix M according to the vehicle trajectory ego , and the three-dimensional Gaussian representation g of the previous frame T-1 The position attribute p is globally aligned, and the aligned three-dimensional Gauss It is expressed as: Among them, A ttr (g; p) represents the property p of the three-dimensional Gaussian g, R ef (g; n; p) means using n to update the attribute p of Gaussian g; The aligned three-dimensional Gaussian is divided according to the semantic probability c to form a dynamic Gaussian set {g D } and the static Gaussian set {g S }.

4. The method according to claim 3, characterized in that The method of modeling the local motion of the dynamic Gaussian set by combining the motion layer module and the visual input of the current frame, and updating the position information of the dynamic Gaussian set; and keeping the position information of the static Gaussian set unchanged includes: Through the motion layer module M ove Based on the aligned 3D Gaussian and the visual input x of the current frame T Learning a dynamic Gaussian set {g D } local movement, update its position information, the update process is achieved through the following formula: in, is an encoder used to model the interaction between 3D Gaussian and visual input; I(·) represents the indicative function; For a static Gaussian set {g S }, keeping its position attribute p unchanged.

5. The method according to claim 4, characterized in that For the new observation area not covered in the current frame, a new 3D Gaussian representation is generated by random initialization, and its attributes are completed through the perception layer module. The dynamic Gaussian, static Gaussian and completed 3D Gaussian are merged into a complete 3D Gaussian representation of the current frame, including: Uniformly sample the same number of randomly initialized 3D Gaussians in the new observation region Through the perception layer module P er Based on the visual input x of the current frame T For the newly initialized three-dimensional Gaussian All attributes of are completed, and the completion process is implemented by the following formula: Among them, E nc With M ove The encoder in uses the same model structure and parameters; The dynamic Gaussian set {g D }、Static Gaussian set {g S } and the completed three-dimensional Gaussian set Merge to form a complete 3D Gaussian representation of the current frame 6. The method according to claim 5, characterized in that A unified evolution layer module is used to iteratively adjust the complete 3D Gaussian representation. Multiple evolution layers are stacked to gradually optimize the 3D Gaussian properties. The fine-tuning layer is combined to adjust the global properties of the Gaussian representation, including: Through the unified evolution layer module E vol For the complete three-dimensional Gaussian representation g T Iterative adjustment is performed, where each evolution layer uses a unified structure to update the attributes {p, s, r, c, f} of the three-dimensional Gaussian representation. The formula is: in, Represents the three-dimensional Gaussian representation in the current evolution layer; Introduce r The fine-tuning layer adjusts the global properties of the three-dimensional Gaussian representation, and the formula is: Among them, R efine It is the fine-tuning layer.

7. The method according to claim 6, characterized in that The training process of the Gaussian world model includes: Cross entropy loss and lovasz loss are used for training, and the Gaussian world model is pre-trained on a single frame prediction task; The model is fine-tuned using a streaming training strategy. The sequence of scene images is input into the model in sequence, and the 3D semantic occupancy prediction is performed by combining the current frame image and the 3D Gaussian representation predicted in the previous frame. In the process of streaming training, the current frame image and the three-dimensional Gaussian representation of the previous frame prediction are input, and the prediction result of the current frame is passed to the next iteration, forming continuous streaming training; Among them, a short sequence of images is input at the beginning of streaming training, and the length of the sequence is gradually increased during the entire training process. At the same time, the three-dimensional Gaussian representation of the previous frame is randomly discarded with probability p to actively terminate the streaming prediction of the sequence; as the training process progresses, the value of p is gradually reduced to 0 to complete the adaptation to longer sequence predictions.

8. A streaming three-dimensional semantic occupancy prediction device based on a Gaussian world model, characterized in that: include: A modeling unit, which is used to explicitly model the evolution of driving scenes based on a three-dimensional Gaussian space, decompose the evolution of driving scenes into the alignment of ego-vehicle motion of static scenes, the local motion of dynamic objects, and the scene completion of new observation areas, and describe the driving scenes using a sparse three-dimensional Gaussian representation; An alignment unit, used for performing a global affine transformation on the three-dimensional Gaussian representation of the previous frame based on the trajectory of the vehicle through the alignment module, aligning the positions of the three-dimensional Gaussian, and dividing the aligned three-dimensional Gaussian into a dynamic Gaussian set and a static Gaussian set according to semantic probability; The motion modeling unit is used to model the local motion of the dynamic Gaussian set by combining the motion layer module and the visual input of the current frame, and update the position information of the dynamic Gaussian; for the static Gaussian set, keep its position information unchanged; The completion unit is used to generate a new 3D Gaussian representation of the new observation area not covered in the current frame by random initialization, and complete its attributes through the perception layer module, and merge the dynamic Gaussian, static Gaussian and completed 3D Gaussian into a complete 3D Gaussian representation of the current frame; An evolution adjustment unit, which is used to iteratively adjust the complete three-dimensional Gaussian representation using a unified evolution layer module, gradually optimize the three-dimensional Gaussian properties by stacking multiple evolution layers, and adjust the global properties of the Gaussian representation in combination with the fine-tuning layer; The prediction unit is used to predict the 3D semantic occupancy of the current scene through the 3D semantic occupancy prediction task based on all optimized 3D Gaussian representations, and use the cross entropy loss and Lovasz loss to constrain the Gaussian world model to output the 3D semantic occupancy prediction result of the current scene.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images