Methods, devices, equipment, media, and products for constructing four-dimensional scenes

By constructing a four-dimensional scene model using multi-view videos, the high cost and noise issues in existing technologies are solved, enabling low-cost, high-quality dynamic scene reconstruction and improving the scene construction capabilities for autonomous driving simulation and digital twins of urban scenes.

CN123071531APending Publication Date: 2026-11-10北京极佳视界科技股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611158205.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-31
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing scene reconstruction technologies rely on dense LiDAR scanning data and manual 3D tracking and annotation, which are costly and prone to introducing noise, making it difficult to achieve high-quality dynamic urban street scene modeling.

Method used

By acquiring target videos from multiple perspectives, a four-dimensional scene model is constructed using pure visual input, including global background point cloud, global ground point cloud, dynamic target point cloud sequence, and environmental texture map. Gaussian parameters and motion information are used for scene reconstruction, eliminating the reliance on LiDAR and manual annotation.

Benefits of technology

It achieves low-cost, high-quality dynamic scene reconstruction, improves scene expression capabilities and spatiotemporal consistency, and provides a high-quality four-dimensional scene construction solution for autonomous driving simulation and digital twins of urban scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071531A_ABST
    Figure CN123071531A_ABST
Patent Text Reader

Abstract

This disclosure provides a method, apparatus, device, medium, and product for constructing a four-dimensional scene. It involves acquiring target videos from multiple perspectives, determining global background point clouds, global ground point clouds, environmental texture maps, and dynamic target point cloud sequences based on each target video, obtaining Gaussian parameters corresponding to the global background point cloud, global ground point cloud, and dynamic target point cloud sequences, determining the motion information of the dynamic targets based on the Gaussian parameters of the dynamic target point cloud sequence, and constructing a four-dimensional scene model based on the Gaussian parameters, motion information, and environmental texture maps. This fully automated four-dimensional reconstruction of dynamic scenes is achieved through pure visual input (target videos from multiple perspectives), effectively reducing data acquisition and modeling costs and eliminating the reliance on LiDAR, pre-calibrated poses, and manual annotation found in traditional methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to scene reconstruction technology, and in particular to a method, apparatus, device, medium and product for constructing a four-dimensional scene. Background Technology

[0002] With the rapid development of autonomous driving technology, simulation systems are increasingly demanding high-fidelity modeling and rendering of dynamic urban street scenes. Currently, scene reconstruction often employs annotation-based supervised methods, such as Street Gaussians and OmniRe. These methods typically rely on dense LiDAR scan data, pre-calibrated vehicle poses, and manual 3D tracking annotations to complete modeling, achieving urban scene reconstruction and rendering by decoupling scene components or optimizing dynamic Gaussian poses. However, this scene reconstruction approach is highly data-dependent, has high modeling costs, and is prone to introducing noise. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a method, apparatus, device, medium, and product for constructing a four-dimensional scene.

[0004] One aspect of this disclosure provides a four-dimensional scene construction method, comprising: acquiring target videos from multiple perspectives, wherein each target video includes: dynamic targets, sky elements, ground elements, and background elements; determining a global background point cloud, a global ground point cloud, an environment texture map, and a dynamic target point cloud sequence based on each target video; acquiring Gaussian parameters corresponding to the global background point cloud, the global ground point cloud, and the dynamic target point cloud sequence, respectively; determining motion information of the dynamic targets based on the Gaussian parameters of the dynamic target point cloud sequence; and constructing a four-dimensional scene model based on the Gaussian parameters, the motion information, and the environment texture map.

[0005] In another aspect of this disclosure, a four-dimensional scene construction apparatus is provided, comprising: a video acquisition module for acquiring target videos from multiple perspectives, wherein the target videos from any perspective include: dynamic targets, sky elements, ground elements, and background elements; a point cloud acquisition module for determining a global background point cloud, a global ground point cloud, an environmental texture map, and a dynamic target point cloud sequence based on each target video; a Gaussian creation module for acquiring Gaussian parameters corresponding to the global background point cloud, the global ground point cloud, and the dynamic target point cloud sequence, respectively; a motion model creation module for determining motion information of the dynamic target based on the Gaussian parameters of the dynamic target point cloud sequence; and a scene creation module for constructing a four-dimensional scene model based on the Gaussian parameters, the motion information, and the environmental texture map.

[0006] In another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the method described above.

[0007] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0008] In another aspect of this disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described above.

[0009] Based on four-dimensional scene construction methods, devices, equipment, media, and products, fully automated four-dimensional reconstruction of dynamic scenes is achieved through pure visual input (target videos from multiple perspectives). This effectively reduces data acquisition and modeling costs and eliminates the reliance on LiDAR, pre-calibrated poses, and manual annotation found in traditional methods. Furthermore, by differentiated representation and unified modeling of background elements, ground elements, dynamic targets, and sky elements in the scene, the static geometric structure of the scene and the motion trajectory of dynamic targets can be more completely and accurately restored. This enhances the model's scene representation capabilities and spatiotemporal consistency, providing a high-quality, low-threshold four-dimensional scene construction solution for applications such as autonomous driving simulation and digital twins of urban scenes.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 This is a flowchart illustrating a four-dimensional scene construction method provided in an exemplary embodiment of this application.

[0013] Figure 2 This is a flowchart illustrating step S110 provided in an exemplary embodiment of this application.

[0014] Figure 3 This is a flowchart illustrating step S112 provided in an exemplary embodiment of this application.

[0015] Figure 4 This is a flowchart illustrating step S113 provided in an exemplary embodiment of this application.

[0016] Figure 5This is a flowchart illustrating a four-dimensional scene construction method provided in another exemplary embodiment of this application.

[0017] Figure 6 This is a flowchart illustrating step S130 provided in an exemplary embodiment of this application.

[0018] Figure 7 This is a flowchart illustrating step S140 provided in an exemplary embodiment of this application.

[0019] Figure 8 This is a schematic diagram of the four-dimensional scene construction method provided in one application implementation of this application.

[0020] Figure 9 This is a schematic diagram of the structure of a four-dimensional scene construction device provided in an exemplary embodiment of this disclosure.

[0021] Figure 10 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0022] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0023] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0024] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0025] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0026] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0027] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0028] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0029] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0030] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0031] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0032] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0033] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0034] In the process of realizing the technology of this application, it was found through research that in the current scene reconstruction technology, it is necessary to rely on dense LiDAR scanning data, pre-calibrated vehicle pose and manual 3D tracking annotation at the same time. However, the high-precision hardware equipment required for data acquisition and pose calibration is expensive, and the time and labor costs of manual 3D annotation are extremely high. Moreover, the annotation coverage is limited and it is easy to introduce noise, which restricts the improvement of modeling accuracy and generalization ability.

[0035] Figure 1This is a flowchart illustrating a four-dimensional scene construction method provided in an exemplary embodiment of this application. This embodiment can be applied to electronic devices. Figure 1 As shown, the four-dimensional scene construction method includes the following steps: Step S100: Acquire target videos from multiple perspectives.

[0036] The target videos from multiple perspectives include: using the vehicle as a reference, simultaneously capturing videos from different directions (viewpoints) around the vehicle via multi-channel video acquisition devices (e.g., cameras, monocular cameras, fisheye cameras, etc.) configured around the vehicle. Each target video corresponds to one viewpoint. For example, target videos from multiple perspectives can include: front view, rear view, left front view, right front view, left rear view, right rear view, bird's-eye view, and panoramic view. Target videos from any single viewpoint include: dynamic targets, sky elements, ground elements, and background elements. Background elements are elements in the target video other than dynamic targets, sky elements, and ground elements. Dynamic targets are entities with independent movement capabilities whose spatial position changes over time; for example, dynamic targets include at least one of the following: pedestrians, various motorized or non-motorized vehicles, drones, robots, etc. In one example, each target video can include one or more dynamic targets.

[0037] Step S110: Based on each target video, determine the global background point cloud, global ground point cloud, environmental texture map, and dynamic target point cloud sequence.

[0038] The global background point cloud describes the 3D point cloud of static background structures in the scene, excluding ground and sky elements. It can include stationary objects such as buildings, vegetation, and transportation facilities. The global ground point cloud describes the 3D point cloud of roads and drivable areas, and can include road surface geometry and texture information. The environment texture map is a learnable spherical or cubic texture map used to represent the appearance of the sky region in the scene. Each pixel corresponds to a viewing direction in 3D space, and sky color is rendered through direction sampling. The dynamic target point cloud sequence refers to a collection of 3D point clouds constructed from tracked dynamic targets (e.g., vehicles, pedestrians) arranged in chronological order. Each dynamic target has a corresponding dynamic target point cloud sequence, and each sequence includes multiple dynamic target point clouds of that target arranged in chronological order. Each dynamic target point cloud in the sequence describes the 3D geometry of the target at the corresponding moment.

[0039] In one implementation, multiple frames of video images with image quality meeting a preset image standard can be selected from each target video to generate a global background point cloud, a global ground point cloud, an environmental texture map, and a dynamic target point cloud sequence; alternatively, all target videos can be used to generate a global background point cloud, a global ground point cloud, an environmental texture map, and a dynamic target point cloud sequence.

[0040] For example, for each video image in each target video, semantic segmentation is performed on the video image to generate a background mask for background elements, a ground mask for ground elements, a dynamic target mask for dynamic targets, and a sky mask for sky elements in the video image. Based on each mask, the semantic regions of background elements, ground elements, dynamic targets, and sky elements are identified and segmented, and extracted to form a background image containing only background elements, a ground image containing only ground elements, a dynamic target image containing only dynamic targets, and a sky image containing only sky elements. The background image, ground image, dynamic target image, and sky image corresponding to each frame image respectively constitute a background image sequence, a ground image sequence, a dynamic target image sequence, and a sky image sequence.

[0041] A sequence of sky images can be input into a pre-trained graph stitching model. This model performs spherical projection transformation and graph fusion on each frame of the sky image sequence, outputting an environmental texture map. The graph stitching model can, for example, employ a convolutional neural network (CNN) or a generative adversarial network (GAN); or... Based on the position and viewing direction of each sky image in the camera coordinate system, it is projected onto a unit sphere. In the spherical coordinate system, the pixel colors of sky images from different frames and perspectives are weighted and fused, with the weights determined based on the pixel's position in the image (e.g., higher weight for the central region). The fused spherical texture is then unfolded into an equidistant rectangular projection image, yielding the initial environment texture image. For each sky image frame, the viewing direction is determined as follows: based on the pixel coordinates of the sky elements, coordinate system transformation is performed using the camera intrinsic parameters (focal length and principal point coordinates) of that frame to obtain the coordinates of the sky elements in the camera coordinate system. Using the rotation matrix in the camera extrinsic parameters of that frame, the coordinates in the camera coordinate system are transformed to the world coordinate system to obtain the viewing direction. First, obtain the video depth map sequence corresponding to each target video. Each video depth map sequence includes the depth information of its corresponding target video. Then, input the background map sequence, ground map sequence, and dynamic target map sequence, along with their corresponding video depth map sequences, into the point cloud generation model. The point cloud generation model outputs a global background point cloud, a global ground point cloud, and a dynamic target point cloud sequence arranged by object identifier and time sequence. The point cloud generation model can, for example, use a convolutional neural network (CNN) or a graph neural network (GNN).

[0042] Step S120: Obtain the Gaussian parameters corresponding to the global background point cloud, global ground point cloud, and dynamic target point cloud sequences, respectively.

[0043] Based on 3D Gaussian sputtering (3DGS) technology, points in the global background point cloud, global ground point cloud, and dynamic target point cloud sequence are converted into Gaussian ellipsoids. The Gaussian parameters of the global background point cloud include: position, scale, rotation, opacity, and spherical harmonic coefficient color information of each Gaussian ellipsoid. The Gaussian parameters of the global ground point cloud also include: position, scale, rotation, opacity, and spherical harmonic coefficient color information of each Gaussian ellipsoid. The Gaussian parameters of the dynamic target point cloud sequence include: position, scale, rotation, opacity, and spherical harmonic coefficient color information of each Gaussian ellipsoid corresponding to each dynamic target point cloud. The position information includes the three-dimensional coordinates of the center point of each Gaussian ellipsoid in three-dimensional space. ); Scale information includes the three-dimensional vectors of each Gaussian ellipsoid in the corresponding point cloud ( The rotation information includes the orientation of each Gaussian ellipsoid, usually represented as a 3×3 rotation matrix; the opacity information includes the transparency of each Gaussian ellipsoid, which is a scalar value, usually ranging from [0, 1]; the spherical harmonic coefficient color information includes the spherical harmonic function coefficients of each Gaussian ellipsoid in the corresponding point cloud.

[0044] The Gaussian parameters mentioned above can be set according to actual needs, or they can be set based on experience. For example, in the Gaussian parameters of a dynamic target, the size information can be set to a fixed single scale, the rotation information can be set to an identity matrix, and the spherical harmonic coefficients and color information can be set to zero-order spherical harmonic coefficients.

[0045] Step S130: Determine the motion information of the dynamic target based on the Gaussian parameters of the dynamic target point cloud sequence.

[0046] Motion information is used to characterize the continuous motion state and spatial position changes of a dynamic target along the time axis. Motion information includes: the dynamic target's identifier (ID) and motion model; the motion model is used to calculate the spatial position of the dynamic target at any given time. Based on the positional information corresponding to each dynamic target point cloud, the motion model of the target point cloud can be fitted using curve fitting algorithms (e.g., least squares method) or fitting software (e.g., MATLAB).

[0047] Step S140: Construct a four-dimensional scene model based on Gaussian parameters, motion information, and environmental texture map.

[0048] Among them, a four-dimensional scene model refers to an editable digital scene capable of responding to spatiotemporal queries from multiple moments and perspectives, and rendering corresponding two-dimensional graphs. The construction process includes: The Gaussian parameters, trajectory information, and environmental texture map are assembled according to a predefined decoupled scene graph structure to form background nodes, ground nodes, dynamic target nodes, and sky nodes. Background nodes include the Gaussian parameters of the global background point cloud, used to represent static background structures in the scene; ground nodes include the Gaussian parameters of the global ground point cloud, used to represent road surfaces; dynamic target nodes include the Gaussian parameters and motion model of the dynamic target point cloud sequence, used to calculate the spatial position of the dynamic target at that moment based on the input time; sky nodes include the environmental texture map, used to provide sky color based on the viewing direction. By calling the rendering function and inputting any time and any camera parameters, a two-dimensional map of the corresponding spatiotemporal point can be rendered based on the Gaussian parameters and environmental texture map, thus constructing a continuously sampled four-dimensional scene model.

[0049] In this embodiment, fully automatic four-dimensional reconstruction of dynamic scenes is achieved through pure visual input (target videos from multiple perspectives), effectively reducing data acquisition and modeling costs and eliminating the reliance on LiDAR, pre-calibrated poses, and manual annotation found in traditional methods. Simultaneously, by differentiated representation and unified modeling of background elements, ground elements, dynamic targets, and sky elements in the scene, the static geometric structure of the scene and the motion trajectory of dynamic targets can be more completely and accurately restored, improving the model's scene representation capability and spatiotemporal consistency. This provides a high-quality, low-threshold four-dimensional scene construction solution for applications such as autonomous driving simulation and digital twins of urban scenes.

[0050] Figure 2 This is a flowchart illustrating step S110 provided in an exemplary embodiment of this application. In some alternative embodiments, such as Figure 2 As shown, step S110 includes the following steps: Step S111: Based on each target video, use a depth estimation model to determine the depth map sequence of each target video.

[0051] The depth estimation model is a deep learning model used to estimate depth information in video images; for example, the Video Depth Anything model can be used. For each target video, each frame of the video image is input into the depth estimation model, which outputs a depth map for each frame. Each depth map includes the depth values ​​of each pixel in its corresponding video image. The depth maps of each frame of the target video form a depth map sequence. The video depth map sequence exhibits scale consistency and continuous depth variation over time. The depth values ​​of each video image and its corresponding depth map sequence are considered together. Figure 1 One-to-one correspondence.

[0052] Step S112: Based on each depth map sequence, determine the background depth map sequence and ground depth map series for each target video.

[0053] The background depth map sequence includes multiple frames of background depth maps, each frame of which only includes background elements and the depth values ​​corresponding to the background elements; the ground depth map sequence includes multiple frames of ground depth maps, each frame of which only includes ground elements and the depth values ​​corresponding to the ground elements.

[0054] For example, for each frame depth map in each depth map sequence, all elements except background elements are deleted from the depth map to obtain the background depth map corresponding to the depth map, and the background depth map sequence is composed of the background depth maps corresponding to each depth map; all elements except ground elements are deleted from the depth map to obtain the ground depth map corresponding to the depth map, and the ground depth map sequence is composed of the ground depth maps corresponding to each depth map.

[0055] Step S113: Obtain the vehicle's position pose.

[0056] Vehicle pose (camera pose) refers to the six-degree-of-freedom rigid body transformation of the vehicle's own coordinate system relative to a fixed world coordinate system when capturing video of any target. It describes the vehicle's position and orientation in three-dimensional space. Vehicle pose includes translation vectors and rotation matrices. The vehicle pose can be determined by selecting a target video from any viewpoint.

[0057] Step S114: Based on the vehicle's pose and the background depth map sequence and ground depth map series of each target video, determine the global background point cloud and the global ground point cloud.

[0058] Global background point cloud refers to a dense 3D point cloud that covers the entire scene and describes all background elements. Global ground point cloud refers to a dense 3D point cloud that describes ground elements such as road surfaces and drivable areas.

[0059] For each frame of background depth map, using the camera intrinsic parameters (focal length and principal point) corresponding to the background depth map, based on the two-dimensional pixel coordinates and depth values ​​of each pixel in the background depth map, each pixel in the background depth map is back-projected onto the camera coordinate system to obtain the first local background point cloud of the background depth map. Based on the rotation matrix and translation vector in the vehicle pose, and the three-dimensional coordinates of each point in the first local background point cloud in the camera coordinate system, the first local background point cloud is transformed to the world coordinate system through matrix multiplication to obtain the second local background point cloud. Then, the second local background point clouds of each background depth map are fused using the Volumetric Fusion algorithm to obtain a dense global background point cloud.

[0060] For each frame of ground depth map, using the corresponding camera intrinsic parameters (focal length and principal point), and based on the 2D pixel coordinates and depth values ​​of each pixel in the ground depth map, each pixel in the ground depth map is back-projected into the camera coordinate system to obtain the first local ground point cloud. Based on the rotation matrix and translation vector in the vehicle pose, and the 3D coordinates of each point in the first local ground point cloud in the camera coordinate system, matrix multiplication is used to transform each point in the first local ground point cloud to the world coordinate system to obtain the second local ground point cloud. Then, a volumetric element fusion algorithm is used to fuse the second local ground point clouds of each ground depth map to obtain a dense global ground point cloud. Outliers in the global background point cloud and the global ground point cloud can also be removed separately to optimize the global background point cloud and the global ground point cloud.

[0061] In this embodiment of the disclosure, depth information of the background and the ground is extracted and decoupled from the multi-view target video through depth estimation, thereby generating high-quality, highly consistent dense global background point cloud and global ground point cloud, providing accurate and complete static geometric priors for subsequent scene sub-node modeling, effectively improving the accuracy and efficiency of scene reconstruction.

[0062] Figure 3 This is a flowchart illustrating step S112 provided in an exemplary embodiment of this application. In some alternative embodiments, such as Figure 3 As shown, step S112 includes the following steps: Step S1121: For each target video, use the segmentation mask model to determine the sky mask, background mask, ground mask, and dynamic target mask for each frame of the target video.

[0063] The segmentation mask model is a deep learning model used for semantic segmentation and masking of images. For example, the Grounded SAM-2 model can be used for segmentation masking. The sky mask, background mask, ground mask, and dynamic target mask are all binary mask images.

[0064] In the sky mask, pixels belonging to the sky element are marked as valid, and the remaining pixels are marked as invalid. In the background mask, pixels belonging to the background element are marked as valid, and the remaining pixels are marked as invalid. In the ground mask, pixels belonging to the ground element are marked as valid, and the remaining pixels are marked as invalid. For dynamic target masks, when there is only a single dynamic target, the pixel values ​​covered by the dynamic target can be set to 1 to mark them as valid, and the remaining pixels are marked as invalid. When there are multiple dynamic targets in the scene, different non-zero identifiers can be assigned to each dynamic target as pixel values ​​to distinguish different dynamic target instances, and the pixels covered by each dynamic target are marked as valid, and the remaining pixels are marked as invalid. For example, for each target video, each frame of the target video is input into the segmentation mask model, and the segmentation mask model outputs the sky mask, background mask, and dynamic target mask for each frame of the video image.

[0065] Step S1122: Based on the sky mask and dynamic target mask of each frame of the target video, the depth map sequence of the target video is erased to obtain the scene depth map sequence of the target video.

[0066] Each frame of the scene depth map sequence is a depth map that retains only the depth values ​​of the areas corresponding to ground and background elements. In each frame of the scene depth map, the depth values ​​of the pixels covered by sky elements and dynamic objects are set to invalid.

[0067] In one implementation, for each frame depth map in the depth map sequence of the target video, based on the sky mask and dynamic target mask corresponding to the depth map, the union of all valid pixel coordinates in the sky mask and all valid pixel coordinates in the dynamic target mask is taken to obtain a set of pixels to be erased; the depth values ​​of each pixel in the depth map belonging to the set of pixel coordinates to be erased are set to invalid to erase the sky elements and dynamic targets, thus obtaining the scene depth map of that frame depth map; the scene depth maps of each frame depth map constitute the scene depth map sequence of the target video.

[0068] Step S1123: Based on the scene depth map sequence of the target video, determine the background depth map sequence and ground depth map series of the target video.

[0069] Specifically, the scene depth map sequence is copied into two copies. One copy of the scene depth map sequence is processed as follows: for each frame of the scene depth map, if a pixel in the scene depth map is marked as valid in the ground mask and its depth value in the scene depth map is valid, then the depth value is retained; otherwise, it is set as invalid, and the background depth map of the scene depth map is obtained. The above operation is performed on each frame of the scene depth map in sequence, and the background depth maps of each frame of the scene depth map constitute the background depth map sequence. Another scene depth map sequence is executed as follows: if a pixel in the scene depth map is marked as invalid in the ground mask, but its depth value in the scene depth map is valid, then the depth value is retained; otherwise, it is set as invalid, and the ground depth map of the scene depth map is obtained. The ground depth map sequence of the target video is composed of the ground depth maps of each frame of the scene depth map. The above operation is performed on each frame of the scene depth map in sequence, and the ground depth map sequence is composed of the ground depth maps of each frame of the scene depth map.

[0070] In this embodiment of the disclosure, the semantic segmentation-guided deep erasure and separation strategy effectively eliminates the interference of dynamic targets and the sky on the depth information of static scenes, ensuring the purity and geometric consistency of the obtained background depth map sequence and ground depth map sequence, and providing an accurate data foundation for the subsequent generation of high-quality, decoupled global background point cloud and global ground point cloud.

[0071] Figure 4 This is a flowchart illustrating step S113 provided in an exemplary embodiment of this application. In some alternative embodiments, such as Figure 4 As shown, step S113 includes the following steps: Step S1131: Based on the sky mask and dynamic target mask corresponding to each frame of the target video in the forward-looking view, the target video in the forward-looking view is erased to obtain the scene map sequence of the target video in the forward-looking view.

[0072] Among them, the target videos from multiple perspectives include at least the target video from the forward-looking perspective. Each frame of the scene map sequence is a color (RGB) image that retains only the ground and background elements.

[0073] In one implementation, for each frame of the target video from the forward-looking perspective, the following operations are performed: obtain the sky mask and dynamic target mask of the video image; take the union of all valid pixel coordinates in the sky mask and all valid pixel coordinates in the dynamic target mask to obtain the set of pixels to be erased; set the RGB values ​​of each pixel in the video image that belongs to the set of pixels to be erased to invalid values ​​(e.g., set to black or null) to erase the sky elements and dynamic targets, thereby obtaining the scene map of the video image. The scene maps of each frame of the video image constitute the scene map sequence of the target video from the forward-looking perspective.

[0074] Step S1132: Based on the scene map sequence and scene depth map sequence of the target video from the forward-looking perspective, the vehicle pose is obtained using a pose estimation model.

[0075] Among them, the pose estimation model is a deep learning model that can perform inter-frame motion estimation based on graph and depth information. For example, the pose estimation model can adopt the DROID-SLAM model.

[0076] In one implementation, the scene map sequence and the scene depth map sequence are input into the pose estimation model, which performs inter-frame pose calculation based on the visual and geometric features of the static scene and outputs the vehicle pose including the rotation matrix and translation vector.

[0077] In this embodiment of the disclosure, by using sky masking and dynamic target masking to perform dynamic target and sky region erasure processing on the target video from the forward view, the influence of motion interference on pose estimation is effectively eliminated, thereby enabling the acquisition of high-precision, scale-consistent vehicle pose from the video image.

[0078] Figure 5 This is a flowchart illustrating a four-dimensional scene construction method provided in another exemplary embodiment of this application. In some alternative implementations, such as... Figure 5 As shown, the following steps are included before step S112: Step S200: Obtain the scaling factor.

[0079] The scaling factor is a scaling coefficient used to unify video depth map sequences from different perspectives to the same global metric. For example, the scaling factor can be set according to actual business needs or based on experience.

[0080] In some alternative implementations, step S200 may include the following steps: S1. Using the camera extrinsic estimation model, the camera extrinsic parameters corresponding to each target video are optimized to obtain the optimized camera extrinsic parameters for each target video.

[0081] Optimizing camera extrinsics defines the camera-to-ego transformation, including optimizing the rotation matrix and translation vector. The camera extrinsic estimation model is a deep learning model used to optimize the camera extrinsics; for example, it may employ the MASt3R model. The MASt3R model is an end-to-end feedforward vision model based on the Transformer architecture.

[0082] In one example, each target video is input into the camera extrinsic estimation model, which then outputs the optimized camera extrinsic parameters for each target video.

[0083] S2, perform preset image processing on each frame of the video image in each target video to obtain the target image sequence of each target video.

[0084] The preset image processing may include scaling and cropping. Each target image sequence includes multiple frames and video. Figure 1 A corresponding target image.

[0085] For example, each frame of video image in each target video is scaled according to a preset ratio so that the short side of each frame of video image is 512 pixels. Then, for each frame of video image, a 512 x 512 pixel square area is cropped with the principal point (i.e., optical center) of the video image as the center to obtain the target image of the video image. The target image sequence of the target video is constructed from the target images corresponding to each frame of video image in the target video.

[0086] S3. For each target video, determine the intermediate point cloud sequence based on the camera intrinsic parameters and depth map sequence of the target video.

[0087] For each depth map in the depth map sequence, the pixel coordinates of each pixel in the depth map are ( , ), depth value ( ) and camera intrinsic parameters, and project each pixel onto the camera coordinate system using back projection formulas (1)-(3) to obtain the coordinate values ​​of each point in the intermediate point cloud in the camera coordinate system. , , The camera's intrinsic parameters include focal length ( , ) and principal point ( , ).

[0088] Formula (1) Formula (2) Formula (3) S4, based on optimized camera extrinsic parameters, backprojects the intermediate point cloud sequence onto the target image sequence to obtain target depth information.

[0089] Among them, the new camera intrinsic parameters corresponding to the target depth map sequence are calculated using formulas (4)-(9). The new camera intrinsic parameters include the new focal length ( , ) and the new master point ( , In formulas (4)-(9), and These are the height and width of the target image, respectively. and The height and width of the video image in the target video corresponding to the target image sequence.

[0090] Formula (4) Formula (5) Formula (6) Formula (7) Formula (8) Formula (9) For each intermediate point cloud in the intermediate point cloud sequence, based on formula (10), each point in the intermediate point cloud is transformed into the vehicle coordinate system, and then reprojected onto the target map using projection formulas (11)-(13). The depth values ​​corresponding to each point reprojected onto the target map are sampled, and the depth values ​​of each point in the target map constitute the target depth information.

[0091] Formula (10) Formula (11) Formula (12) Formula (13) In formulas (10)-(13), ( , , () represents the coordinates of the intermediate point cloud in the vehicle coordinate system. To optimize the rotation matrix, To optimize the transpose of the rotation matrix, To optimize the translation vector, ( , Let be the coordinates of any point in the midpoint cloud in the target image. Let be the depth value of any point in the midpoint cloud in the target image.

[0092] S5. Based on the target depth information and the depth map sequence of the target video, determine the scale factor.

[0093] In this process, the depth values ​​corresponding to the pixels in the road area of ​​the depth map sequence are determined as the source depth information, and the size factor is determined using formula (14).

[0094] Formula (14) In formula (14), For size factor, For target depth information, For source depth information, This indicates taking the median.

[0095] Step S210: Based on the scale factor, the dimensions of each depth map sequence and each target video are unified to obtain each updated depth map sequence and each updated video map sequence, so as to determine the background depth map sequence and ground depth map series of each target video based on each updated depth map sequence and each updated video map sequence.

[0096] Specifically, the scale factor is multiplied by all pixels in each depth map of each depth map sequence to unify the size of each depth map sequence, thereby obtaining the updated depth map sequence corresponding to each depth map sequence. Then, step S112 is performed based on each updated depth map sequence.

[0097] In one implementation, the scale factor can be multiplied by all pixels in each video graph of each target video to unify the size of each target video and obtain the updated video graph sequence corresponding to each target video.

[0098] In this embodiment of the disclosure, by introducing a scale factor to unify the global scale of the depth maps from different perspectives, the measurement differences in depth estimation between different cameras are effectively eliminated, ensuring the geometric consistency of multi-view depth data in the world coordinate system, and providing a reliable data foundation for the subsequent accurate separation of background and ground depth maps and the seamless fusion of dense point clouds.

[0099] Figure 6 This is a flowchart illustrating step S130 provided in an exemplary embodiment of this application. In some alternative embodiments, such as Figure 6 As shown, step S130 includes the following steps: Step S131: Use the position sequence in the Gaussian parameters of the point cloud sequence to fit the B-spline curve of the dynamic target.

[0100] B-spline curves are a form of curve representation in numerical analysis, representing a linear combination of B-spline basis functions. The B-spline curves of a dynamic target are used to describe the time-varying displacement function of the target's local motion. The position sequence of each dynamic target includes the position information of each Gaussian ellipsoid corresponding to each point cloud of that dynamic target. .

[0101] Set the order of the B-spline The B-spline curve of the dynamic target at any time is: , , For the first indivual B-spline basis functions, the B-spline basis functions are: when hour, ;when hour, ; The curve parameters are normalized to the [0,1] interval, and the corresponding time is the time of the dynamic target. , , ..., Let be the node vector on the B-spline curve, which is a non-decreasing sequence of real numbers. This represents the number of control points on the B-spline curve.

[0102] Step S132: Based on the B-spline curve, determine the motion information using the motion model parameters.

[0103] Among them, motion information includes new position information and motion model of dynamic target. The motion model can be determined by using the B-spline curve of dynamic target and preset motion model parameters. The motion model corresponds to formula (15). New position information can be determined based on the motion model (formula (15)).

[0104] Formula (15) In formula (15), and Let be the order of the trigonometric function. , The value of is 1 to 6. and These are the motion model parameters.

[0105] In this embodiment, B-spline curves are used to model the discrete position sequence as a continuous model, and motion model parameters are introduced to generate smooth new position information, which significantly improves the fitting accuracy and prediction stability of the dynamic target motion trajectory and avoids the jitter and noise interference caused by frame-by-frame position estimation.

[0106] Figure 7 This is a flowchart illustrating step S140 provided in an exemplary embodiment of this application. In some alternative embodiments, such as Figure 7 As shown, step S140 includes the following steps: Step S141: Obtain truth information.

[0107] The truth information includes at least a truth color image. For example, any video image can be selected as the truth color image.

[0108] Step S142: Render based on each Gaussian parameter to obtain rendering information.

[0109] The rendering information includes at least a rendered color image. The rendered color image is an RGB two-dimensional image. It can be rendered using image rendering tools or based on various Gaussian parameters. - Rendering methods such as alpha blending algorithms are used to render a color image. Alternatively, based on the positional information in each Gaussian parameter and the current camera's intrinsic and extrinsic parameters (rotation matrix, translation vector), all Gaussian ellipsoids corresponding to the global background point cloud, global ground point cloud, environment texture map, and dynamic target point cloud sequence are projected onto a two-dimensional plane to determine the pixel area and depth value covered by each Gaussian ellipsoid; the current camera intrinsic and extrinsic parameters are the camera intrinsic and extrinsic parameters corresponding to any target video. All Gaussian ellipsoids projected to the same pixel position are sorted from near to far according to their depth values, and then processed in depth order using... The AlphaBlending algorithm blends all Gaussian ellipsoids to obtain a rendered color image.

[0110] Step S143: Based on the rendering information and the ground truth information, determine the total loss value between the rendering information and the ground truth information.

[0111] Step S144: Iterate the Gaussian parameters and motion model parameters a preset number of times based on the total loss value, and construct a four-dimensional scene model based on the environment texture map and the Gaussian parameters and motion information obtained in the last iteration.

[0112] The process involves calculating the total loss between rendered and ground truth information. Then, backpropagation is performed using a 3D Gaussian splashing differentiable rasterization rendering pipeline to calculate the total loss with respect to the parameters to be optimized (Gaussianization parameters and motion model coefficients). An optimizer (e.g., Adam) then iteratively updates the parameters along the gradient descent direction based on the calculated gradient. When updating the Gaussian parameters of ground elements, their position information can be frozen, and only opacity, scale, and rotation information are optimized. This iterative optimization process is executed a preset number of times (e.g., 30,000 times), and Gaussian parameters are pruned and densed during the optimization process to dynamically adjust the Gaussian density and distribution. After iteration, a four-dimensional scene model is constructed based on the final optimized Gaussian parameters and motion model parameters of each node, as well as the environment texture map.

[0113] In this embodiment of the disclosure, by constructing a self-supervised optimization closed loop of "rendering-comparison-iteration", driven by the multi-dimensional loss of rendering information and ground truth information, the end-to-end joint iterative solution is performed on all learnable parameters corresponding to scene geometry, dynamic target motion and sky appearance. High-fidelity 4D city scene reconstruction with consistent geometry and motion can be achieved without any manual annotation.

[0114] In some alternative implementations, step S143 in this embodiment of the present disclosure may include: determining multidimensional loss information based on rendering information and ground truth information, and then determining the total loss value based on the multidimensional loss information.

[0115] The multi-dimensional loss information includes at least one of the following: photometric loss value, dynamic target mask loss value, sky mask loss value, scale translation depth loss value, local rigidity regularization loss value, sharpening shape regularization loss value, background opacity loss value, and ground maximum scale regularization loss value.

[0116] The ground truth information includes: ground truth color image, ground truth depth image, ground truth dynamic target mask, ground truth sky mask, and ground truth background mask. The rendering information includes: rendering color image, rendering depth image, rendering dynamic target mask, rendering sky mask, and rendering background mask.

[0117] In one example, any depth map can be selected as the ground truth depth map, any sky mask as the ground truth sky mask, any background mask as the ground truth background mask, any ground mask as the ground truth ground mask, and any dynamic target mask as the ground truth dynamic target mask.

[0118] The rendered depth map includes the depth values ​​of each pixel. The rendered sky mask, rendered background mask, rendered ground mask, and rendered dynamic target mask are all binary mask images. The rendered sky mask includes only sky elements, the rendered background mask includes only background elements, the rendered ground mask includes only ground elements, and the rendered dynamic target mask includes only dynamic targets.

[0119] Based on the Gaussian parameters and the depth values ​​of all point clouds, image rendering tools or - Rendering methods such as AlphaBlending are used to obtain a rendered depth map, a rendered sky mask, a rendered background mask, a rendered ground mask, and a rendered dynamic target mask. Alternatively, the rendered color image can be input into a segmentation mask model to obtain the rendered sky mask, rendered background mask, rendered ground mask, and rendered dynamic target mask. The rendered color image can also be input into a depth estimation model to obtain a rendered depth map.

[0120] Photometric loss is used to constrain the pixel-level color and structure similarity between the rendered color image and the ground truth color image. Specifically, the photometric loss value can be determined based on formula (16). In formula (16), A color chart representing the truth value. This indicates rendering a color image. and Represents the weight parameters ( , ), This means calculating the absolute difference of the RGB (red, green, blue) channels pixel by pixel between the rendered color image and the true color image, and then averaging the values ​​over all pixels; This means dividing the ground truth color image and the rendered color image into multiple local windows (e.g., 11x11), calculating the similarity of brightness (mean), contrast (variance), and structure (covariance) within the windows in the ground truth color image and the rendered color image respectively, and finally obtaining a comprehensive score between 0 and 1 (1 indicates that they are completely identical).

[0121] Formula (16) The dynamic target mask loss and sky mask loss are used to constrain the rendering masks of dynamic targets and sky regions, respectively, to ensure consistency with the pre-segmented semantic masks. The dynamic target mask loss value can be determined based on formula (17). The sky mask loss value can be determined based on formula (18). .

[0122] Formula (17) Formula (18) In formulas (17) and (18), Indicates rendering a dynamic target mask. Represents a truth-based dynamic target mask. Indicates the rendering sky mask. Indicates the truth value sky mask. This represents the binary cross-entropy function, which calculates the cross-entropy loss pixel by pixel.

[0123] The scale-translation depth loss is applied to the inverse of the rendered depth map to align the scale with the translation, thus matching the video depth map as a geometrically supervised image; the scale-translation depth loss value can be calculated based on formula (19). .

[0124] Formula (19) In formula (19), This represents the depth value in the rendered depth map. This represents the depth value in the truth depth map. Indicates dimensional parameters, Indicates the translation parameters, collects data pairs of all pixels ( , The optimal solution is obtained through least squares linear regression. and .

[0125] The local stiffness regularization loss is used to select spatially adjacent Gaussians through K-nearest neighbors (KNN) to constrain the consistency of their displacement vectors, thereby maintaining the local stiffness of the dynamic target. The local stiffness regularization loss value can be calculated based on formula (20). ).

[0126] Formula (20) In formula (20), Represents the square of the L2 norm. Expressing the location information of dynamic targets. for The location information of adjacent Gaussian points, Indicates the use of The new location information is calculated based on formula (15). Indicates the use of The new location information is calculated based on formula (15). Represents the location information of dynamic targets Since the location information changes continuously during the iteration process, the KNN index can be recalculated and cached every 100 iterations to balance computational load and accuracy.

[0127] The sharpening shape regularization loss is used to limit the scale ratio of each axis of the Gaussian ellipsoid, suppressing excessively elongated Gaussians, thereby improving the sharpness of the scene's geometric surface (different scale thresholds and penalty weights are set for background nodes and ground nodes). The sharpening shape regularization loss value can be calculated based on formula (21). .

[0128] Formula (21) In formula (21), This represents the size information, specifically the three-dimensional vectors of each Gaussian ellipsoid. The tolerance threshold for the expression ratio can be set according to actual needs, for example, it can be 2 or 10.

[0129] Background opacity loss is used to constrain the cumulative opacity mask of the rendered background, making it approximate the true background mask derived from the sky and ground masks, thereby preventing Gaussian overflow of background elements into the ground region. The background opacity loss value can be calculated based on formula (22). .

[0130] Formula (22) In formula (22), Indicates the background mask for rendering. This represents the truth background mask.

[0131] The ground maximum scale regularization loss is used to constrain the squared value of the maximum scale component in each Gaussian parameter of the ground source, forcing the Gaussians of ground elements to maintain a small size to ensure the rendering clarity of road surface textures such as lane lines from the new perspective. The ground maximum scale regularization loss value is calculated based on formula (23). Formula (23) The total loss value L can be calculated based on formula (24).

[0132] Formula (24) In this embodiment of the disclosure, by constructing a loss function system covering multiple dimensions such as color, geometry, semantics and motion, the reconstruction results are comprehensively constrained from multiple levels such as pixel-level appearance, scene structure, dynamic target deformation and geometric consistency of new perspectives. This enables the gradient signals of each loss term to collaboratively drive the end-to-end joint optimization of all learnable parameters such as background, ground, dynamic targets and sky, thereby achieving high-fidelity, geometrically robust and motion-coherent 4D urban scene modeling.

[0133] Figure 8 This is a schematic diagram of the four-dimensional scene construction method provided in one application implementation of this application. For example... Figure 8 As shown, package: Phase 1: Video Reconstruction Pipeline Input: Target video from multiple perspectives.

[0134] Parallel processing: Each target video is fed into the Grounded SAM 2 model (segmentation mask model), which outputs segmentation mask and object tracking trajectory information. The segmentation mask and object tracking trajectory include: sky mask, background mask, ground mask and dynamic target mask of each frame of video image in each target video. Each target video is fed into the Video Depth Anything model (depth estimation model) to generate a sequence of depth maps (video depth), and the scene map sequence and scene depth map sequence of the target video from the forward view are fed into the DROID-SLAM model (pose estimation model) to estimate the vehicle pose (cam pose).

[0135] Phase Two: Gaussian Scene Graph The extracted geometric and semantic information is fused to construct a Gaussian SceneGraph.

[0136] The scene graph is decoupled into three independent node types to leverage different training strategies: Ground Node: Represents ground elements and is used to construct the Gaussian parameters of the global ground point cloud; Background Node: Represents the background element and is used to construct the Gaussian parameters of the global background point cloud; Object Node: Represents a dynamic target and is used to construct the Gaussian parameters of a dynamic target point cloud sequence; Modeling strategies for different nodes: For ground nodes and background nodes, standard Gaussian is used directly for representation and optimization.

[0137] For object nodes: The Gaussian Marble strategy (isotropic, SH order 0) is adopted.

[0138] Input into the Gaussian Deformation module (i.e., B-spline + motion model).

[0139] Output the deformed Gaussian to obtain new position information.

[0140] All nodes converge to the Efficient rendering module (referring to 3DGS's fast differentiable rasterization).

[0141] Finally, the rendered image is produced.

[0142] Figure 9 This is a structural block diagram of a four-dimensional scene construction apparatus provided in an exemplary embodiment of this disclosure. Figure 9 As shown, the four-dimensional scene construction device includes: The video acquisition module 300 is used to acquire target videos from multiple perspectives. The target video from any perspective includes: dynamic targets, sky elements, ground elements, and background elements. The point cloud acquisition module 310 is used to determine the global background point cloud, global ground point cloud, environmental texture map and dynamic target point cloud sequence based on each target video; Gaussian creation module 320 is used to obtain the Gaussian parameters corresponding to the global background point cloud, the global ground point cloud and the dynamic target point cloud sequence, respectively. The motion model creation module 330 is used to determine the motion information of the dynamic target based on the Gaussian parameters of the dynamic target point cloud sequence; The scene creation module 340 is used to construct a four-dimensional scene model based on the Gaussian parameters, the motion information, and the environmental texture map.

[0143] In some optional implementations, the point cloud acquisition module 310 is specifically used to determine the depth map sequence of each target video based on each target video using a depth estimation model; determine the background depth map sequence and ground depth map series of each target video based on each depth map sequence; acquire the vehicle pose; and determine the global background point cloud and the global ground point cloud based on the vehicle pose and the background depth map sequence and ground depth map series of each target video.

[0144] In some optional implementations, based on each depth map sequence, a background depth map sequence and a ground depth map series for each target video are determined, further used for: For each target video, a segmentation mask model is used to determine the sky mask, background mask, ground mask, and dynamic target mask for each frame of the target video. Based on the sky mask and dynamic target mask of each frame of the video, the depth map sequence is erased to obtain the scene depth map sequence of the target video. Based on the scene depth map sequence of the target video, the background depth map sequence and the ground depth map sequence of the target video are determined.

[0145] In some alternative implementations, the acquisition of the vehicle position pose is further used for: Based on the sky mask and dynamic target mask corresponding to each frame of the target video from the forward-looking perspective, the target video from the forward-looking perspective is erased to obtain the scene map sequence of the target video from the forward-looking perspective. Based on the scene map sequence and scene depth map sequence of the target video from the forward-looking perspective, the vehicle pose is obtained using a pose estimation model.

[0146] In some optional implementations, the four-dimensional scene construction device in this embodiment further includes: Size factor acquisition module, used to acquire scale factor; The size unification module is used to unify the size of each depth map sequence based on the scale factor to obtain each updated depth map sequence, so as to determine the background depth map sequence and ground depth map series of each target video based on each updated depth map sequence.

[0147] In some optional implementations, the size factor acquisition module is specifically used to optimize the camera extrinsics corresponding to each target video using a camera extrinsic estimation model to obtain optimized camera extrinsics for each target video; perform preset image processing on each frame of video image in each target video to obtain a target image sequence for each target video; for each target video, determine an intermediate point cloud sequence based on the camera intrinsic parameters and depth map sequence of the target video; backproject the intermediate point cloud sequence onto the target image sequence based on the optimized camera extrinsics to obtain target depth information; and determine the scale factor based on the target depth information and the depth map sequence.

[0148] In some alternative implementations, the motion model creation module 330 is specifically used to fit the B-spline curve of the dynamic target using the position sequence in the Gaussian parameters of the point cloud sequence; and to determine the motion information based on the B-spline curve using motion model parameters, wherein the motion information includes the new position information and motion model of the dynamic target.

[0149] In some optional implementations, the scene creation module 340 is specifically used to obtain ground truth information; to render based on each Gaussian parameter to obtain rendering information; to determine the total loss value between the rendering information and the ground truth information based on the rendering information and the ground truth information; to iterate the Gaussian parameters and motion model parameters a preset number of times based on the total loss value; and to construct the four-dimensional scene model based on the environment texture map and the Gaussian parameters and motion information obtained in the last iteration.

[0150] In some optional implementations, the truth information includes: a truth color image, a truth depth image, a truth dynamic target mask, a truth sky mask, and a truth background mask; the rendering information includes: a rendering color image, a rendering depth image, a rendering dynamic target mask, a rendering sky mask, and a rendering background mask. The step of determining the loss information between the rendering information and the ground truth information based on the rendering information and the ground truth information is further used for: Based on the rendering information and the ground truth information, multidimensional loss information is determined, which includes at least one of the following: photometric loss value, dynamic target mask loss value, sky mask loss value, scale translation depth loss value, local rigidity regularization loss value, sharpening shape regularization loss value, background opacity loss value, and ground maximum scale regularization loss value; based on the multidimensional loss information, the total loss value is determined.

[0151] The four-dimensional scene construction device of this disclosure corresponds to the four-dimensional scene construction method described above, and the relevant contents can be referred to each other, which will not be repeated here.

[0152] The beneficial technical effects of the exemplary embodiments of the four-dimensional scene construction device disclosed herein can be found in the corresponding beneficial technical effects of the exemplary methods and systems described above, and will not be repeated here.

[0153] In addition, this disclosure also provides an electronic device, including: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the four-dimensional scene construction method described in any of the above embodiments of the present disclosure.

[0154] Figure 10 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 10 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0155] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0156] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0157] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the four-dimensional scene construction methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0158] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0159] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0160] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0161] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0162] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the four-dimensional scene construction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0163] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0164] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the four-dimensional scene construction methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0165] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0166] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0167] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0168] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0169] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0170] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0171] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0172] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0173] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for constructing a four-dimensional scene, characterized in that, include: Acquire target videos from multiple perspectives. Target videos from any perspective include: dynamic targets, sky elements, ground elements, and background elements. Based on each target video, the global background point cloud, global ground point cloud, environmental texture map, and dynamic target point cloud sequence are determined. Obtain the Gaussian parameters corresponding to the global background point cloud, the global ground point cloud, and the dynamic target point cloud sequence, respectively; Based on the Gaussian parameters of the dynamic target point cloud sequence, the motion information of the dynamic target is determined; A four-dimensional scene model is constructed based on the Gaussian parameters, the motion information, and the environmental texture map.

2. The method according to claim 1, characterized in that, Based on the target videos, the global background point cloud and the global ground point cloud are determined, including: Based on the target videos, a depth map sequence for each target video is determined using a depth estimation model; Based on each depth map sequence, the background depth map sequence and ground depth map series of each target video are determined; Obtained from vehicle position; Based on the vehicle's pose, and the background depth map sequence and ground depth map series of each target video, the global background point cloud and the global ground point cloud are determined.

3. The method according to claim 2, characterized in that, The process of determining the background depth map sequence and ground depth map series for each target video based on each depth map sequence includes: For each target video, the sky mask, background mask, ground mask, and dynamic target mask of each frame of the target video are determined using a segmentation mask model. Based on the sky mask and dynamic target mask of each video frame, the depth map sequence is erased to obtain the scene depth map sequence of the target video. Based on the scene depth map sequence of the target video, the background depth map sequence and the ground depth map series of the target video are determined.

4. The method according to claim 3, characterized in that, The acquisition of the vehicle position pose includes: Based on the sky mask and dynamic target mask corresponding to each frame of the target video from the forward-looking perspective, the target video from the forward-looking perspective is erased to obtain the scene map sequence of the target video from the forward-looking perspective. Based on the scene map sequence and scene depth map sequence of the target video from the forward-looking perspective, the vehicle pose is obtained using a pose estimation model.

5. The method according to claim 2, characterized in that, After determining the depth map sequence of each target video using a depth estimation model, the process further includes: Obtain the scaling factor; Based on the scale factor, the dimensions of each depth map sequence are unified to obtain each updated depth map sequence, so as to determine the background depth map sequence and ground depth map series of each target video based on each updated depth map sequence.

6. The method according to claim 5, characterized in that, The acquisition of the scale factor includes: Using a camera extrinsic estimation model, the camera extrinsic parameters corresponding to each target video are optimized to obtain the optimized camera extrinsic parameters for each target video; Perform preset image processing on each frame of the target video to obtain a sequence of target images for each target video. For each target video, an intermediate point cloud sequence is determined based on the camera intrinsic parameters and depth map sequence of the target video; Based on the optimized camera extrinsic parameters, the intermediate point cloud sequence is back-projected onto the target image sequence to obtain target depth information; The scale factor is determined based on the target depth information and the depth map sequence.

7. The method according to claim 1, characterized in that, Determining the motion information of the dynamic target based on the Gaussian parameters of the dynamic target point cloud sequence includes: Using the position sequence in the Gaussian parameters of the point cloud sequence, fit the B-spline curve of the dynamic target; Based on the B-spline curve, the motion information is determined using motion model parameters. The motion information includes the new position information and motion model of the dynamic target.

8. The method according to claim 1, characterized in that, The construction of a four-dimensional scene model based on the Gaussian parameters, the motion information, and the environmental texture map includes: Obtain truth information; Rendering is performed based on various Gaussian parameters to obtain rendering information; Based on the rendering information and the truth information, determine the total loss value between the rendering information and the truth information; Based on the total loss value, the Gaussian parameters and motion model parameters are iterated a preset number of times, and the four-dimensional scene model is constructed based on the environment texture map and the Gaussian parameters and motion information obtained in the last iteration.

9. The method according to claim 8, characterized in that, The ground truth information includes: ground truth color image, ground truth depth image, ground truth dynamic target mask, ground truth sky mask, and ground truth background mask; the rendering information includes: rendering color image, rendering depth image, rendering dynamic target mask, rendering sky mask, and rendering background mask; The step of determining the loss information between the rendering information and the ground truth information based on the rendering information and the ground truth information includes: Based on the rendering information and the ground truth information, multidimensional loss information is determined, which includes at least one of the following: photometric loss value, dynamic target mask loss value, sky mask loss value, scale translation depth loss value, local rigidity regularization loss value, sharpening shape regularization loss value, background opacity loss value, and ground maximum scale regularization loss value. Based on the multi-dimensional loss information, the total loss value is determined.

10. A four-dimensional scene construction device, characterized in that, include: The video acquisition module is used to acquire target videos from multiple perspectives. Target videos from any perspective include: dynamic targets, sky elements, ground elements, and background elements. The point cloud acquisition module is used to determine the global background point cloud, global ground point cloud, environmental texture map, and dynamic target point cloud sequence based on each target video. The Gaussian creation module is used to obtain the Gaussian parameters corresponding to the global background point cloud, the global ground point cloud, and the dynamic target point cloud sequence, respectively. The motion model creation module is used to determine the motion information of the dynamic target based on the Gaussian parameters of the dynamic target point cloud sequence. The scene creation module is used to construct a four-dimensional scene model based on the Gaussian parameters, the motion information, and the environmental texture map.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-9.

13. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-9.