Spatial scene video generation method, apparatus, equipment and storage medium
By using a spatial scene video generation method based on 3D Gaussian point cloud data, combined with reference images and camera trajectory rendering features, the problem of insufficient spatial consistency and detail representation in existing technologies is solved, and efficient, high-quality 3D scene video generation is achieved.
Patent Information
- Application Number
- CN202611587711.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-13
- Publication Date
- 2026-11-10
AI Technical Summary
Existing scene generation methods based on video modeling are difficult to guarantee spatial consistency, while generation methods based on Gaussian representation are limited in representation ability and training data scale, and lack detail representation.
A spatial scene video generation method based on 3D Gaussian point cloud data is adopted. By acquiring reference images of the target 3D scene and preset camera trajectories, a Gaussian representation of the 3D scene is generated. Color and depth image sequences are rendered along the preset camera trajectory. Generative denoising is performed by combining trajectory-guided features and scene content-guided features to generate a target 3D scene video.
It achieves spatial and visual consistency in the generation of 3D scene videos, improving generation efficiency and quality, and avoiding the accumulation of errors in iterative generation and the high computational cost of traditional frame-by-frame modeling.
Smart Images

Figure CN123075243A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for generating spatial scene videos. Background Technology
[0002] With the rapid development of robotics and the emergence of the concept of embodied intelligence, world models, as the data generation engine supporting these technologies, have become a core technology of common concern in academia and industry. An ideal world model should be able to generate realistic scenes that conform to common sense and physical laws, providing high-quality data for the training of robots and embodied intelligence models. However, existing scene generation methods based on video modeling are limited by the model framework and struggle to guarantee spatial consistency, while generation methods based on Gaussian representations are limited in representational ability and training data scale; although they offer good consistency, they lack detailed representation. Summary of the Invention
[0003] This disclosure provides a method, apparatus, device, and storage medium for generating spatial scene videos to solve or alleviate one or more technical problems in the prior art.
[0004] Firstly, this disclosure provides a method for generating spatial scene videos based on 3D Gaussian point cloud data, including: Acquire reference images corresponding to the target 3D scene, and acquire preset camera trajectories; Based on the reference image, a Gaussian representation of the target 3D scene is generated, wherein the Gaussian representation of the 3D scene includes multiple 3D Gaussian volumes that characterize the geometric information and / or appearance information of the target 3D scene; Based on the preset camera trajectory, the Gaussian representation of the 3D scene is rendered to obtain the color image sequence and depth image sequence under the preset camera trajectory; Color image sequences and depth image sequences are encoded into a compressed latent space to obtain color latent features and depth latent features, respectively. A reference image is encoded into the compressed latent space to obtain scene content guidance features. The color latent features and the depth latent features are concatenated to obtain the trajectory guidance features; Based on the trajectory guidance features and the scene content guidance features, generative denoising processing is performed on the initial noisy video data to obtain a target 3D scene video, wherein the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
[0005] Secondly, this disclosure provides a spatial scene video generation device based on 3D Gaussian point cloud data, including: The acquisition unit is used to acquire a reference image corresponding to the target 3D scene and to acquire a preset camera trajectory; A 3D Gaussian generation unit is used to generate a 3D Gaussian representation of the target 3D scene based on the reference image. The 3D Gaussian representation of the target 3D scene includes multiple 3D Gaussian volumes that characterize the geometric information and / or appearance information of the target 3D scene. A spatial video generation unit is used to render the Gaussian representation of the 3D scene based on the preset camera trajectory to obtain a color image sequence and a depth image sequence under the preset camera trajectory; encode the color image sequence and the depth image sequence into a compressed latent space to obtain color latent features and depth latent features respectively, and encode a reference image into the compressed latent space to obtain scene content guidance features; concatenate the color latent features and the depth latent features to obtain trajectory guidance features; and perform generative denoising processing on the initial noisy video data based on the trajectory guidance features and the scene content guidance features to obtain a target 3D scene video, wherein the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
[0006] Thirdly, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.
[0007] Fourthly, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of the present disclosure.
[0008] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of the present disclosure.
[0009] In this way, the disclosed solution effectively avoids the error accumulation caused by iterative generation by generating a Gaussian representation of the 3D scene through a coarse-grained scene generation stage and combining the reference image of the target 3D scene to obtain a spatially consistent 3D scene. Furthermore, the 3D scene Gaussian representation is rendered along a preset camera trajectory to efficiently obtain continuous color image sequences and depth image sequences, avoiding the high computational cost of traditional frame-by-frame modeling and significantly improving the efficiency and real-time performance of video generation. Finally, trajectory guidance features and scene content guidance features are generated by combining the reference image and the rendered color image sequence and depth image sequence. Generative denoising processing is then performed on the initial noisy video data by combining the trajectory guidance features and scene content guidance features. In this way, the generated 3D scene video can accurately match the scene content of the reference image and the perspective changes of the preset camera trajectory, thereby achieving visual consistency and dynamic smoothness of the target 3D scene video, thus improving the generation efficiency and quality of the scene video.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments provided according to this disclosure and should not be construed as limiting the scope of this disclosure.
[0012] Figure 1 This is an illustrative flowchart of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application. Figure 1 ; Figure 2 This is an illustrative flowchart of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application. Figure 2 ; Figure 3 This is an illustrative flowchart of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application. Figure 3 ; Figure 4 This is a schematic diagram of the process implementation of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application in a specific example; Figure 5 This is a schematic diagram of a spatial scene video generation device based on 3D Gaussian point cloud data according to an embodiment of this application; Figure 6This is a block diagram of an electronic device used to implement the spatial scene video generation method based on 3D Gaussian point cloud data according to the embodiments of this disclosure. Detailed Implementation
[0013] The present disclosure will now be described in further detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.
[0014] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.
[0015] This disclosure provides a method for generating spatial scene videos based on 3D Gaussian point cloud data. The method includes a 3D scene Gaussian representation generation stage and a scene video generation stage. Through these two stages, and using a given camera trajectory and a single scene image (i.e., a reference image corresponding to the target 3D scene) as the input for scene video generation (the first frame of the scene video to be generated as the output), a 3D scene video conforming to the camera trajectory and the input scene image is generated. Specifically, in the 3D scene Gaussian representation generation stage (also called the scene generation stage), the method generates a 3D scene Gaussian representation corresponding to the target 3D scene based on the input scene image. In the scene video generation stage, the 3D scene Gaussian representation is first rendered based on the given camera trajectory to obtain a coarse-grained color image sequence and a depth image sequence under the camera trajectory. Based on this, trajectory guidance features for generating the spatial scene video of the target 3D scene are determined, as well as scene content guidance features obtained from the reference image. Then, based on the trajectory guidance features and scene content guidance features, generative denoising processing is performed on the initial noisy video data to obtain a high-fidelity target 3D scene video. The above process directly generates point clouds of the 3D scene through a coarse-grained scene generation stage, effectively avoiding the error accumulation problem caused by iterative generation of 3D scenes. In this way, spatial consistency of the 3D scene is achieved. At the same time, combined with condition-guided generative denoising, the detail performance of the scene video is further improved, and high-quality and consistent 3D scene video generation is achieved.
[0016] Specifically, Figure 1 This is an illustrative flowchart of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application. Figure 1 This method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.
[0017] Furthermore, the method includes at least a portion of the following: For example... Figure 1 As shown, it includes: Step S101: Obtain the reference image corresponding to the target 3D scene, and obtain the preset camera trajectory.
[0018] Here, the target 3D scene can specifically be one of the following: indoor scene, outdoor scene, road scene, building scene, natural environment scene, and other scenes with spatial structure, from which the spatial scene video to be generated.
[0019] Furthermore, the reference image may specifically be a scene image of the target 3D scene observed from a reference viewpoint, and the reference image contains at least the color, texture, object layout, and part of the spatial structure of the target 3D scene.
[0020] In one example, the reference image can also serve as a content reference for generating the target 3D scene video. In other words, it is used to constrain the construction of the subsequent Gaussian representation of the 3D scene and the generation of the target 3D scene video, so that the generated scene content and video content are consistent with the scene content presented in the reference image.
[0021] Here, the preset camera trajectory can specifically represent the motion path of the virtual camera in the target 3D scene. Further, the preset camera trajectory can specifically include multiple camera poses (or multiple viewing angles) arranged in chronological order, and each camera pose can correspond to a different moment (or different video frame) in the desired generated 3D scene video. In other words, there is a correspondence between the moment (or video frame) in the desired generated target 3D scene video and the camera pose in the preset camera trajectory, for example, a one-to-one correspondence. In practical applications, this can be set based on the actual scene requirements, and this disclosure does not impose specific limitations on this.
[0022] In other words, the preset camera trajectory contains the camera pose (or viewing angle) corresponding to each moment (or each video frame) in the desired 3D scene video. This provides a viewpoint basis for the generation of subsequent video frames in the 3D scene video.
[0023] Furthermore, the camera pose may specifically include at least one of the virtual camera's position and orientation.
[0024] Thus, the present invention can control the observation angle corresponding to each video frame in the desired generated 3D scene video by setting the camera pose in the preset camera trajectory.
[0025] Step S102: Based on the reference image, generate a Gaussian representation of the target 3D scene.
[0026] Here, the Gaussian representation of the three-dimensional scene includes multiple three-dimensional Gaussian volumes that characterize the geometric and / or appearance information of the target three-dimensional scene.
[0027] It should be noted that a 3D Gaussian volume can be specifically understood as a parameterized 3D basic unit (such as an anisotropic 3D Gaussian volume) used to describe a local spatial region in a target 3D scene. A 3D Gaussian volume can include parameters characterizing its position, spatial extent, shape, appearance, and transparency in 3D space. For example, the parameters of a 3D Gaussian volume can include at least one of the following: center position parameter, covariance parameter, color parameter, and transparency parameter. The center position parameter characterizes the spatial position of the 3D Gaussian volume in 3D space; the covariance parameter characterizes the spatial shape and scale of the 3D Gaussian volume; the color parameter can be represented by a spherical harmonic function to describe the degree of color change of the 3D Gaussian volume under different viewing angles; and the transparency parameter characterizes the contribution of the 3D Gaussian volume to the visibility or occlusion of pixels during rendering.
[0028] Furthermore, the plurality of three-dimensional Gaussian bodies characterizing the geometric information and / or appearance information of the target three-dimensional scene may specifically include at least two of the following: a three-dimensional Gaussian body characterizing the geometric information of the target three-dimensional scene, a three-dimensional Gaussian body characterizing the appearance information of the target three-dimensional scene, and a three-dimensional Gaussian body characterizing both the geometric information and the appearance information of the target three-dimensional scene.
[0029] For example, in one instance, the aforementioned multiple 3D Gaussian volumes could specifically include 3D Gaussian volumes representing the geometric information of the target 3D scene and 3D Gaussian volumes representing the appearance information of the target 3D scene; or, they could specifically include 3D Gaussian volumes representing the geometric information of the target 3D scene and 3D Gaussian volumes representing both the geometric and appearance information of the target 3D scene; or, they could specifically include 3D Gaussian volumes representing the appearance information of the target 3D scene and 3D Gaussian volumes representing both the geometric and appearance information of the target 3D scene; or even, they could specifically include all three types of 3D Gaussian volumes simultaneously. Thus, a 3D scene Gaussian representation representing the geometric and appearance information of the target 3D scene is constructed using multiple 3D Gaussian volumes.
[0030] Here, the geometric information may include, but is not limited to: spatial positional relationships, depth relationships, scene outlines and spatial structures in the target 3D scene; the appearance information may include, but is not limited to: color, texture, brightness and other appearance features that change with the viewing direction.
[0031] Additionally, it should be noted that in practical applications, the Gaussian representation of a 3D scene can also be specifically the 3D Gaussian point cloud corresponding to the target 3D scene.
[0032] Step S103: Based on the preset camera trajectory, render the Gaussian representation of the 3D scene to obtain the color image sequence and depth image sequence under the preset camera trajectory.
[0033] For example, in one example, for multiple camera poses in a preset camera trajectory, the Gaussian representation of the 3D scene under each camera pose is rendered to obtain the color image corresponding to each camera pose and the depth image corresponding to each camera pose.
[0034] Here, the color image is used to characterize the color or appearance information of the target 3D scene under the corresponding camera pose; the depth image is used to characterize the depth information of different spatial positions in the target 3D scene relative to the virtual camera under the corresponding camera pose.
[0035] Furthermore, in one example, after obtaining the color image and the depth image corresponding to each camera pose, the color images corresponding to each target camera pose can be arranged according to the order of the multiple camera poses in the preset camera trajectory to obtain a color image sequence; correspondingly, the depth images corresponding to each camera pose are arranged to obtain a depth image sequence.
[0036] Therefore, different images in the color image sequence and the depth image sequence have a corresponding relationship with the camera pose in the preset camera trajectory, such as a one-to-one relationship.
[0037] It should be noted that the color image sequence obtained by rendering through the Gaussian representation of the 3D scene can be specifically understood as the coarse color video corresponding to the target 3D scene video, while the depth image sequence can be specifically understood as the spatial depth sequence corresponding to each frame of the coarse color video.
[0038] In this way, compared with video generation schemes that only use text information or a single reference image as a condition, the color image sequence and depth image sequence in this disclosed scheme can convert the preset camera trajectory into frame-by-frame scene information that can be used by the subsequent video generation model, thereby achieving high-quality generation of 3D scene videos.
[0039] Step S104: Encode the color image sequence and depth image sequence into the compressed latent space to obtain color latent features and depth latent features, and encode the reference image into the compressed latent space to obtain scene content guidance features.
[0040] Here, since the color image sequence and the depth image sequence are obtained by rendering the same Gaussian representation of the 3D scene along a preset camera trajectory, both can provide continuous viewpoint information corresponding to the preset camera trajectory. For example, the depth image sequence can provide the spatial geometry and depth relationships corresponding to consecutive moments or video frames, while the color image sequence can provide coarse appearance information corresponding to consecutive moments or video frames.
[0041] Furthermore, in one example, the color latent feature carries scene appearance information that varies along a preset camera trajectory, and the depth latent feature carries spatial geometric information that varies along a preset camera trajectory.
[0042] For example, in one instance, a video encoder encodes color image sequences and depth image sequences into a compressed latent space to obtain color latent features and depth latent features, and encodes a reference image into the compressed latent space to obtain scene content guidance features.
[0043] Here, the video encoder may be specifically a video variational autoencoder (VAE) encoder, or it may be any other video encoder used to encode image sequences into a compressed latent space. This disclosure does not impose any specific limitations on this.
[0044] Furthermore, in one example, the scene content guidance feature may specifically be a feature representation of the reference image in the latent space, and carries at least one of the following information: scene category, scene layout, object content, color, and texture in the reference image.
[0045] Step S105: Concatenate the color latent features with the depth latent features to obtain the trajectory guidance features.
[0046] For example, in one instance, the color latent features and depth latent features are concatenated along the width direction to obtain trajectory guidance features. This provides subsequent video generation models with color and spatial depth references for each camera pose.
[0047] It should be noted that since the color image sequence carries frame-by-frame appearance information that changes along the preset camera trajectory, and the depth image sequence carries frame-by-frame spatial depth information that changes along the same trajectory, using both to generate trajectory guidance features can simultaneously provide the video generation model with appearance constraints and spatial geometric constraints during the viewpoint change process.
[0048] Furthermore, in one example, trajectory guidance features are used to constrain the viewpoint and spatial structure corresponding to different video frames during video generation, so that the changes in the observation viewpoint in the generated scene video can match the preset camera trajectory.
[0049] Step S106: Based on the trajectory guidance features and scene content guidance features, perform generative denoising processing on the initial noisy video data to obtain the target 3D scene video.
[0050] Here, the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
[0051] In this way, the disclosed solution effectively avoids the error accumulation caused by iterative generation by generating a Gaussian representation of the 3D scene through a coarse-grained scene generation stage and combining the reference image of the target 3D scene to obtain a spatially consistent 3D scene. Furthermore, the 3D scene Gaussian representation is rendered along a preset camera trajectory to efficiently obtain continuous color image sequences and depth image sequences, avoiding the high computational cost of traditional frame-by-frame modeling and significantly improving the efficiency and real-time performance of video generation. Finally, trajectory guidance features and scene content guidance features are generated by combining the reference image and the rendered color image sequence and depth image sequence. Generative denoising processing is then performed on the initial noisy video data by combining the trajectory guidance features and scene content guidance features. In this way, the generated 3D scene video can accurately match the scene content of the reference image and the perspective changes of the preset camera trajectory, thereby achieving visual consistency and dynamic smoothness of the target 3D scene video, thus improving the generation efficiency and quality of the scene video.
[0052] Moreover, since the present invention can combine trajectory guidance features and scene content guidance features to perform generative denoising on the initial noisy video data, compared with the existing video diffusion technology that lacks explicit 3D supervision, the present invention can further improve the detail performance of the scene video while ensuring the spatial consistency of the 3D scene, thereby achieving high-quality generation of 3D scene video.
[0053] Furthermore, in a specific example, the above-described method of generating a Gaussian representation of the target 3D scene based on the reference image includes: Depth estimation is performed on the reference image to obtain the initial point cloud data corresponding to the reference image; The initial point cloud data is voxelized to obtain the first sparse voxel data; Based on the first sparse voxel data, a Gaussian representation of the target 3D scene is generated.
[0054] For example, in one instance, a Gaussian representation of the target 3D scene is generated based on first sparse voxel data carrying at least some geometric information of a reference image and the reference image, and the reference image itself.
[0055] Thus, this disclosure provides a specific scheme for generating a Gaussian representation of a 3D scene. This scheme can directly obtain first sparse voxel data carrying at least part of the geometric information of the reference image from the initial point cloud data corresponding to the reference image, and generate a 3D scene Gaussian representation based on this data. In this way, the 3D scene Gaussian representation generated through the above process not only inherits the geometric features of the reference image, but also integrates appearance information, realizing a unified expression of geometry and appearance, thereby ensuring the accuracy and integrity of the Gaussian representation, and thus providing a more stable scene representation for subsequent video generation.
[0056] Alternatively, in another specific example, in order to improve the representation accuracy of the Gaussian representation of the 3D scene, the present disclosure can also use a preset camera trajectory to voxelize the initial point cloud data corresponding to the reference image, so that the resulting sparse voxel data carries the viewpoint change corresponding to the preset camera trajectory.
[0057] Specifically, Figure 2 This is an illustrative flowchart of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application. Figure 2 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that the above... Figure 1 The methods shown can also be applied to this example, and the related content will not be elaborated further in this example.
[0058] Furthermore, the method includes at least a portion of the following: For example... Figure 2 As shown, it includes: Step S201: Obtain the reference image corresponding to the target 3D scene, and obtain the preset camera trajectory.
[0059] It should be noted that the relevant content regarding the target 3D scene, reference image, and preset camera trajectory can be referred to the above example, and will not be repeated here.
[0060] Step S202: Perform depth estimation on the reference image to obtain the initial point cloud data corresponding to the reference image.
[0061] Here, by performing depth estimation on the reference image, the visible scene content corresponding to the reference image can be converted into initial point cloud data. However, since there are occluded or unobserved scene parts in the reference image, the initial point cloud data is difficult to fully represent all scene content corresponding to the reference image.
[0062] Step S203: Based on the preset camera trajectory, the initial point cloud data is voxelized to obtain conditionally sparse voxel data.
[0063] Here, the conditional sparse voxel data carries the viewpoint changes corresponding to the preset camera trajectory.
[0064] It should be noted that since the initial point cloud data corresponding to the reference image represents the visible scene content corresponding to the reference image, the conditional sparse voxel data obtained after voxelizing the initial point cloud data carries the geometric information corresponding to the visible scene content in the reference image. In other words, the conditional sparse voxel data cannot represent the complete geometric information in the reference image, so voxel completion processing is required on the conditional sparse voxel data to form a complete three-dimensional skeleton.
[0065] Step S204: Based on the conditional sparse voxel data, perform generative denoising on the initial noise voxel data corresponding to the conditional sparse voxel data to obtain target sparse voxel data carrying geometric information of the reference image.
[0066] It should be noted that the present invention can infer the geometric information corresponding to occluded or unobserved scene parts in the reference image based on sparse voxel data representing known geometric information in the reference image, so as to obtain target sparse voxel data. Moreover, the target sparse voxel data contains a more complete three-dimensional spatial structure than the conditional sparse voxel data. Specifically, the target sparse voxel data not only contains the geometric information corresponding to the visible scene content in the input reference image, but also contains the geometric information corresponding to the occluded or unobserved scene content in the reference image. In this way, a complete three-dimensional skeleton is represented.
[0067] Step S205: Based on the target sparse voxel data carrying geometric information of the reference image, generate a Gaussian representation of the target 3D scene.
[0068] Here, the Gaussian representation of the three-dimensional scene includes multiple three-dimensional Gaussian volumes that characterize the geometric and / or appearance information of the target three-dimensional scene.
[0069] It should be noted that the relevant content on Gaussian representation of 3D scenes and 3D Gaussian volumes can be found in the above examples, and will not be repeated here.
[0070] Furthermore, in one example, a Gaussian representation of the target 3D scene, which carries geometric and appearance information of the reference image, can be generated based on the target sparse voxel data carrying geometric information of the reference image and the reference image.
[0071] Step S206: Based on the preset camera trajectory, render the Gaussian representation of the 3D scene to obtain the color image sequence and depth image sequence under the preset camera trajectory.
[0072] It should be noted that the relevant content regarding color image sequences and depth image sequences can be found in the examples above, and will not be repeated here.
[0073] Step S207: Encode the color image sequence and depth image sequence into the compressed latent space to obtain color latent features and depth latent features, and encode the reference image into the compressed latent space to obtain scene content guidance features.
[0074] It should be noted that the relevant content of step S207 can be referred to the above example, and will not be repeated here.
[0075] Step S208: Concatenate the color latent features with the depth latent features to obtain the trajectory guidance features.
[0076] Step S209: Based on the trajectory guidance features and scene content guidance features, perform generative denoising processing on the initial noisy video data to obtain the target 3D scene video.
[0077] Here, the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
[0078] In this way, the disclosed solution can quickly obtain the initial point cloud data corresponding to the visible scene content of the reference image by performing depth estimation, and perform voxelization processing in combination with the preset camera trajectory to obtain conditional sparse voxel data, so that the voxel data is correlated with the viewpoint changes corresponding to the camera trajectory. This provides a spatial basis for the subsequent generation of conditional sparse voxel data and Gaussian representation of the 3D scene. Furthermore, the geometric information represented by the conditional sparse voxel data is completed by using generative denoising, thereby obtaining target sparse voxel data carrying complete geometric information from the reference image. This effectively solves the problem of incomplete geometric information caused by occlusion or viewpoint limitation in the reference image. Finally, the 3D scene Gaussian representation carrying complete geometric information generated from the target sparse voxel data not only preserves the geometric details of the reference image, but also makes up for the missing structural information, significantly improving the ability to express the geometric information of the target 3D scene. As a result, the color image sequence and depth image sequence rendered along the preset camera trajectory are more accurate, thereby improving the visual quality and viewpoint consistency of the scene video generated from the target 3D scene.
[0079] Further, in a specific example, the initial point cloud data corresponding to the reference image can be obtained in the following manner; specifically, the above-described depth estimation of the reference image to obtain the initial point cloud data corresponding to the reference image (e.g., step S202) can specifically include: Step S202-1: Perform depth estimation on the reference image to obtain the depth map corresponding to the reference image.
[0080] Here, different pixels in the depth map can each correspond to a depth value, which is used to represent the spatial distance or depth relationship between the scene position corresponding to that pixel and the reference camera.
[0081] For example, in one instance, a depth estimation network is used to perform monocular depth prediction on a reference image to obtain the depth map corresponding to the reference image.
[0082] Step S202-2: Based on the camera parameters corresponding to the reference image and the reference camera pose corresponding to the reference image, project each pixel in the depth map into three-dimensional space to obtain the initial point cloud data.
[0083] Here, camera parameters can specifically include camera intrinsic parameters (such as field of view (FOV)) and camera extrinsic parameters.
[0084] For example, with the top-left corner of the depth map as the origin of the pixel coordinates, the u-axis points to the top-right corner of the depth map, the v-axis points to the bottom-left corner of the depth map, and the depth value is d; the origin of the camera coordinate system is located at the camera's optical center, the x-axis points directly to the right of the camera, the y-axis points directly below the camera, and the z-axis points directly in front of the camera (i.e., the optical axis direction). The depth value d represents the distance of the 3D point along the optical axis direction, that is, the z-dimensional coordinate of the point in the camera coordinate system; for any pixel in the depth map... Given the camera's intrinsic parameters (such as field of view (FOV)). The pixels, along with the camera's extrinsic parameters in the world coordinate system (such as the rotation matrix R and translation vector S), can be obtained using the following steps. Coordinates of a 3D point in the world coordinate system: Step 1: Construct the intrinsic parameter matrix K.
[0085] Specifically, focal length in pixels. With field of view The following relationship must be satisfied: here, The image height is the depth map height.
[0086] If the given FOV is the horizontal field of view, then the image width W is used instead. In this case, the focal length... With field of view The following relationship must be satisfied: Under the assumption of square pixels, the focal lengths in both directions are equal, that is... Principal point coordinates Take the center of the depth map image: At this point, the intrinsic parameter matrix is: Step 2: Based on the camera's intrinsic parameter matrix, backproject the pixel coordinates onto the camera coordinate system.
[0087] Specifically, firstly, the pixel coordinates Translate the coordinates to a point with the principal point as the origin, then divide by the focal length to obtain the normalized imaging plane coordinates: Secondly, the normalized imaging plane coordinates are multiplied by the depth value d to recover its three-dimensional point coordinates in the camera coordinate system. : here, , , Its specific geometric meaning is: starting from the camera's optical center and passing through the pixel. The direction of the rays A point with a distance d from the optical axis along the ray direction can be taken as a pixel. 3D point coordinates in the camera coordinate system .
[0088] Step 3: Based on the camera's extrinsic parameters in the world coordinate system (including the rotation matrix R and translation vector S used to transform the camera coordinate system to the world coordinate system), the pixels... 3D point coordinates in the camera coordinate system Transform to world coordinates to obtain pixels 3D point coordinates in the world coordinate system The coordinates of this three-dimensional point The following formula can be used to obtain it: In this way, the pixels in the depth map can be obtained. The point cloud (i.e., 3D point) and its coordinates in the world coordinate system are obtained by traversing all pixels in the depth map to obtain the initial point cloud data.
[0089] In this way, the disclosed solution obtains a depth map by performing depth estimation on a reference image, and combines the camera parameters corresponding to the reference image with the reference camera pose to accurately project each pixel in the depth map onto three-dimensional space, thereby generating initial point cloud data. The above process makes full use of known camera parameters (such as FOV) to ensure the accuracy of the mapping from two-dimensional pixels to three-dimensional coordinates, avoiding point cloud distortion or offset caused by projection errors, making the obtained initial point cloud data more reliable in geometric position. This provides a high-quality input foundation for subsequent voxelization and generative voxel completion, and provides favorable support for improving the geometric accuracy of Gaussian representation of three-dimensional scenes.
[0090] Furthermore, in a specific example, conditional sparse voxel data can be obtained in the following manner; specifically, the above-described voxelization process of the initial point cloud data based on a preset camera trajectory to obtain conditional sparse voxel data (e.g., step S203) can specifically include: Step S203-1: Based on the preset camera trajectory and the initial point cloud data, estimate the three-dimensional bounding box corresponding to the target three-dimensional scene.
[0091] Here, by combining the spatial distribution range of multiple 3D points in the initial point cloud data and the observation range covered by the preset camera trajectory, a 3D spatial bounding box is constructed for the target 3D scene to limit its 3D spatial range.
[0092] Step S203-2: Based on the location of each three-dimensional point contained in the initial point cloud data, activate the voxel units in the three-dimensional bounding box corresponding to the location of each three-dimensional point to obtain conditional sparse voxel data.
[0093] In other words, based on the location of the 3D points contained in the initial point cloud data, it is determined whether they fall within a voxel unit in the 3D spatial bounding box. If so, the voxel unit to which the 3D point falls is activated, and the activated voxel unit can be assigned an occupancy flag (such as "1"). In this way, conditional sparse voxel data carrying geometric information corresponding to the visible scene content in the reference image can be obtained.
[0094] Thus, this disclosed solution constructs a 3D spatial bounding box by combining a preset camera trajectory with initial point cloud data. This limits the reasonable spatial range for voxelization operations and avoids invalid processing of irrelevant regions. Furthermore, based on the location of each 3D point in the initial point cloud data, voxel units within the 3D spatial bounding box are activated to obtain conditional sparse voxel data. This method not only preserves the geometric information corresponding to the visible scene content carried by the initial point cloud data, but also eliminates blank areas, significantly reducing the redundancy and computational overhead of voxel data. At the same time, since voxel activation is associated with the preset camera trajectory, the obtained conditional sparse voxel data can provide a structurally clear and semantically accurate input foundation for subsequent generative voxel completion, and can further improve the construction efficiency and accuracy of the 3D scene Gaussian representation.
[0095] Further, in a specific example, the initial noise voxel data mentioned above includes Gaussian noise; in this case, the above-mentioned generative denoising process performed on the initial noise voxel data corresponding to the conditional sparse voxel data based on the conditional sparse voxel data to obtain target sparse voxel data carrying geometric information of the reference image (e.g., step S204) may specifically include: Step S204-1: Using a voxel encoder, the conditional sparse voxel data is encoded into a compressed space to obtain conditional sparse voxel features.
[0096] Here, compressed space can specifically refer to a potential space that has lower data dimensionality or more compact feature representations compared to the original voxel space.
[0097] In one example, the voxel encoder may be specifically a variational autoencoder (VAE), or it may be specifically a general encoding module for voxel encoding, and this disclosure does not impose any specific limitations on it.
[0098] Thus, by encoding conditional sparse voxel data into a compressed space, the main spatial structure information of the conditional sparse voxel data can be preserved while reducing the data size that the generative model needs to process, which provides favorable support for reducing the resource consumption required for subsequent generative denoising processing.
[0099] Step S204-2: The conditional sparse voxel features are concatenated with Gaussian noise to obtain conditional sparse voxel features with Gaussian noise.
[0100] Step S204-3: Input the conditional sparse voxel features with Gaussian noise and the image features of the reference image into the voxel generation model, so as to use the image features of the reference image as a global condition, and combine the conditional sparse voxel features with Gaussian noise to perform generative denoising processing on the Gaussian noise, so as to obtain the target sparse voxel data carrying the geometric information of the reference image.
[0101] Here, the voxel generation model can be specifically a conditional flow matching model based on a 3D diffusion transformer, or it can be other diffusion networks used for three-dimensional diffusion processing. This disclosure does not impose any specific limitations on this.
[0102] It should be noted that, in one example, the voxel generation model described above can be trained in the following manner: Step 1a: Obtain the truth sparse voxels (e.g., resolution is n×n×n (e.g., n is 256), and the value is 0 or 1) Gaussian noise (e.g., resolution is) and feature dimension d (and reference images) .
[0103] Step 2a: Convert the truth value sparse voxels The data is input into a VAE to encode ground truth sparse voxel features. (e.g., resolution is) Latent space features with feature dimension d (corresponding to the conditional sparse voxel feature mentioned above).
[0104] Step 3a: Perform analysis on the ground truth sparse voxel features at time step t. With Gaussian noise Linear interpolation is performed to obtain noisy voxel features. (Corresponding to the conditionally sparse voxel feature with Gaussian noise mentioned above), that is... .
[0105] Step 4a: The reference image is processed using a self-distillation with no-labels encoder (DINO). Feature extraction is performed to obtain the reference image. The visual characteristics (also known as DINO characteristics).
[0106] Step 5a: Transfer noisy voxel features The time step t is used as input, and the visual features of the reference image are used as global conditions. These are input to the voxel generation model to be trained to obtain the estimated denoising speed (also known as the estimated denoising speed field). .
[0107] In practical applications, when inputting the visual features of a reference image into the voxel generation model to be trained, the visual features of the reference image can be input into the model through cross-attention injection.
[0108] Step 6a: Utilize the estimated denoising speed Denoising speed compared to truth value ( for Loss calculation is performed, such as calculating the mean square error of the two above. The calculated mean square error is used to fine-tune the parameters of the voxel generation model to be trained, so as to obtain the trained voxel generation model (that is, the voxel generation model in step S204-3).
[0109] The above is merely an example; in practical applications, other training methods may also be used, and this disclosure does not impose any specific restrictions on them.
[0110] Thus, this disclosed solution provides a specific approach for voxel completion of conditional sparse voxel data using generative denoising. The solution first uses a voxel encoder to encode the conditional sparse voxel data into a compressed space, effectively reducing feature dimensionality and computational complexity while preserving key geometric information. Further, the obtained conditional sparse voxel features are concatenated with Gaussian noise, and the concatenated features and image features from a reference image are input into a voxel generation model, achieving generative denoising of the Gaussian noise. The image features from the reference image in the above process provide global semantic guidance for voxel completion, ensuring that the denoised target sparse voxel data not only inherits the existing geometric information from the conditional sparse voxels but also reasonably infers the geometric information corresponding to occluded or missing parts. This results in target sparse voxel data carrying a complete and accurate 3D geometric structure, laying the foundation for significantly improving the geometric integrity of the Gaussian representation of the 3D scene and enhancing the visual realism and viewpoint consistency of the spatial scene video.
[0111] Further, in a specific example, the Gaussian representation of the target 3D scene can be obtained in the following manner; specifically, the above-mentioned generation of the Gaussian representation of the target 3D scene based on the target sparse voxel data carrying the geometric information of the reference image (e.g., step S205) can specifically include: Step S205-1: Sample the voxels in the target sparse voxel data to obtain voxel noise features.
[0112] For example, in one example, Gaussian noise is sampled at the position corresponding to each voxel in the target sparse voxel data to obtain Gaussian noise data, and then voxel noise features are obtained.
[0113] Step S205-2: Input the voxel noise features and the image features of the reference image into the Gaussian generation model, and use the image features of the reference image as a global condition to perform generative denoising on the voxel noise features to obtain voxel latent space features carrying geometric and appearance information of the reference image.
[0114] It should be noted that, since the image features of the reference image are used as global conditions to participate in the Gaussian generation process, the voxel latent space features obtained by the Gaussian generation model can further carry appearance information such as color and texture that match the reference image.
[0115] Here, the Gaussian generative model can be specifically a flow model based on Structured Latent (SLAT), or it can be specifically any other flow model used for diffusion processing of the latent representation. This disclosure does not impose any specific limitations on this.
[0116] Step S205-3: Use a Gaussian decoder to decode the latent space features of the voxels to obtain the Gaussian representation of the target 3D scene.
[0117] Here, the Gaussian decoder can be specifically a lightweight neural network capable of decoding the latent features of each voxel into a set of 3D Gaussian volumes carrying multiple Gaussian parameters, including the center position, covariance matrix, spherical harmonic coefficients, and opacity. Thus, the decoded 3D Gaussian volumes collectively form a 3D Gaussian representation of the target 3D scene. This representation possesses advantages such as a complete explicit geometric structure, support for differentiable rendering, and fast rendering speed, making it highly suitable as the geometric input representation for subsequent video generation modules.
[0118] It should be noted that, in one example, the Gaussian generation model described above can be trained in the following manner: Step 1b: Obtain the latent space features of the ground truth voxels Reference image and true Gaussian noise .
[0119] Here, the latent space features of the true voxels. The following method can be used: First, acquire scene images from multiple perspectives corresponding to the target 3D scene; then, use a DINO encoder to extract features from the scene images from multiple perspectives to obtain the visual features corresponding to each perspective. Based on the camera parameters of each perspective, map the visual features corresponding to each perspective onto a preset 3D voxel grid to obtain the initial voxel grid data; finally, extract features from the initial voxel grid data to obtain the ground truth voxel latent space features.
[0120] Step 2b: Using true Gaussian noise For true voxel latent space features Noise addition processing is performed to obtain voxel noise characteristics. .
[0121] The specific noise-adding logic can be found in step 3a, and will not be repeated here.
[0122] Step 3b: Translate the reference image using the DINO encoder. Feature extraction is performed to obtain the reference image. Visual features.
[0123] Step 4b: Voxel noise features and reference images The visual features of the reference image are input into a Gaussian generative model, which uses the image features of the reference image as global conditions to perform generative denoising on the voxel noise features, thereby obtaining estimated voxel latent space features carrying geometric and appearance information of the reference image. .
[0124] Here, in practical applications, the reference image The visual features can be injected using cross-attention injection, and this disclosure does not impose specific restrictions on this injection method.
[0125] Step 5b: Based on the latent space features of the truth voxels Potential spatial features of voxels The loss value is calculated, and the parameters of the Gaussian generative model to be trained are fine-tuned based on the loss value to obtain the trained Gaussian generative model (i.e., the Gaussian generative model in step S205-2).
[0126] In practical applications, the estimated Gaussian noise can also be obtained from the output of the Gaussian generator model to be trained. With true Gaussian noise Perform loss calculation, or estimate the Gaussian noise based on the output of the Gaussian generative model to be trained. With true Gaussian noise This yields a loss value, and based on the latent space features of the truth voxels. Potential spatial features of voxels This yields another loss value, and then, based on the two loss values mentioned above, the total loss value is obtained. This disclosed solution does not impose specific limitations on this.
[0127] Thus, this disclosed solution samples voxels from the target sparse voxel data to obtain voxel noise features, and inputs these features, along with the image features of a reference image, into a Gaussian generation model to obtain voxel latent space features that simultaneously carry geometric and appearance information of the reference image. This process transfers the appearance information of the reference image to the voxel noise features carrying complete geometric information, resulting in voxel latent space features that not only contain complete geometric information but also include appearance information such as texture and color. Furthermore, by decoding these voxel latent space features using a Gaussian decoder, a Gaussian representation of the 3D scene with both geometric accuracy and visual realism can be generated. This lays the foundation for subsequently rendering color and depth image sequences along a preset camera trajectory, enhancing the generation quality and immersiveness of the spatial scene video.
[0128] Figure 3 This is an illustrative flowchart of a spatial scene video generation method based on 3D Gaussian point cloud data according to an embodiment of this application. Figure 3 This method can be optionally applied to electronic devices, such as personal computers, servers, and server clusters. It is understood that the above... Figure 1 and Figure 2 The methods shown can also be applied to this example, and the related content will not be elaborated further in this example.
[0129] Furthermore, the method includes at least a portion of the following: For example... Figure 3 As shown, it includes: Step S301: Obtain the reference image corresponding to the target 3D scene, and obtain the preset camera trajectory.
[0130] It should be noted that the relevant content regarding the target 3D scene, reference image, and preset camera trajectory can be referred to the above example, and will not be repeated here.
[0131] Step S302: Based on the reference image, generate a Gaussian representation of the target 3D scene.
[0132] Here, the Gaussian representation of the three-dimensional scene includes multiple three-dimensional Gaussian volumes that characterize the geometric and / or appearance information of the target three-dimensional scene.
[0133] It should be noted that the relevant content on Gaussian representation of 3D scenes can be found in the above example, and will not be repeated here.
[0134] Step S303: Based on the preset camera trajectory, render the Gaussian representation of the 3D scene to obtain the color image sequence and depth image sequence under the preset camera trajectory.
[0135] It should be noted that the relevant content regarding color image sequences and depth image sequences can be found in the examples above, and will not be repeated here.
[0136] Step S304: Encode the color image sequence and depth image sequence into the compressed latent space to obtain color latent features and depth latent features, and encode the reference image into the compressed latent space to obtain scene content guidance features.
[0137] It should be noted that the relevant content regarding scene content guidance features, as well as color latent features and depth latent features, can be found in the examples above, and will not be repeated here.
[0138] Step S305: Concatenate the color latent features with the depth latent features to obtain the trajectory guidance features.
[0139] It should be noted that the relevant content regarding trajectory guidance features can be found in the above example, and will not be repeated here.
[0140] It should be noted that the scene content guidance feature in step S304 can be used as a content reference condition for the entire video generation process, while the trajectory guidance feature in step S305 focuses on providing frame-by-frame constraints that change with the preset camera trajectory.
[0141] Step S306: The trajectory guidance feature is concatenated with the initial noisy video latent feature to obtain a noisy trajectory guidance feature, and the scene content guidance feature is appended to the noisy trajectory guidance feature along a preset sequence (such as a token) dimension to obtain the target conditional input feature.
[0142] For example, in one instance, the noisy trajectory guidance feature can be obtained by concatenating the trajectory guidance feature with the initial noisy video latent features along the channel dimension.
[0143] Step S307: Divide the target conditional input features into N conditional input feature blocks according to the time dimension.
[0144] For example, in one example, the multiple frames contained in the target conditional input feature can be divided according to the preset number of frames N to obtain a preset number of conditional input feature blocks.
[0145] Here, in one example, the value of the preset frame number N can be determined by the sequence length of the preset camera trajectory, that is, it is related to the frame number of the preset camera trajectory, for example, it is less than the sequence length of the preset camera trajectory.
[0146] Step S308: Input N conditional input feature blocks into the video generation model, and combine the conditional information in each conditional input feature block to perform generative denoising processing on each conditional input feature block to obtain the target latent feature block corresponding to each conditional input feature block.
[0147] Here, the video generation model can be specifically a streaming video diffusion model (VDM), or other streaming diffusion models used for video generation. This disclosure does not impose any specific limitations on this.
[0148] Step S309: Perform streaming decoding on the target latent feature blocks corresponding to each condition input feature block to obtain the video frames corresponding to each target latent feature block.
[0149] For example, in one example, the video frames corresponding to each potential frame in the target potential feature block can be decoded according to the temporal order of each target potential feature block after the target potential feature block is obtained, and the video frames corresponding to each potential frame in the target potential feature block can be output.
[0150] It should be noted that in practical applications, when decoding the current target latent feature block, it may depend on the state of the previous block or the features of adjacent blocks. In this case, it is also necessary to obtain the historical decoding state obtained from decoding the previous target latent feature block. Based on the obtained historical decoding state and the current target latent feature block, the video frame corresponding to the current target latent feature block is generated. Furthermore, the historical decoding state can be updated for use in decoding the next target latent feature block. In this way, streaming decoding allows target latent feature blocks to be generated, decoded, and output into video frames step by step, thereby shortening the first frame output time, reducing the memory overhead of caching all feature blocks at once, and also helping to maintain the temporal continuity of adjacent video frames.
[0151] Step S310: Obtain the target 3D scene video based on the video frames corresponding to each target potential feature block.
[0152] Here, the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
[0153] In this way, the disclosed solution significantly improves the coherence and efficiency of spatial scene video generation by concatenating trajectory-guided features with the latent features of the initial noisy video, attaching scene content-guided features along the sequence dimension, and inputting them into the video generation model for block denoising and streaming decoding after being divided according to the time dimension. Moreover, through the block conditional input and streaming decoding mechanism, the computational complexity of a single operation is effectively reduced, avoiding memory overflow or generation interruption caused by excessively long videos. At the same time, with the dual conditional constraints of trajectory and content, it is ensured that the generated video is strictly aligned with the preset camera trajectory in terms of viewpoint changes and highly consistent with the reference image in terms of scene content. This enables control over the "dynamic viewpoint changes" and "static scene content" during the video generation process, thereby achieving highly controllable and efficient 3D scene video generation while ensuring visual quality (such as visual consistency and dynamic smoothness).
[0154] Further, in a specific example, the target latent feature blocks corresponding to each conditional input feature block can be obtained in the following manner; specifically, the above-described method of inputting N conditional input feature blocks into a video generation model to combine the conditional information in each conditional input feature block and perform generative denoising processing on each conditional input feature block to obtain the target latent feature blocks corresponding to each conditional input feature block (e.g., step S308) can specifically include: Step S308-1: When it is necessary to process the i-th conditional input feature, determine the key vector and value vector of the i-1 target latent feature block corresponding to the i-1 conditional input feature block from the preset buffer.
[0155] Step S308-2: Input the key vector of the i-th conditional input feature block and the value vector of the (i-1)-th target latent feature block into the video generation model, so as to combine the key vector and value vector of the (i-1)-th target latent feature block to perform generative denoising on the i-th conditional input feature block and obtain the i-th target latent feature block corresponding to the i-th conditional input feature block.
[0156] Here, for the case where the conditional input feature to be processed is the first one, the key vector and value vector of the (i-1)th target latent feature block input to the video generation model can correspond to the preset key vector and preset value vector.
[0157] Furthermore, after obtaining the i-th target latent feature block, the key vector and value vector of the i-th target latent feature block can be obtained and cached in a preset buffer so as to process the conditional information of the next conditional input feature block as a video generation model.
[0158] Furthermore, in one example, the above-described method of streaming the target latent feature blocks corresponding to each conditional input feature block to obtain the video frames corresponding to each target latent feature block can specifically include: decoding the i-th target latent feature block using a video decoder to obtain the video frames corresponding to the i-th target latent feature block.
[0159] In other words, the video generation model in this disclosed solution processes each of the N conditional input feature blocks sequentially, and after obtaining the target latent feature block, it decodes it to obtain the video frame corresponding to the conditional input feature block, so as to display the video frame corresponding to the conditional input feature block to the user. In this way, by streaming the spatial scene video, the user experience of generating and previewing the spatial scene video at the same time is realized while reducing the consumption of computing resources.
[0160] In this way, the proposed solution introduces a caching mechanism, using the key vector and value vector of the target latent feature block corresponding to the previous conditional input feature block as auxiliary conditions for denoising the current conditional input map feature block. This significantly enhances the temporal coherence and contextual consistency between adjacent feature blocks. Moreover, by reusing the key-value pair information of the previous target latent feature block, the current feature block can fully perceive and continue the semantic and structural features of the preceding frame during the generative denoising process, effectively avoiding problems such as inter-frame jumps, flickering, or content breaks that may be caused by block processing. At the same time, this mechanism also greatly reduces computational redundancy and memory consumption, improving the real-time performance and stability of spatial scene video generation while ensuring generation quality.
[0161] Furthermore, in a specific example, the video generation model includes an intra-block bidirectional attention module for performing intra-block information fusion and an inter-block causal attention module for performing inter-block information fusion.
[0162] In one example, the above-described input of the key vector and value vector of the i-th conditional input feature block and the (i-1)-th target latent feature block into the video generation model, in order to combine the key vector and value vector of the (i-1)-th target latent feature block to perform generative denoising on the i-th conditional input feature block, thereby obtaining the i-th target latent feature block corresponding to the i-th conditional input feature block (e.g., step S308-2), can specifically include: Step S308-2-1: Input the key vector of the i-th conditional input feature block and the value vector of the (i-1)-th target latent feature block into the video generation model.
[0163] Step S308-2-2: Using the intra-block bidirectional attention module in the video generation model, perform intra-block bidirectional attention processing on multiple frames in the i-th conditional input feature block to obtain the i-th initial potential feature block after intra-block information fusion.
[0164] Step S308-2-3: Through the inter-block causal attention module in the video generation model, and based on the query vector of the i-th initial latent feature block after intra-block information fusion, the key vector of the i-1 target latent feature block and the value vector of the i-1 target latent feature block, perform inter-block causal attention processing to obtain the i-th initial latent feature block after inter-block information fusion.
[0165] Step S308-2-4: Based on the i-th initial potential feature block after inter-block information fusion, obtain the i-th target potential feature block.
[0166] For example, in one instance, the video generation model further includes an output projection layer, in which the i-th target latent feature block can be determined from the i-th initial latent feature block after inter-block information fusion through the output projection layer.
[0167] In this way, the proposed solution achieves bidirectional information fusion of multiple frames within a feature block and unidirectional temporal transmission across feature blocks by setting up an intra-block bidirectional attention module and an inter-block causal attention module, respectively. This significantly improves the richness of local details and the logicality of global temporal sequence while maintaining the overall smoothness of video generation. Specifically, the intra-block bidirectional attention module enables different frames within the same input feature block to mutually enhance texture and structural information, avoiding blurring or distortion caused by isolated single-frame generation. The inter-block causal attention module utilizes the key-value pairs of the previous feature block to provide historical context for the denoising processing of the current feature block, ensuring natural and seamless transitions between frames. The synergistic effect of these two modules ensures the spatiotemporal consistency of the spatial scene video, optimizes the allocation of computational resources, and improves generation efficiency and quality.
[0168] It should be noted that, in one example, the video generation model described above can be obtained using the following training method: (a) Data processing stage Step 1a: Obtain training samples, which include: sample camera trajectory, sample reference image corresponding to sample 3D scene, Gaussian representation of sample 3D scene corresponding to target 3D scene, and labeled scene video under sample camera trajectory that matches scene content in sample reference image.
[0169] Step 2a: Based on the sample camera trajectory, render the Gaussian representation of the sample 3D scene to obtain the sample color image sequence and sample depth image sequence under the sample camera trajectory.
[0170] Step 3a: Encode the sample color image sequence and sample depth image sequence into the compressed latent space to obtain sample color latent features and sample depth latent features respectively. Then concatenate the sample color latent features and sample depth latent features to obtain sample trajectory guidance features. Encode the sample reference image into the compressed latent space to obtain sample scene content guidance features.
[0171] (II) Warm-up Training Phase Step 1b: Encode the labeled scene video to obtain the latent features of the labeled video, and add noise to the latent features of the labeled video to obtain the latent features of the labeled video with noise.
[0172] Step 2b: Concatenate the sample trajectory guidance features with the noisy label video latent features to obtain the noisy sample trajectory guidance features, and attach the sample scene content guidance features along the preset sequence dimension to the noisy sample trajectory guidance features to obtain the sample target condition input features.
[0173] Step 3b: Divide the sample target condition input features into N sample condition input feature blocks according to the time dimension, and divide the label video latent features into N label latent feature blocks according to the time dimension.
[0174] Step 4b: When processing the i-th sample condition input feature in the N sample condition input feature blocks, determine the (i-1)-th label latent feature block corresponding to the (i-1)-th sample condition input feature block from the N label latent feature blocks, and obtain the key vector and value vector of the (i-1)-th label latent feature block.
[0175] Step 5b: Input the i-th sample condition input feature block from the N sample condition input feature blocks, as well as the key vector and value vector of the (i-1)-th label latent feature block, into the video generation model to be trained. The intra-block bidirectional attention module in the video generation model will then perform intra-block bidirectional attention processing on multiple frames in the i-th sample condition input feature block to obtain the i-th sample initial latent feature block after intra-block information fusion. Additionally, the inter-block causal attention module in the video generation model will perform inter-block causal attention processing based on the query vector of the i-th sample initial latent feature block after intra-block information fusion, as well as the key vector and value vector of the (i-1)-th label latent feature block, to obtain the i-th sample initial latent feature block after inter-block information fusion. Based on the i-th sample initial latent feature block after inter-block information fusion, the i-th estimated latent feature block will be obtained.
[0176] Step 6b: Based on the difference between the i-th estimated latent feature block and the i-th labeled latent feature block, fine-tune the parameters of the video generation model to be trained (such as the inter-block causal attention module in the video generation model to be trained) to obtain the pre-trained video generation model.
[0177] It should be noted that during this warm-up training phase, when processing each conditional input feature block, the key vector and value vector of the label latent feature block corresponding to the previous sample conditional input feature block are used as the conditional input of the video generation model to be trained. In this way, the video generation model can adapt to the processing logic of the causal attention module.
[0178] (III) Self-regression and self-expansion training phase Step 1c: Encode the labeled scene video to obtain the latent features of the labeled video.
[0179] Step 2c: Concatenate the sample trajectory guidance features with the initial video noise latent features to obtain the noisy sample trajectory guidance features, and attach the sample scene content guidance features along the preset sequence dimension to the noisy sample trajectory guidance features to obtain the sample target condition input features.
[0180] Step 3c: Divide the sample target condition input features into N sample condition input feature blocks according to the time dimension, and divide the label video latent features into N label latent feature blocks according to the time dimension.
[0181] Step 4c: Input the N sample conditional input feature blocks into the video generation model to be trained, and obtain the Nth estimated latent feature corresponding to the last sample conditional input feature block among the N sample conditional input feature blocks.
[0182] Specifically, when processing the i-th sample conditional input feature from N sample conditional input feature blocks, firstly, the key vector and value vector of the (i-1)-th estimated latent feature block corresponding to the (i-1)-th sample conditional input feature block are determined from a preset buffer (such as a key-value cache or a ring buffer). Secondly, the i-th sample conditional input feature block, along with the key vector and value vector of the (i-1)-th estimated latent feature block, are input into the video generation model to be trained. This allows the intra-block bidirectional attention module in the video generation model to perform intra-block bidirectional attention processing on multiple frames within the i-th sample conditional input feature block, resulting in the i-th initial latent feature block after intra-block information fusion. Finally, the inter-block causal attention module in the video generation model is used to further process the inter-block causal attention. The force module, based on the query vector of the initial latent feature block of the i-th sample after intra-block information fusion, and the key vector and value vector of the (i-1)-th estimated latent feature block, performs inter-block causal attention processing to obtain the initial latent feature block of the i-th sample after inter-block information fusion. Based on the initial latent feature block of the i-th sample after inter-block information fusion, the i-th estimated latent feature block is obtained. Here, the key vector and value vector corresponding to the i-th estimated latent feature block can be obtained and cached in a preset cache area. They are used as condition inputs when processing the next sample condition input feature block until the processing of the N-th sample condition input block is completed. At this time, the N-th estimated latent feature is obtained.
[0183] Step 5c: Based on the difference between the Nth estimated latent feature and the Nth label latent feature block among the Nth label latent feature blocks, fine-tune the parameters of the pre-trained video generation model to obtain the trained video generation model.
[0184] In this way, the model can accurately generate spatial videos by leveraging its generative denoising capabilities; in other words, it has the ability to generate high-quality spatial videos.
[0185] The following provides a detailed explanation of the disclosed solution with specific examples. The disclosed solution provides a scene generation scheme based on a 3D Gaussian and video diffusion model. The entire generation process of this scheme uses camera trajectory and a single scene image (as the first frame image of the output scene video) to output a 3D scene video that conforms to the image content of the camera trajectory and the scene image.
[0186] Furthermore, from a modal perspective, the scene generation scheme of this disclosure is divided into two stages: a 3D Gaussian point cloud generation stage and a scene video generation stage. Specifically, (1) The goal of the 3D Gaussian point cloud generation stage is to generate a rough 3D Gaussian point cloud representing the target 3D scene (corresponding to the 3D scene Gaussian representation mentioned above): (a) Using a depth model, the depth information of the input scene image (corresponding to the reference image mentioned above) is estimated to obtain the initial point cloud corresponding to the scene image, which is used to control the generation of the 3D Gaussian point cloud of the 3D scene; (b) Based on the initial point cloud and camera trajectory corresponding to the scene image, the cube size corresponding to the 3D scene is estimated and converted into a voxel mesh to obtain conditional sparse voxel data; (c) Using a sparse voxel VAE encoding network (corresponding to the voxel encoder mentioned above), the conditional sparse voxel data is encoded and compressed to obtain conditional sparse voxel features, and noise is added to them. Then, the noise-added conditional sparse voxel features are input into a 3D denoising diffusion network for generative denoising to obtain complete sparse voxel data (corresponding to the target sparse voxel data mentioned above); Finally, the complete sparse voxel data is converted into a renderable 3D Gaussian point cloud.
[0187] (2) Scene video generation stage, the goal is to generate a scene video of the target 3D scene based on the coarse 3D Gaussian point cloud generated in the previous stage, which conforms to the image content of the camera trajectory and the scene image: (a) Render the coarse 3D Gaussian point cloud according to the input camera trajectory to obtain color-depth (RGB-Depth, RGB-D) video; (b) Encode and add noise to the color video and depth video respectively, and input the processed results into the video diffusion model to perform generative denoising in order to obtain a high-quality 3D scene video that conforms to the image content of the camera trajectory and the scene image.
[0188] Furthermore, such as Figure 4 As shown, the core steps of this disclosed solution include: Phase 1, 3D Gaussian point cloud generation phase: Step S401: Perform point cloud prediction on the input scene image to generate the initial point cloud corresponding to the scene image.
[0189] Here, point cloud prediction is used to construct initial geometric anchor points from the input single scene image, which serve as spatial constraints for subsequent 3D scene Gaussian point cloud generation.
[0190] Specifically, a depth estimation network is first used to perform monocular depth prediction on the input single scene image to obtain the depth map corresponding to the scene image; then, based on the camera intrinsic parameters of the scene image (such as the field of view) and the preset reference camera pose, each pixel in the depth map is back-projected into the three-dimensional space to obtain the partial point cloud (i.e., the initial point cloud) corresponding to the scene image.
[0191] The processing logic for obtaining point clouds through pixel back projection can be found in the example above, and will not be repeated here.
[0192] Step S402: Perform voxelization on the initial point cloud corresponding to the scene image.
[0193] Specifically, after completing the point cloud prediction, the predicted part of the point cloud is voxelized to form an initial sparse voxel structure (corresponding to the conditional sparse voxel data mentioned above). The specific processing logic is as follows: first, based on the initial point cloud and the camera trajectory (corresponding to the preset camera trajectory mentioned above), a 3D bounding box corresponding to the target 3D scene is constructed; then, based on the 3D point coordinates in the initial point cloud, the voxel unit containing the 3D point in the 3D bounding box is activated to construct the conditional sparse voxel data, and the voxels in the conditional sparse voxel data all carry an occupancy marker (such as "1").
[0194] Step S403: Using a sparse voxel VAE encoding network, voxels in the obtained conditional sparse voxel data are voxel encoded and compressed to obtain conditional sparse voxel features. Gaussian noise sampled in the same dimension space is then concatenated with the conditional sparse voxel features along the token dimension to obtain noisy conditional sparse voxel features, which are used as input to the subsequent 3D denoising diffusion network.
[0195] Step S404: Using a 3D denoising diffusion network and taking the image features of the scene image as a global condition, perform generative denoising on the Gaussian noise in the noisy conditional sparse voxel features to obtain complete sparse voxel data, and convert the complete sparse voxel data into a renderable 3D Gaussian point cloud.
[0196] Specifically, a conditional flow matching model based on a 3D diffusion transformer is first used, with the image features of the scene image as global conditions, to perform multiple iterations of denoising on the Gaussian noise in the noisy conditional sparse voxel features to gradually generate complete sparse voxel data; then, the complete sparse voxel data is converted into a renderable 3D Gaussian point cloud. The specific processing logic is as follows: (a) Sampling voxels in the complete sparse voxel data to obtain Gaussian noise data, and then obtaining voxel noise features; (b) Using a SLAT-based flow model, with the image features of the reference image as global conditions, generative denoising is performed on the voxel noise features to obtain voxel latent space features carrying geometric and appearance information of the reference image; (c) Using a Gaussian decoder, voxel latent space features are voxel decoded to obtain a renderable 3D Gaussian point cloud corresponding to the target 3D scene.
[0197] In this example, the 3D denoising diffusion network includes a conditional flow matching model based on a 3D diffusion transformer and a flow model based on SLAT. Furthermore, the training logic for the two models can be found in the example above, and will not be repeated here.
[0198] Phase Two, Scene Video Generation Phase: Step S405: Based on the input camera trajectory, render the 3D Gaussian point cloud corresponding to the target 3D scene, and obtain a coarse RGB-D video (corresponding to the color image sequence and depth image sequence mentioned above).
[0199] Specifically, for each frame of camera pose in the camera trajectory, the 3D Gaussian point cloud corresponding to the target 3D scene is rendered using a 3D Gaussian point cloud differentiable renderer, generating the color image and depth image corresponding to each frame.
[0200] Here, the differentiable renderer described above can quickly generate viewpoint-consistent images and geometric information by sorting, alpha blending, and depth accumulation of all Gaussian point clouds.
[0201] Step S406: Process the coarse RGB-D video, scene image, and Gaussian noise to obtain the target input sequence (corresponding to the target conditional input features mentioned above), and input the target input sequence into the pre-trained video diffusion model (corresponding to the video generation model mentioned above) to obtain the target video latent features.
[0202] Specifically, in this step, firstly, the pre-trained video diffusion model (VDM) is subjected to low-rank adaptive fine-tuning (LORA) and used as a generative delay optimizer; secondly, using a video VAE encoder, the scene image, as well as the color image sequence and depth image sequence in the coarse RGB-D video, are encoded into the same compressed latent space, obtaining the reference latent features corresponding to the scene image (corresponding to the scene content guidance features mentioned above), the color image sequence, the color latent features of each frame, and the depth latent features of each frame corresponding to the depth image sequence; thirdly, the color latent features and the corresponding depth latent features of each frame are concatenated along the width direction to obtain the first concatenated feature sequence (corresponding to the trajectory guidance features mentioned above), and the first concatenated feature sequence is concatenated with the noisy video latent features along the channel dimension to obtain the noisy first concatenated feature sequence, and then the reference latent features are appended to the noisy first concatenated features along the token dimension to obtain the target input sequence; finally, the target input sequence is input into the pre-trained video diffusion model to obtain the target video latent features.
[0203] The specific processing and training logic of the video diffusion model can be referred to in the above example, and will not be repeated here.
[0204] Step S407: Use the video VAE decoder (corresponding to the video decoder mentioned above) to decode the latent features of the target video to obtain a high-fidelity RGB-D video (corresponding to the target 3D scene video mentioned above).
[0205] In summary, the disclosed solution has the following advantages: First, the generated results exhibit good spatial consistency. This disclosed solution directly generates 3D scene Gaussian point clouds through a coarse-grained 3D Gaussian point cloud generation stage, avoiding the error accumulation caused by iterative generation. Therefore, it can obtain spatially consistent 3D scenes. Furthermore, during the scene video generation stage, corresponding depth information is generated simultaneously with the video, which further enhances the spatial consistency of the scene.
[0206] Second, it supports more flexible and diverse camera trajectories. In the fine-grained scene video generation stage, the control conditions required by the video diffusion model are obtained from the coarse-grained 3D scene Gaussian point cloud rendering obtained in the previous stage. Compared with directly encoding the camera trajectory, this method can provide stronger control capabilities and ensure that the generated video conforms to the set camera trajectory.
[0207] Third, it has the ability to repair Gaussian point clouds in 3D scenes. In the coarse-grained 3D Gaussian point cloud generation stage, the disclosed solution can use a conditional flow matching model based on a 3D diffusion transformer to perform voxel-level repair on the conditional sparse voxel data representing the target 3D scene, thus obtaining complete sparse voxel data. In this way, the problems of holes, artifacts and imperfections in the generated 3D scene are effectively avoided, and the representation ability of Gaussian point clouds in 3D scenes is significantly improved.
[0208] Fourth, it supports real-time streaming generation of high-quality video. This disclosed solution divides the target conditional input features requiring denoising into blocks and sequentially inputs them into the video generation model. When processing each conditional input feature block, the model references relevant information (such as key vectors and value vectors) from the target latent feature block corresponding to the previous conditional input feature block. Thus, while generating spatial scene videos in real-time streaming, the quality of the video is ensured.
[0209] This disclosure provides a spatial scene video generation device based on 3D Gaussian point cloud data, such as... Figure 5 As shown, it includes: The acquisition unit 501 is used to acquire a reference image corresponding to the target 3D scene and to acquire a preset camera trajectory; The 3D Gaussian generation unit 502 is used to generate a 3D Gaussian representation of the target 3D scene based on the reference image. The 3D Gaussian representation of the target 3D scene includes multiple 3D Gaussian volumes that characterize the geometric information and / or appearance information of the target 3D scene. The spatial video generation unit 503 is used to render the Gaussian representation of the 3D scene based on the preset camera trajectory to obtain a color image sequence and a depth image sequence under the preset camera trajectory; encode the color image sequence and the depth image sequence into a compressed latent space to obtain color latent features and depth latent features respectively, and encode the reference image into the compressed latent space to obtain scene content guidance features; concatenate the color latent features and the depth latent features to obtain trajectory guidance features; and perform generative denoising processing on the initial noisy video data based on the trajectory guidance features and the scene content guidance features to obtain a target 3D scene video, wherein the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
[0210] In a specific example of the scheme disclosed herein, the initial noisy video data includes potential features of the initial noisy video; The spatial video generation unit is specifically used for: The trajectory guidance feature is concatenated with the initial noisy video latent feature to obtain a noisy trajectory guidance feature, and the scene content guidance feature is appended to the noisy trajectory guidance feature along a preset sequence dimension to obtain the target condition input feature. The target conditional input features are divided into N conditional input feature blocks according to the time dimension; N conditional input feature blocks are input into the video generation model. By combining the conditional information in each conditional input feature block, generative denoising is performed on each conditional input feature block to obtain the target latent feature block corresponding to each conditional input feature block. Streaming decoding is performed on the target latent feature blocks corresponding to each conditional input feature block to obtain the video frames corresponding to each target latent feature block; Based on the video frames corresponding to the potential feature blocks of each target, the 3D scene video of the target is obtained.
[0211] In a specific example of the scheme disclosed herein, the spatial video generation unit is specifically used for: When it is necessary to process the i-th conditional input feature, the key vector and value vector of the i-1 target latent feature block corresponding to the i-1 conditional input feature block are determined from the preset buffer. The key vector and value vector of the i-th conditional input feature block and the (i-1)-th target latent feature block are input into the video generation model. The key vector and value vector of the (i-1)-th target latent feature block are combined to perform generative denoising on the i-th conditional input feature block to obtain the i-th target latent feature block corresponding to the i-th conditional input feature block.
[0212] In a specific example of the scheme disclosed herein, the video generation model includes an intra-block bidirectional attention module for performing intra-block information fusion and an inter-block causal attention module for performing inter-block information fusion. The spatial video generation unit is specifically used for: By using the intra-block bidirectional attention module in the video generation model, multiple frames in the i-th conditional input feature block are processed by intra-block bidirectional attention to obtain the i-th initial potential feature block after intra-block information fusion. By using the inter-block causal attention module in the video generation model, and based on the query vector of the i-th initial latent feature block after intra-block information fusion, and the key vector and value vector of the (i-1)-th target latent feature block, inter-block causal attention processing is performed to obtain the i-th initial latent feature block after inter-block information fusion. Based on the i-th initial potential feature block after inter-block information fusion, the i-th target potential feature block is obtained.
[0213] In a specific example of the disclosed solution, the three-dimensional Gaussian generation unit is specifically used for: Depth estimation is performed on the reference image to obtain the initial point cloud data corresponding to the reference image; Based on the preset camera trajectory, the initial point cloud data is voxelized to obtain conditionally sparse voxel data. Based on the conditional sparse voxel data, the initial noise voxel data corresponding to the conditional sparse voxel data is subjected to generative denoising processing to obtain target sparse voxel data carrying geometric information of the reference image. Based on the target sparse voxel data carrying geometric information of the reference image, a Gaussian representation of the target 3D scene is generated.
[0214] In a specific example of the disclosed solution, the three-dimensional Gaussian generation unit is specifically used for: Depth estimation is performed on the reference image to obtain a depth map corresponding to the reference image; Based on the camera parameters corresponding to the reference image and the reference camera pose corresponding to the reference image, each pixel in the depth map is projected into three-dimensional space to obtain the initial point cloud data.
[0215] In a specific example of the disclosed solution, the three-dimensional Gaussian generation unit is specifically used for: Based on the preset camera trajectory and the initial point cloud data, the three-dimensional bounding box corresponding to the target three-dimensional scene is estimated; Based on the location of each 3D point contained in the initial point cloud data, the voxel units corresponding to the location of each 3D point in the 3D bounding box are activated to obtain conditionally sparse voxel data.
[0216] In one specific example of the scheme disclosed herein, the initial noise voxel data includes Gaussian noise; Specifically, the three-dimensional Gaussian generation unit is used for: The conditional sparse voxel data is encoded into a compressed space using a voxel encoder to obtain conditional sparse voxel features. The conditional sparse voxel features are concatenated with Gaussian noise to obtain conditional sparse voxel features with Gaussian noise. The conditional sparse voxel features with Gaussian noise and the image features of the reference image are input into the voxel generation model. The image features of the reference image are used as global conditions, and the Gaussian noise is generated by combining the conditional sparse voxel features with Gaussian noise to obtain target sparse voxel data carrying the geometric information of the reference image.
[0217] In a specific example of the disclosed solution, the three-dimensional Gaussian generation unit is specifically used for: Voxels in the target sparse voxel data are sampled to obtain voxel noise features; The voxel noise features and the image features of the reference image are input into the Gaussian generation model. The image features of the reference image are used as global conditions to perform generative denoising on the voxel noise features, so as to obtain the voxel latent space features carrying the geometric and appearance information of the reference image. The voxel latent space features are decoded using a Gaussian decoder to obtain a Gaussian representation of the target 3D scene.
[0218] For a description of the specific functions and examples of each unit of the apparatus in this disclosure embodiment, please refer to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be repeated here.
[0219] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0220] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 As shown, the electronic device includes a memory 610 and a processor 620. The memory 610 stores a computer program that can run on the processor 620. There can be one or more memories 610 and processors 620. The memory 610 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the methods provided in the above-described method embodiments. The electronic device may also include a communication interface 630 for communicating with external devices and performing data exchange and transmission.
[0221] If the memory 610, processor 620, and communication interface 630 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0222] Optionally, in a specific implementation, if the memory 610, processor 620, and communication interface 630 are integrated on a single chip, then the memory 610, processor 620, and communication interface 630 can communicate with each other through an internal interface.
[0223] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.
[0224] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).
[0225] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)). It is worth noting that the computer-readable storage media mentioned in this disclosure can be non-volatile storage media; in other words, it can be non-transient storage media.
[0226] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0227] In the description of the embodiments of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0228] In the description of the embodiments disclosed herein, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.
[0229] In the description of embodiments of this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.
[0230] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.
Claims
1. A method for generating spatial scene videos based on 3D Gaussian point cloud data, comprising: Acquire reference images corresponding to the target 3D scene, and acquire preset camera trajectories; Based on the reference image, a Gaussian representation of the target 3D scene is generated, wherein the Gaussian representation of the 3D scene includes multiple 3D Gaussian volumes that characterize the geometric information and / or appearance information of the target 3D scene; Based on the preset camera trajectory, the Gaussian representation of the 3D scene is rendered to obtain the color image sequence and depth image sequence under the preset camera trajectory; Color image sequences and depth image sequences are encoded into a compressed latent space to obtain color latent features and depth latent features, respectively. A reference image is encoded into the compressed latent space to obtain scene content guidance features. The color latent features and the depth latent features are concatenated to obtain the trajectory guidance features; Based on the trajectory guidance features and the scene content guidance features, generative denoising processing is performed on the initial noisy video data to obtain a target 3D scene video, wherein the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
2. The method according to claim 1, wherein, The initial noisy video data includes potential features of the initial noisy video; The step of performing generative denoising on the initial noisy video data based on the trajectory guidance features and the scene content guidance features to obtain the target 3D scene video includes: The trajectory guidance feature is concatenated with the initial noisy video latent feature to obtain a noisy trajectory guidance feature, and the scene content guidance feature is appended to the noisy trajectory guidance feature along a preset sequence dimension to obtain the target condition input feature. The target conditional input features are divided into N conditional input feature blocks according to the time dimension; N conditional input feature blocks are input into the video generation model. By combining the conditional information in each conditional input feature block, generative denoising is performed on each conditional input feature block to obtain the target latent feature block corresponding to each conditional input feature block. Streaming decoding is performed on the target latent feature blocks corresponding to each conditional input feature block to obtain the video frames corresponding to each target latent feature block; Based on the video frames corresponding to the potential feature blocks of each target, the 3D scene video of the target is obtained.
3. The method according to claim 2, wherein, The step of inputting N conditional input feature blocks into the video generation model, combining the conditional information in each conditional input feature block, and performing generative denoising processing on each conditional input feature block to obtain the target latent feature block corresponding to each conditional input feature block includes: When it is necessary to process the i-th conditional input feature, the key vector and value vector of the i-1 target latent feature block corresponding to the i-1 conditional input feature block are determined from the preset buffer. The key vector and value vector of the i-th conditional input feature block and the (i-1)-th target latent feature block are input into the video generation model. The key vector and value vector of the (i-1)-th target latent feature block are combined to perform generative denoising on the i-th conditional input feature block to obtain the i-th target latent feature block corresponding to the i-th conditional input feature block.
4. The method according to claim 3, wherein, The video generation model includes an intra-block bidirectional attention module for fusing information within feature blocks, and an inter-block causal attention module for fusing information between feature blocks. The step of inputting the key vector and value vector of the i-th conditional input feature block and the (i-1)-th target latent feature block into the video generation model, and combining the key vector and value vector of the (i-1)-th target latent feature block to perform generative denoising on the i-th conditional input feature block to obtain the i-th target latent feature block corresponding to the i-th conditional input feature block, includes: By using the intra-block bidirectional attention module in the video generation model, multiple frames in the i-th conditional input feature block are processed by intra-block bidirectional attention to obtain the i-th initial potential feature block after intra-block information fusion. By using the inter-block causal attention module in the video generation model, and based on the query vector of the i-th initial latent feature block after intra-block information fusion, and the key vector and value vector of the (i-1)-th target latent feature block, inter-block causal attention processing is performed to obtain the i-th initial latent feature block after inter-block information fusion. Based on the i-th initial potential feature block after inter-block information fusion, the i-th target potential feature block is obtained.
5. The method according to claim 1, wherein, The step of generating a Gaussian representation of the target 3D scene based on the reference image includes: Depth estimation is performed on the reference image to obtain the initial point cloud data corresponding to the reference image; Based on the preset camera trajectory, the initial point cloud data is voxelized to obtain conditionally sparse voxel data. Based on the conditional sparse voxel data, the initial noise voxel data corresponding to the conditional sparse voxel data is subjected to generative denoising processing to obtain target sparse voxel data carrying geometric information of the reference image. Based on the target sparse voxel data carrying geometric information of the reference image, a Gaussian representation of the target 3D scene is generated.
6. The method according to claim 5, wherein, The step of performing depth estimation on the reference image to obtain the initial point cloud data corresponding to the reference image includes: Depth estimation is performed on the reference image to obtain a depth map corresponding to the reference image; Based on the camera parameters corresponding to the reference image and the reference camera pose corresponding to the reference image, each pixel in the depth map is projected into three-dimensional space to obtain the initial point cloud data.
7. The method according to claim 5 or 6, wherein, The process of voxelizing the initial point cloud data based on a preset camera trajectory to obtain conditionally sparse voxel data includes: Based on the preset camera trajectory and the initial point cloud data, the three-dimensional bounding box corresponding to the target three-dimensional scene is estimated; Based on the location of each 3D point contained in the initial point cloud data, the voxel units corresponding to the location of each 3D point in the 3D bounding box are activated to obtain conditionally sparse voxel data.
8. The method according to claim 5 or 6, wherein, The initial noise voxel data includes Gaussian noise; The step of performing generative denoising on the initial noise voxel data corresponding to the conditional sparse voxel data based on the conditional sparse voxel data to obtain target sparse voxel data carrying geometric information of the reference image includes: The conditional sparse voxel data is encoded into a compressed space using a voxel encoder to obtain conditional sparse voxel features. The conditional sparse voxel features are concatenated with Gaussian noise to obtain conditional sparse voxel features with Gaussian noise. The conditional sparse voxel features with Gaussian noise and the image features of the reference image are input into the voxel generation model. The image features of the reference image are used as global conditions, and the Gaussian noise is generated by combining the conditional sparse voxel features with Gaussian noise to obtain target sparse voxel data carrying the geometric information of the reference image.
9. The method according to claim 5 or 6, wherein, The process of generating a Gaussian representation of the 3D scene corresponding to the target 3D scene based on the target sparse voxel data carrying geometric information of the reference image includes: Voxels in the target sparse voxel data are sampled to obtain voxel noise features; The voxel noise features and the image features of the reference image are input into the Gaussian generation model. The image features of the reference image are used as global conditions to perform generative denoising on the voxel noise features, so as to obtain the voxel latent space features carrying the geometric and appearance information of the reference image. The voxel latent space features are decoded using a Gaussian decoder to obtain a Gaussian representation of the target 3D scene.
10. A spatial scene video generation device based on 3D Gaussian point cloud data, comprising: The acquisition unit is used to acquire a reference image corresponding to the target 3D scene and to acquire a preset camera trajectory; A 3D Gaussian generation unit is used to generate a 3D Gaussian representation of the target 3D scene based on the reference image. The 3D Gaussian representation of the target 3D scene includes multiple 3D Gaussian volumes that characterize the geometric information and / or appearance information of the target 3D scene. A spatial video generation unit is used to render the Gaussian representation of the three-dimensional scene based on the preset camera trajectory to obtain a color image sequence and a depth image sequence under the preset camera trajectory; the color image sequence and the depth image sequence are encoded into a compressed latent space to obtain color latent features and depth latent features, and a reference image is encoded into a compressed latent space to obtain scene content guidance features; The color latent features and the depth latent features are concatenated to obtain the trajectory guidance features; Based on the trajectory guidance features and the scene content guidance features, generative denoising processing is performed on the initial noisy video data to obtain a target 3D scene video, wherein the scene content of the target 3D scene video matches the reference image, and the viewpoint change corresponding to the target 3D scene video matches the preset camera trajectory.
11. The apparatus according to claim 10, wherein, The initial noisy video data includes potential features of the initial noisy video; The spatial video generation unit is specifically used for: The trajectory guidance feature is concatenated with the initial noisy video latent feature to obtain a noisy trajectory guidance feature, and the scene content guidance feature is appended to the noisy trajectory guidance feature along a preset sequence dimension to obtain the target condition input feature. The target conditional input features are divided into N conditional input feature blocks according to the time dimension; N conditional input feature blocks are input into the video generation model. By combining the conditional information in each conditional input feature block, generative denoising is performed on each conditional input feature block to obtain the target latent feature block corresponding to each conditional input feature block. Streaming decoding is performed on the target latent feature blocks corresponding to each conditional input feature block to obtain the video frames corresponding to each target latent feature block; Based on the video frames corresponding to the potential feature blocks of each target, the 3D scene video of the target is obtained.
12. The apparatus according to claim 11, wherein, The spatial video generation unit is specifically used for: When it is necessary to process the i-th conditional input feature, the key vector and value vector of the i-1 target latent feature block corresponding to the i-1 conditional input feature block are determined from the preset buffer. The key vector and value vector of the i-th conditional input feature block and the (i-1)-th target latent feature block are input into the video generation model. The key vector and value vector of the (i-1)-th target latent feature block are combined to perform generative denoising on the i-th conditional input feature block to obtain the i-th target latent feature block corresponding to the i-th conditional input feature block.
13. The apparatus according to claim 12, wherein, The video generation model includes an intra-block bidirectional attention module for fusing information within feature blocks, and an inter-block causal attention module for fusing information between feature blocks. The spatial video generation unit is specifically used for: By using the intra-block bidirectional attention module in the video generation model, multiple frames in the i-th conditional input feature block are processed by intra-block bidirectional attention to obtain the i-th initial potential feature block after intra-block information fusion. By using the inter-block causal attention module in the video generation model, and based on the query vector of the i-th initial latent feature block after intra-block information fusion, and the key vector and value vector of the (i-1)-th target latent feature block, inter-block causal attention processing is performed to obtain the i-th initial latent feature block after inter-block information fusion. Based on the i-th initial potential feature block after inter-block information fusion, the i-th target potential feature block is obtained.
14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.
15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.
16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.
