Efficient scene generation method and device based on local three-dimensional Gaussian rendering

Abstract

The invention provides an efficient scene generation method and device based on local three-dimensional Gaussian rendering, and the method comprises the steps: obtaining a depth image, a text condition and a track list of a first visual angle; generating a plane image of a second view angle based on the space jump step; based on the time interpolation step, generating at least one interpolation frame image between the first visual angle and the second visual angle, depth information corresponding to the plane image of the first visual angle and depth information corresponding to the plane image of the second visual angle; determining a depth image of a second view angle; and so on, until at least one interpolation frame image between the Nth view angle and the (N + 1) th view angle and a depth image of the (N + 1) th view angle are determined; and generating a three-dimensional scene video. Therefore, on the basis of the space jump step and the time interpolation step, according to the scene description text corresponding to the depth image of the previous view angle and the depth image of the current view angle, the interpolation frame image and the depth image of the current view angle can be quickly generated, and the efficiency of three-dimensional scene generation is improved.

Description

Technical Field

[0001] The present disclosure relates to computer three-dimensional scene generation and three-dimensional Gaussian splash rendering technologies, and in particular to an efficient scene generation method and device based on local three-dimensional Gaussian rendering. Background Art

[0002] The booming development of 2D generative models is revitalizing 3D generation, whose generated results are more similar to the real 3D world and have wider applications in fields such as autonomous driving, animation creation, and virtual reality. In the research field of 3D generation, scene generation is a more complex problem because scene-level samples contain various different types of objects and have more complex internal relationships than object-level samples. In addition, each view can only capture a small part of the entire scene, which increases the diversity of generated content and makes it more difficult to achieve cross-view consistency requirements.

[0003] To address these challenges, scene generation models in related technologies usually follow a "build-optimize-render" paradigm, first iteratively building the entire scene through perspective transformation, diffusion-assisted restoration, depth estimation, and heuristic 3D alignment. Each scene is then optimized to remove artifacts and produce a smoother global representation of the entire scene, and finally densely rendered. Although this paradigm is prevalent, low generation efficiency has always been a problem to be solved. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an efficient scene generation method and device based on local three-dimensional Gaussian rendering.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an efficient scene generation method based on local 3D Gaussian rendering, comprising: obtaining a depth image of a first perspective, a text condition and a track list, wherein the text condition includes scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the track list, where N is a positive integer; generating a plane image of a second perspective based on a spatial jump step according to the depth image of the first perspective and the scene description texts corresponding to the depth image of a second perspective adjacent to the first perspective; generating at least one plane image between the first perspective and the second perspective based on a temporal interpolation step according to the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective interpolated frame images, as well as depth information corresponding to the plane image of the first perspective and depth information corresponding to the plane image of the second perspective; determine the depth image of the second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective and the plane image of the second perspective; and so on, until at least one interpolated frame image between the Nth perspective and the N+1th perspective and the depth image of the N+1th perspective are determined; generate a three-dimensional scene video according to the depth image of the first perspective, the depth images of the N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0006] In some embodiments of the present disclosure, a planar image of a second perspective is generated based on a spatial jump step according to a depth image of a first perspective and scene description text corresponding to a depth image of a second perspective adjacent to the first perspective, including: determining an initial depth image of the second perspective according to the depth image of the first perspective, wherein the depth image of the first perspective and the initial depth image of the second perspective have consistent scene contents in some areas, there is a new repair area in the initial depth image of the second perspective, and the scene description text corresponding to the depth image of the second perspective is used to describe the scene content in the new repair area; determining an initial image of the second perspective obtained by projecting the initial depth image of the second perspective, wherein the initial image of the second perspective has a blank area formed by the projection of the new repair area; inputting the scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective into a pre-trained diffusion repair model to generate a planar image of the second perspective.

[0007] In some embodiments of the present disclosure, based on the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective, at least one interpolated frame image between the first perspective and the second perspective, as well as the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the plane image of the second perspective are generated based on a time interpolation step, including: inputting the plane image of the first perspective and the plane image of the second perspective into a pre-trained local Gaussian splatter model to generate at least one interpolated frame image between the first perspective and the second perspective, as well as the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the plane image of the second perspective.

[0008] In some embodiments of the present disclosure, the method further includes: obtaining a denoising training data set, wherein the denoising training data set includes a first plane image obtained by projecting a first depth image, a second plane image generated based on a spatial jump step according to a scene description text corresponding to the first plane image and the second depth image, and a third plane image generated based on a spatial jump step according to the scene description text corresponding to the second plane image and the third depth image; inputting the first plane image and the third plane image into a local Gaussian splash model to generate a predicted second plane image and a predicted third plane image; calculating a first loss based on the second plane image and the predicted second plane image, and calculating a second loss based on the third plane image and the predicted third plane image; updating parameters of the local Gaussian splash model according to the first loss and the second loss to generate a pre-trained local Gaussian splash model.

[0009] In some embodiments of the present disclosure, if the pre-trained local Gaussian splash model is determined based on low-resolution image training, wherein a three-dimensional scene video is generated according to a depth image of a first perspective, depth images of N perspectives after the first perspective, and interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective, including: inputting the interpolated frame image into a super-resolution model to generate a high-resolution interpolated frame image; generating a three-dimensional scene video according to the depth image of the first perspective, depth images of N perspectives after the first perspective, and interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0010] In some embodiments of the present disclosure, a depth image of the second perspective is determined based on depth information corresponding to the depth image of the first perspective, depth information corresponding to the plane image of the first perspective, depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective, including: determining the transformation conditions required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective; determining the depth image of the second perspective based on the transformation conditions required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided an efficient scene generation device based on local 3D Gaussian rendering, comprising: an information acquisition unit, for acquiring a depth image of a first perspective, text conditions and a trajectory list, wherein the text conditions include scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the trajectory list, where N is a positive integer; a first processing unit, for generating a plane image of the second perspective based on a spatial jump step according to the depth image of the first perspective and the scene description texts corresponding to the depth image of the second perspective adjacent to the first perspective; a second processing unit, for generating at least one interpolated frame image between the first perspective and the second perspective based on a time interpolation step according to the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective, as well as the depth information corresponding to the plane image of the first perspective and the second perspective; The invention relates to a method for generating a three-dimensional scene video according to the present invention. The method comprises: obtaining a three-dimensional scene video according to the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the depth image of the second perspective; a condition determination unit, used to determine the transformation condition required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective; a third processing unit, used to determine the depth image of the second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective and the plane image of the second perspective; a fourth processing unit, used to determine at least one interpolated frame image between the Nth perspective and the N+1th perspective, and the depth image of the N+1th perspective by analogy; and a scene generation unit, used to generate a three-dimensional scene video according to the depth image of the first perspective, the depth images of the N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0012] In some embodiments of the present disclosure, a first processing unit is specifically used to: determine an initial depth image of a second perspective based on a depth image of a first perspective, wherein the depth image of the first perspective and the initial depth image of the second perspective have consistent scene contents in some areas, there is a new repair area in the initial depth image of the second perspective, and a scene description text corresponding to the depth image of the second perspective is used to describe the scene content in the new repair area; determine an initial image of the second perspective obtained by projecting the initial depth image of the second perspective, wherein the initial image of the second perspective has a blank area formed by the projection of the new repair area; input the scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective into a pre-trained diffusion repair model to generate a planar image of the second perspective.

[0013] In some embodiments of the present disclosure, the second processing unit is specifically used to: input the plane image of the first perspective and the plane image of the second perspective into a pre-trained local Gaussian splash model to generate at least one interpolated frame image between the first perspective and the second perspective, as well as depth information corresponding to the plane image of the first perspective and depth information corresponding to the plane image of the second perspective.

[0014] In some embodiments of the present disclosure, the above-mentioned device also includes: a data set acquisition unit, used to acquire a denoising training data set, wherein the denoising training data set includes a first plane image obtained by projecting a first depth image, a second plane image generated based on a spatial jump step according to a scene description text corresponding to the first plane image and the second depth image, and a third plane image generated based on a spatial jump step according to the scene description text corresponding to the second plane image and the third depth image; a prediction processing unit, used to input the first plane image and the third plane image into a local Gaussian splash model to generate a predicted second plane image and a predicted third plane image; a loss calculation unit, used to calculate a first loss based on the second plane image and the predicted second plane image, and to calculate a second loss based on the third plane image and the predicted third plane image; a training processing unit, used to update the parameters of the local Gaussian splash model according to the first loss and the second loss to generate a pre-trained local Gaussian splash model.

[0015] In some embodiments of the present disclosure, if the pre-trained local Gaussian splash model is determined based on low-resolution image training, the scene generation unit is specifically used to: input the interpolated frame image into the super-resolution model to generate a high-resolution interpolated frame image; generate a three-dimensional scene video based on the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the high-resolution interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0016] In some embodiments of the present disclosure, the third processing unit is specifically used to: determine the transformation conditions required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective; determine the depth image of the second perspective based on the transformation conditions required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspects is implemented.

[0018] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described in any one of the first aspects when executed by a processor.

[0020] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by acquiring a depth image of a first perspective, a text condition and a track list, wherein the text condition includes scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the track list, where N is a positive integer; based on the depth image of the first perspective and the scene description texts corresponding to the depth image of the second perspective adjacent to the first perspective, a plane image of the second perspective is generated based on a spatial jump step; based on the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective, at least one interpolation frame image between the first perspective and the second perspective is generated based on a time interpolation step, and the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the plane image of the second perspective; according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective and the plane image of the second perspective, the depth image of the second perspective is determined; and so on, until at least one interpolation frame image between the Nth perspective and the N+1th perspective and the depth image of the N+1th perspective are determined; according to the depth image of the first perspective, the depth images of the N perspectives after the first perspective and the interpolation frame images between any two perspectives from the first perspective to the N perspectives after the first perspective, a three-dimensional scene video is generated. Thus, based on the spatial jump step and the time interpolation step, on the basis of the scene description text corresponding to the depth image of the previous perspective and the depth image of the current perspective, an interpolation frame image and a depth image of the current perspective can be quickly generated, and then a three-dimensional scene video is generated according to the depth images of all perspectives and the interpolation frame images between any two perspectives, which can improve the efficiency of three-dimensional scene generation.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1 is a flow chart of an efficient scene generation method based on local three-dimensional Gaussian rendering according to an exemplary embodiment;

[0024] Figure 2 is a schematic diagram of S2 and S3 in an efficient scene generation method based on local three-dimensional Gaussian rendering according to an exemplary embodiment;

[0025] Figure 3 is a schematic diagram showing a denoising method and a comparison of denoising effects shown in an embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram of a three-dimensional scene video shown in an embodiment of the present disclosure;

[0027] Figure 5 is a structural diagram of an efficient scene generation device based on local three-dimensional Gaussian rendering according to an exemplary embodiment;

[0028] Figure 6 is a structural diagram of another efficient scene generation device based on local three-dimensional Gaussian rendering according to an exemplary embodiment;

[0029] Figure 7 It is a structural diagram of another efficient scene generation device based on local three-dimensional Gaussian rendering according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0031] The terms used in the disclosed embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the disclosed embodiments. The singular forms "a", "an" and "the" used in the disclosed embodiments and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0032] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the disclosed embodiments, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the disclosed embodiments, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at" or "when" or "in response to determination".

[0033] In addition, various forms of processes shown in the embodiments of the present disclosure may be used to reorder, add or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present disclosure can be achieved, and this document does not limit this.

[0034] Scene generation models in related technologies usually follow a "build-optimize-render" paradigm, first iteratively building the entire scene through perspective transformation, diffusion-assisted restoration, depth estimation, and heuristic 3D alignment. Each scene is then optimized to remove artifacts and produce a smoother global representation of the entire scene, and finally densely rendered. Although this paradigm is prevalent, low generation efficiency has always been a problem to be solved.

[0035] For example, Text2Room takes half an hour to refine the global mesh, while Text2Nerf takes 1.5 hours to optimize NeRF parameters for each scene. Even with the use of ultra-fast Gaussian splash rendering, LucidDreamer still takes about 10 minutes to generate a scene, and the bottleneck is still preparing samples and optimizing Gaussian parameters. In addition to the redundancy of each scene optimization, hand-designed 3D alignment methods are prone to error accumulation problems and have not yet converged to a mature method.

[0036] Therefore, the main purpose of this disclosure is to explore how to design an efficient 3D scene generation method using local Gaussian splatter model rendering. Similar to NeRF, 3D Gaussian splatter model rendering (3DGS) designs an implicit representation method to render fine-grained and consistent new views using compact anisotropic volume spots. The 3D geometry is modeled by a set of 3D Gaussian functions, each Gaussian primitive is defined by a mean, a covariance matrix, an opacity, and spherical harmonic coefficients. These parameters are updated through a per-sample optimization that starts with structured motion point initialization, then passes through a fast and differentiable Gaussian rasterizer, and ends with a pixel-level image loss between the rendering result and the real image of the selected training view. Although per-sample optimization is still required, 3D Gaussian splatter model rendering is much faster and has comparable performance than the corresponding NeRF method, which can improve the efficiency of 3D scene video generation.

[0037] In the embodiment of the present disclosure, a depth image of a first perspective, a text condition, and a track list are obtained, wherein the text condition includes scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the track list, where N is a positive integer; based on the depth image of the first perspective and the scene description texts corresponding to the depth image of the second perspective adjacent to the first perspective, a plane image of the second perspective is generated based on a spatial jump step; based on the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective, at least one interpolated frame image between the first perspective and the second perspective is generated based on a time interpolation step, as well as a plane image of the first perspective. The depth information corresponding to the image and the depth information corresponding to the plane image of the second perspective; according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective and the plane image of the second perspective, the depth image of the second perspective is determined; and so on, until at least one interpolation frame image between the Nth perspective and the N+1th perspective and the depth image of the N+1th perspective are determined; according to the depth image of the first perspective, the depth images of the N perspectives after the first perspective and the interpolation frame images between any two perspectives from the first perspective to the N perspectives after the first perspective, a three-dimensional scene video is generated. Thus, based on the spatial jump step and the time interpolation step, on the basis of the scene description text corresponding to the depth image of the previous perspective and the depth image of the current perspective, an interpolation frame image and a depth image of the current perspective can be quickly generated, and then a three-dimensional scene video is generated according to the depth images of all perspectives and the interpolation frame images between any two perspectives, which can improve the efficiency of three-dimensional scene generation.

[0038] Figure 1 The invention is a flowchart of an efficient scene generation method based on local three-dimensional Gaussian rendering according to an exemplary embodiment.

[0039] like Figure 1 As shown, it should be noted that the efficient scene generation method based on local three-dimensional Gaussian rendering of the embodiment of the present disclosure is applied to the efficient scene generation device based on local three-dimensional Gaussian rendering. Figure 1 As shown, the method may include but is not limited to the following steps:

[0040] S1: Acquire a depth image of a first perspective, text conditions, and a track list, wherein the text conditions include scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the track list, where N is a positive integer.

[0041] In the embodiment of the present disclosure, a depth image of a first perspective is acquired, wherein the depth image of the first perspective is used as a first frame image of a generated three-dimensional scene video.

[0042] In the disclosed embodiment, the acquired text condition includes scene description text corresponding to the depth images of N perspectives after the first perspective corresponding to the trajectory list, where N is a positive integer.

[0043] In some embodiments, N is 8, 9, or 10.

[0044] It can be understood that the depth image of the first perspective and the depth images of N perspectives after the first perspective require a total of N+1 depth images of perspectives to be acquired, wherein there is overlap between any two adjacent perspectives in the N+1 perspectives, and there is an overlapping part between the depth images of any two adjacent perspectives in the N+1 perspectives.

[0045] For example, Figure 2 As shown, I1 and I2 are depth images of two adjacent perspectives, and there is an overlapping portion between I1 and I2, that is, there is an identical portion in the image contents of I1 and I2.

[0046] S2: Generate a planar image of a second perspective based on a spatial jump step according to the depth image of the first perspective and a scene description text corresponding to the depth image of a second perspective adjacent to the first perspective.

[0047] In the disclosed embodiment, when the depth image, text conditions and trajectory list of the first perspective are obtained, a planar image of the second perspective can be generated based on the spatial jump step according to the scene description text corresponding to the depth image of the first perspective and the depth image of the second perspective adjacent to the first perspective.

[0048] In some embodiments, S2: based on the depth image of the first perspective and the scene description text corresponding to the depth image of the second perspective adjacent to the first perspective, a planar image of the second perspective is generated based on the spatial jump step, including: determining the initial depth image of the second perspective based on the depth image of the first perspective, wherein the scene content of some areas of the depth image of the first perspective and the initial depth image of the second perspective are consistent, there is a new repair area in the initial depth image of the second perspective, and the scene description text corresponding to the depth image of the second perspective is used to describe the scene content in the new repair area; determining the initial image of the second perspective obtained by projecting the initial depth image of the second perspective, wherein the initial image of the second perspective has a blank area formed by the projection of the new repair area; inputting the scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective into a pre-trained diffusion repair model to generate a planar image of the second perspective.

[0049] In the disclosed embodiment, for a first perspective corresponding to a first time step and a second perspective corresponding to a second time step, an initial depth image of the second perspective is determined based on the depth image of the first perspective, wherein the scene content of some areas of the depth image of the first perspective and the initial depth image of the second perspective are consistent, a new repair area exists in the initial depth image of the second perspective, and a scene description text corresponding to the depth image of the second perspective is used to describe the scene content in the new repair area.

[0050] When an initial depth image of the second perspective is obtained, an initial image of the second perspective obtained by projecting the initial depth image of the second perspective is determined, wherein the initial image of the second perspective has a blank area formed by the projection of the new repair area.

[0051] In the disclosed embodiment, when the scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective is obtained, the scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective is input into a pre-trained diffusion restoration model to generate a planar image of the second perspective.

[0052] It can be understood that the extension of different time steps can be equivalent to the switching of different perspectives, for example, the first time step corresponds to the first perspective, the second time step corresponds to the second perspective, and so on.

[0053] It is understandable that the input of the spatial jump step contains the initial image, the corresponding depth map, a text condition, and a predefined list of trajectories (containing the internal and external parameters of the camera pose). Initialize a binary mask with all 1s and an empty point cloud.

[0054] Among them, different values ​​of the binary mask are used to mark the new repair area and the mask area. The binary mask value is 1 to mark the mask area, and the value is 0 to mark the new repair area. Among them, the mask area in the depth map corresponding to the image of the current time step is the area overlapping with the depth map corresponding to the image of the previous time step, and the new repair area is the area where the scene content is to be generated.

[0055] Among them, at any time step (such as the second view), it is assumed that the necessary output (including the image, depth, mask, and point cloud of the previous time step) has been obtained from the previous time step (that is, the first view). First, select a new repair area relative to the previous time step (that is, the first view) from the depth map corresponding to the image of the current time step (such as the second view) (the initial depth image of the second view), and project the mask area back to the 3D space to obtain a newly generated point cloud. Combine the newly generated point cloud with the previous global point cloud, update the global point cloud, and then use 3D to 2D perspective projection to project it to the new view (that is, the initial image of the second view). Then we can repair the mask by filtering out the binary. Finally, we use the pre-trained diffusion repair model to repair part of the image (that is, the initial image of the second view) to obtain the image of the current time step (that is, the plane image of the second view).

[0056] S3: Based on the planar image of the first perspective and the planar image of the second perspective obtained by projecting the depth image of the first perspective, at least one interpolated frame image between the first perspective and the second perspective, as well as the depth information corresponding to the planar image of the first perspective and the depth information corresponding to the planar image of the second perspective are generated based on the time interpolation step.

[0057] In the embodiment of the present disclosure, after obtaining the planar image of the second perspective, at least one interpolated frame image between the first perspective and the second perspective, as well as depth information corresponding to the planar image of the first perspective and depth information corresponding to the planar image of the second perspective can be generated based on a time interpolation step according to the planar image of the first perspective and the planar image of the second perspective obtained by projecting the depth image of the first perspective.

[0058] In some embodiments, the number of interpolated frame images between the first viewing angle and the second viewing angle is 8, 9, or 10.

[0059] In some embodiments, S3: based on the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective, based on the time interpolation step, at least one interpolated frame image between the first perspective and the second perspective, as well as the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the plane image of the second perspective are generated, including: inputting the plane image of the first perspective and the plane image of the second perspective into a pre-trained local Gaussian splash model to generate at least one interpolated frame image between the first perspective and the second perspective, as well as the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the plane image of the second perspective.

[0060] In an embodiment of the present disclosure, a plane image of a first perspective and a plane image of a second perspective are input into a pre-trained local Gaussian splash model to generate at least one interpolated frame image between the first perspective and the second perspective, as well as depth information corresponding to the plane image of the first perspective and depth information corresponding to the plane image of the second perspective.

[0061] It is understandable that if Figure 3 As shown in part (a), since point clouds are usually sparse, projection will produce artifacts. Therefore, the image generated by 3D to 2D projection and patching will have problems such as holes and deformation, which may cause the obtained image frame to be not realistic enough and accumulate errors. To solve this problem, the embodiment of the present disclosure proposes a denoising method to train a generalized Gaussian parameter estimator (i.e., a local Gaussian splash model). Select the image I at time t t (a plane image at a certain viewing angle) and the noisy image I' at time t+2 obtained by projection repair t+2 (a plane image of two views after a certain view) is used as the key frame for estimating the local Gaussian and as the input data for training the generalized Gaussian parameter estimator (i.e., the local Gaussian splash model). The image I at time t+1 is t+1 (a plane image of a certain perspective after a certain perspective) as the interpolation frame, as the prediction target data for training the generalized Gaussian parameter estimator (that is, the local Gaussian splash model). In addition to supervising the generated image L at time t+1 t+1 In addition, an additional denoising loss at time t+2 is added so that the model can implicitly learn to adjust the Gaussian parameters to denoise the image L at time t+2 t+2 .

[0062] In some embodiments, the efficient scene generation method based on local three-dimensional Gaussian rendering provided by the embodiments of the present disclosure also includes: obtaining a denoising training data set, wherein the denoising training data set includes a first plane image obtained by projecting a first depth image, a second plane image generated based on a spatial jump step according to the scene description text corresponding to the first plane image and the second depth image, and a third plane image generated based on a spatial jump step according to the scene description text corresponding to the second plane image and the third depth image; inputting the first plane image and the third plane image into a local Gaussian splash model to generate a predicted second plane image and a predicted third plane image; calculating a first loss based on the second plane image and the predicted second plane image, and calculating a second loss based on the third plane image and the predicted third plane image; updating the parameters of the local Gaussian splash model according to the first loss and the second loss to generate a pre-trained local Gaussian splash model.

[0063] In an embodiment of the present disclosure, a denoising training data set is obtained, wherein the denoising training data set includes a first plane image obtained by projecting a first depth image, a second plane image generated based on a spatial jump step according to a scene description text corresponding to the first plane image and the second depth image, and a third plane image generated based on a spatial jump step according to the scene description text corresponding to the second plane image and the third depth image; it can be understood that the denoising training data includes multiple groups of first plane images, second plane images and third plane images.

[0064] In the disclosed embodiment, a denoising training data set is used to update the parameters of a local Gaussian splash model to generate a pre-trained local Gaussian splash model, which enables the pre-trained local Gaussian splash model to implicitly denoise a noisy image.

[0065] like Figure 3 As shown in part (b), the upper image is the image obtained by directly using the local Gaussian splash model when the local Gaussian splash model is not trained with the denoising training dataset, and the lower image is the pre-trained local Gaussian splash model obtained by training the local Gaussian splash model with the denoising training dataset. It can be clearly seen that the lower image does not have the problems of holes and deformation.

[0066] S4: determining a depth image of the second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the planar image of the first perspective, the depth information corresponding to the planar image of the second perspective, and the planar image of the second perspective.

[0067] In some embodiments, S4: determining the depth image of the second perspective based on the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective, including: determining the transformation conditions required to align the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective; determining the depth image of the second perspective based on the transformation conditions required to align the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

[0068] In the disclosed embodiment, the depth information corresponding to the plane image of the first perspective is obtained by the least squares method, and the offset and scale scaling amount for linear transformation are aligned with the depth information corresponding to the depth image of the first perspective, and then the depth image of the second perspective is determined based on the offset and scale scaling amount, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

[0069] In the disclosed embodiment, the depth image of the second perspective can be aligned to the same scale based on the same transformation condition as the depth image of the first perspective. Through this integrated design, the use of an exogenous depth estimator can be avoided, and only a simple and fast depth alignment is introduced, making the generation framework more lightweight and efficient.

[0070] S5: And so on, until at least one interpolated frame image between the Nth viewing angle and the N+1th viewing angle, and the depth image of the N+1th viewing angle are determined.

[0071] Referring to the aforementioned steps S1 to S4, it is possible to determine the depth image of the second perspective and at least one interpolated frame image between the first perspective and the second perspective based on the depth image of the first perspective and the scene description text corresponding to the depth image of the second perspective adjacent to the first perspective. Similarly, according to steps S1 to S4, the depth image of the next perspective and at least one interpolated frame image between any perspective and the next perspective can be determined based on the depth image of any perspective and the scene description text corresponding to the depth image of the next perspective adjacent to the any perspective. The specific process will not be repeated here.

[0072] For example, Figure 2 As shown in step ① of the process, the process from I2 to I3 is shown, I2 can be a plane image of the second perspective obtained by projecting the depth image of the second perspective, and I3 is a plane image of the third perspective. As shown in step ②, I2 and I3 generate at least one interpolation frame image between the first perspective and the third perspective based on the time interpolation step ( Figure 2 Middle I 2→3 ), and the depth information corresponding to the planar image of the second perspective ( Figure 2 D2) and the depth information corresponding to the plane image of the third perspective ( Figure 2 Then, according to the depth information corresponding to the depth image of the second viewing angle and the depth information corresponding to the plane image of the second viewing angle ( Figure 2 D2), the depth information corresponding to the plane image of the third perspective ( Figure 2 D3) and the plane image I3 of the third perspective, determine the depth image of the third perspective.

[0073] S6: Generate a three-dimensional scene video according to the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0074] In the disclosed embodiment, when a depth image of a first perspective, depth images of N perspectives after the first perspective, and an interpolated frame image between any two perspectives from the first perspective to the N perspectives after the first perspective are obtained, they can be played at a certain frequency to generate a three-dimensional scene video.

[0075] In some embodiments, the 3D scene video has 16, 20 or 30 frames per second.

[0076] It is understandable that if the pre-trained local Gaussian splash model is determined based on high-resolution image training, the interpolated frame image generated as the high-resolution image can make the three-dimensional scene video obtained based on the interpolated frame image smoother and more realistic. The pre-trained local Gaussian splash model is directly obtained based on high-resolution image training, without increasing the resolution of the interpolated frame image, saving time for increasing the resolution of the interpolated frame image, which can save time.

[0077] In some embodiments, if the pre-trained local Gaussian splash model is determined based on low-resolution image training, wherein a three-dimensional scene video is generated according to a depth image of a first perspective, depth images of N perspectives after the first perspective, and interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective, including: inputting the interpolated frame image into a super-resolution model to generate a high-resolution interpolated frame image; generating a three-dimensional scene video according to the depth image of the first perspective, depth images of N perspectives after the first perspective, and high-resolution interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0078] In the disclosed embodiment, when the pre-trained local Gaussian splash model is determined based on low-resolution image training, the interpolated frame image is input into the super-resolution model to generate a high-resolution interpolated frame image; a three-dimensional scene video is generated based on the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the high-resolution interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0079] like Figure 4 As shown, three three-dimensional scene videos (a), (b) and (c) are obtained, wherein the image of the three-dimensional scene video (a) is shown in the first row of images below, the image of the three-dimensional scene video (b) is shown in the second row of images below, and the image of the three-dimensional scene video (c) is shown in the third row of images below.

[0080] certainly, Figure 4 The figures are merely examples and are not intended to be specific limitations on the embodiments of the present disclosure. The number of images included in each 3D scene video may also be other values, such as 100 images, 50 images, etc.

[0081] In the disclosed embodiment, the above-mentioned three-dimensional scene generation process based on local three-dimensional Gaussian splash rendering is adopted to generate a 360-degree scene video within 1 minute, and the generation efficiency is improved by 10 to 100 times compared with the methods in the related art.

[0082] In the disclosed embodiment, a spatial leap step is proposed to convert an image from a source view to a target view by projection and patching between two-dimensional and three-dimensional images, and there is a large viewing angle difference between the source view and the target view.

[0083] In the disclosed embodiment, a temporal interpolation step is proposed to insert smoothly changing frames between the two views generated in the spatial jump step, and render the local aligned depth map of the key frame, and then perform a simple depth alignment process with the previous step. The aligned depth map of the target view is used for projection in the next spatial jump step. This step introduces local Gaussian splash rendering to render image frames and depth maps, taking only the two overlapping images of the spatial jump step as input, and directly predicting the optimized Gaussian parameters through a generalizable neural network.

[0084] In the disclosed embodiment, a denoising training method is proposed. Since the target image obtained by the spatial jump step is generated by projection and patching, there may be some artifacts and distortions. Therefore, a denoising training method is proposed to train a Gaussian parameter estimation model that is more suitable for an efficient scene generation framework.

[0085] Figure 5 is a structural diagram of an efficient scene generation device 1 based on local three-dimensional Gaussian rendering according to an exemplary embodiment. Figure 5 The device includes an information acquisition unit 11, a first processing unit 12, a second processing unit 13, a condition determination unit 14, a third processing unit 15, a fourth processing unit 16 and a scene generation unit 17.

[0086] The information acquisition unit 11 is used to acquire the depth image of the first perspective, text conditions and trajectory list, wherein the text conditions include scene description texts corresponding to the depth images of N perspectives after the first perspective corresponding to the trajectory list, where N is a positive integer.

[0087] The first processing unit 12 is used to generate a planar image of a second perspective based on a spatial jump step according to the depth image of the first perspective and a scene description text corresponding to the depth image of a second perspective adjacent to the first perspective.

[0088] The second processing unit 13 is used to generate at least one interpolated frame image between the first perspective and the second perspective based on the time interpolation step, and the depth information corresponding to the plane image of the first perspective and the depth information corresponding to the plane image of the second perspective according to the plane image of the first perspective and the plane image of the second perspective projected from the depth image of the first perspective.

[0089] The condition determination unit 14 is used to determine the transformation condition required for aligning the depth information corresponding to the planar image of the first viewing angle with the depth information corresponding to the depth image of the first viewing angle.

[0090] The third processing unit 15 is used to determine the depth image of the second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

[0091] The fourth processing unit 16 is configured to perform the above steps in a similar manner until at least one interpolated frame image between the Nth viewing angle and the N+1th viewing angle, and a depth image of the N+1th viewing angle are determined.

[0092] The scene generation unit 17 is used to generate a three-dimensional scene video according to the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0093] In some embodiments of the present disclosure, the first processing unit 12 is specifically used to: determine an initial depth image of a second perspective based on a depth image of a first perspective, wherein the depth image of the first perspective and the initial depth image of the second perspective have consistent scene contents in some areas, there is a new repair area in the initial depth image of the second perspective, and a scene description text corresponding to the depth image of the second perspective is used to describe the scene content in the new repair area; determine an initial image of the second perspective obtained by projecting the initial depth image of the second perspective, wherein the initial image of the second perspective has a blank area formed by the projection of the new repair area; input the scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective into a pre-trained diffusion repair model to generate a planar image of the second perspective.

[0094] In some embodiments of the present disclosure, the second processing unit 13 is specifically used to: input the plane image of the first perspective and the plane image of the second perspective into a pre-trained local Gaussian splash model to generate at least one interpolated frame image between the first perspective and the second perspective, as well as depth information corresponding to the plane image of the first perspective and depth information corresponding to the plane image of the second perspective.

[0095] like Figure 6 As shown, in some embodiments, the efficient scene generation device 1 based on local three-dimensional Gaussian rendering also includes: a data set acquisition unit 18, a prediction processing unit 19, a loss calculation unit 20 and a training processing unit 21.

[0096] Among them, the data set acquisition unit 18 is used to obtain a denoising training data set, wherein the denoising training data set includes a first plane image obtained by projecting a first depth image, a second plane image generated based on a spatial jump step according to a scene description text corresponding to the first plane image and the second depth image, and a third plane image generated based on a spatial jump step according to the scene description text corresponding to the second plane image and the third depth image.

[0097] The prediction processing unit 19 is used to input the first plane image and the third plane image into the local Gaussian splash model to generate a predicted second plane image and a predicted third plane image.

[0098] The loss calculation unit 20 is used to calculate the first loss according to the second plane image and the predicted second plane image, and to calculate the second loss according to the third plane image and the predicted third plane image.

[0099] The training processing unit 21 is used to update the parameters of the local Gaussian splash model according to the first loss and the second loss to generate a pre-trained local Gaussian splash model.

[0100] In some embodiments, if the pre-trained local Gaussian splash model is determined based on low-resolution image training, the scene generation unit 17 is specifically used to: input the interpolated frame image into the super-resolution model to generate a high-resolution interpolated frame image; generate a three-dimensional scene video based on the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the high-resolution interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

[0101] In some embodiments, the third processing unit 15 is specifically used to: determine the transformation conditions required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective; determine the depth image of the second perspective based on the transformation conditions required for aligning the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

[0102] Figure 7 7 is a structural diagram of an efficient scene generation device based on local three-dimensional Gaussian rendering according to an exemplary embodiment. For example, the device 700 may be an electronic device, such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0103] Reference Figure 7, the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0104] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0105] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0106] The power component 706 provides power to the various components of the device 700. The power component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0107] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0108] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0109] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0110] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, and the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0111] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0112] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0113] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0114] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, which implements the above method when executed by the processor 720 of the device 700.

[0115] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0116] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An efficient scene generation method based on local three-dimensional Gaussian rendering, characterized in that: include: Acquire a depth image of a first perspective, a text condition, and a track list, wherein the text condition includes scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the track list, where N is a positive integer; Generate a planar image of a second perspective based on the spatial jump step according to the depth image of the first perspective and the scene description text corresponding to the depth image of a second perspective adjacent to the first perspective; According to the planar image of the first perspective and the planar image of the second perspective obtained by projecting the depth image of the first perspective, at least one interpolated frame image between the first perspective and the second perspective is generated based on the time interpolation step, as well as depth information corresponding to the planar image of the first perspective and depth information corresponding to the planar image of the second perspective; Determine a depth image of a second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective; And so on, until at least one interpolated frame image between the Nth viewing angle and the N+1th viewing angle, and a depth image of the N+1th viewing angle are determined; A three-dimensional scene video is generated according to the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

2. The method according to claim 1, characterized in that The step of generating a planar image of a second perspective based on the depth image of the first perspective and the scene description text corresponding to the depth image of a second perspective adjacent to the first perspective and based on the spatial jump step includes: Determine an initial depth image of a second perspective according to the depth image of the first perspective, wherein the depth image of the first perspective and the initial depth image of the second perspective have consistent scene contents in some regions, a new repair region exists in the initial depth image of the second perspective, and a scene description text corresponding to the depth image of the second perspective is used to describe the scene content in the new repair region; Determine an initial image of the second perspective obtained by projecting the initial depth image of the second perspective, wherein the initial image of the second perspective has a blank area formed by the projection of the new repair area; The scene description text corresponding to the initial image of the second perspective and the depth image of the second perspective is input into a pre-trained diffusion restoration model to generate a planar image of the second perspective.

3. The method according to claim 1, characterized in that The planar image of the first perspective and the planar image of the second perspective obtained by projecting the depth image of the first perspective, based on the time interpolation step, generate at least one interpolated frame image between the first perspective and the second perspective, as well as depth information corresponding to the planar image of the first perspective and depth information corresponding to the planar image of the second perspective, including: The plane image of the first perspective and the plane image of the second perspective are input into a pre-trained local Gaussian splash model to generate at least one interpolated frame image between the first perspective and the second perspective, as well as depth information corresponding to the plane image of the first perspective and depth information corresponding to the plane image of the second perspective.

4. The method according to claim 3, characterized in that The method further comprises: Acquire a denoising training data set, wherein the denoising training data set includes a first plane image obtained by projecting the first depth image, a second plane image generated based on a spatial jump step according to scene description texts corresponding to the first plane image and the second depth image, and a third plane image generated based on a spatial jump step according to scene description texts corresponding to the second plane image and the third depth image; Inputting the first plane image and the third plane image into a local Gaussian splash model to generate a predicted second plane image and a predicted third plane image; calculating a first loss according to the second planar image and the predicted second planar image, and calculating a second loss according to the third planar image and the predicted third planar image; Parameters of the local Gaussian splash model are updated according to the first loss and the second loss to generate the pre-trained local Gaussian splash model.

5. The method according to claim 3 or 4, characterized in that: If the pre-trained local Gaussian splash model is determined based on low-resolution image training, wherein the generating of a three-dimensional scene video according to the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective includes: Inputting the interpolated frame image into a super-resolution model to generate a high-resolution interpolated frame image; A three-dimensional scene video is generated according to the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the high-resolution interpolation frame image between any two perspectives from the first perspective to the N perspectives after the first perspective.

6. The method according to claim 1, characterized in that The determining the depth image of the second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective includes: Determining a transformation condition required for aligning depth information corresponding to the planar image of the first viewing angle with depth information corresponding to the depth image of the first viewing angle; Determine the depth image of the second perspective based on the transformation conditions required to align the depth information corresponding to the plane image of the first perspective with the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective.

7. An efficient scene generation device based on local three-dimensional Gaussian rendering, characterized in that: include: An information acquisition unit, configured to acquire a depth image of a first perspective, a text condition, and a track list, wherein the text condition includes scene description texts corresponding to depth images of N perspectives after the first perspective corresponding to the track list, where N is a positive integer; A first processing unit, configured to generate a planar image of a second perspective based on a spatial jump step according to the depth image of the first perspective and a scene description text corresponding to the depth image of a second perspective adjacent to the first perspective; a second processing unit, configured to generate, based on the time interpolation step, at least one interpolated frame image between the first perspective and the second perspective, and depth information corresponding to the plane image of the first perspective and the plane image of the second perspective obtained by projecting the depth image of the first perspective; a condition determination unit, configured to determine a transformation condition required for aligning depth information corresponding to the planar image of the first viewing angle with depth information corresponding to the depth image of the first viewing angle; a third processing unit, configured to determine a depth image of a second perspective according to the depth information corresponding to the depth image of the first perspective, the depth information corresponding to the plane image of the first perspective, the depth information corresponding to the plane image of the second perspective, and the plane image of the second perspective; A fourth processing unit is used to determine, by analogy, at least one interpolated frame image between the Nth viewing angle and the N+1th viewing angle, and a depth image of the N+1th viewing angle; A scene generation unit is used to generate a three-dimensional scene video based on the depth image of the first perspective, the depth images of N perspectives after the first perspective, and the interpolated frame images between any two perspectives from the first perspective to the N perspectives after the first perspective.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that The computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

Images