Method, device, equipment, medium and program product for generating three-dimensional scene

Abstract

The application discloses a three-dimensional scene generation method, device, equipment, medium and program product. The method comprises the following steps: based on a first view image and a first depth image, performing a denoising process on noise data under the constraint of a multi-plane synchronization condition to obtain a second view image and a second depth image corresponding to at least two second planes respectively; and generating a three-dimensional scene corresponding to a scene element based on a multi-patch map, the first depth image and the at least two second depth images. In the denoising process, the constraint of the multi-plane synchronization condition is used to synchronously analyze a plurality of planes corresponding to a plurality of patches, so that the generation results of all the planes can be globally optimized in the denoising process instead of processing a single plane in isolation, thereby reducing the inconsistency problem of the joints. For example, when the second view image corresponding to a certain plane is generated, the first view image and the generation results of other planes are synchronously referred to, so that the visual information at the joints is naturally transitioned between adjacent planes.

Description

Technical Field

[0001] This application relates to the field of vision, and in particular to a method, apparatus, device, medium, and program product for generating three-dimensional scenes. Background Technology

[0002] Two-dimensional panoramic images are a two-dimensional representation of a three-dimensional scene, showcasing the visual content of a three-dimensional scene from different perspectives. A three-dimensional scene can be constructed by generating two-dimensional panoramic images. A two-dimensional panoramic image includes a cubemap, which consists of six images from different perspectives. These six images are used to present the three-dimensional scene from six perspectives corresponding to the faces of a cube.

[0003] In related technologies, cube maps can be generated through an image diffusion model. Specifically, a single-view image and noise data are input into the image diffusion model. The image diffusion model performs denoising processing on the noise data to obtain images of the other five views in the cube map, excluding the single-view image. These five images and the single-view image constitute a complete cube map.

[0004] However, the cube map generated by the image diffusion model usually has the problem of inconsistent seams. The problem of inconsistent seams means that at the edge connection between the 6 images of the cube map, visual information such as color, lighting, and texture are discontinuous or misaligned, resulting in obvious breaks or incongruities in the 3D scene generated by the cube map, that is, the quality of the generated 3D scene is low. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and program product for generating three-dimensional scenes, the technical solution of which is as follows:

[0006] On the one hand, a method for generating a three-dimensional scene is provided, the method comprising:

[0007] Obtain a first-view image corresponding to a scene element and a first-depth image corresponding to the first-view image. The first-view image covers the first plane of the polyhedron, and the first plane corresponds to the first view.

[0008] Obtain noise data corresponding to at least two second planes, where the second planes refer to planes in the polyhedron other than the first plane;

[0009] Based on the first viewpoint image and the first depth image, a denoising process is performed on the noisy data under the constraint of multi-plane synchronization conditions to obtain the second viewpoint image and the second depth image corresponding to the at least two second planes respectively. The multi-plane synchronization conditions are used to synchronously analyze multiple planes corresponding to the polyhedron during the denoising process. The at least two second viewpoint images and the first viewpoint image are used to form a polyhedron texture covering the polyhedron.

[0010] A 3D scene corresponding to the scene elements is generated based on the polyhedral texture, the first depth image, and at least two second depth images.

[0011] On the other hand, a three-dimensional scene generation apparatus is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire a first-view image corresponding to a scene element and a first-depth image corresponding to the first-view image. The first-view image covers the first plane of the polyhedron, and the first plane corresponds to the first view.

[0013] The acquisition module is used to acquire noise data corresponding to at least two second planes, where the second planes refer to planes in the polyhedron other than the first plane.

[0014] The generation module is used to perform a denoising process on the noisy data based on the first view image and the first depth image under the constraint of multi-plane synchronization conditions, to obtain the second view image and the second depth image corresponding to the at least two second planes respectively. The multi-plane synchronization conditions are used to synchronously analyze multiple planes corresponding to the polyhedron during the denoising process. The at least two second view images and the first view image are used to form a polyhedron texture covering the polyhedron.

[0015] The generation module is used to generate a 3D scene corresponding to the scene elements based on the polyhedral texture, the first depth image, and at least two second depth images.

[0016] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement any of the above-described methods for generating three-dimensional scenes.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being loaded and executed by a processor to implement the method for generating a three-dimensional scene as described above.

[0018] On the other hand, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the above-described methods for generating a three-dimensional scene.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] Based on the first-view image and first-depth image corresponding to a certain plane of the polyhedron, a denoising process is performed on the noisy data under the constraint of multi-plane synchronization, resulting in second-view images and second-depth images corresponding to the other planes of the polyhedron. The second-view image and the first-view image are used to construct a polyhedron texture covering the polyhedron. A 3D scene corresponding to scene elements can be generated using the polyhedron texture, the first-depth image, and the second-depth image. During the denoising process, the constraints of multi-plane synchronization allow for the simultaneous analysis of multiple planes corresponding to the polyhedron. This enables the global optimization of the generation results of all planes during denoising, rather than processing a single plane in isolation, thereby reducing inconsistencies at seams. For example, when generating the second-view image corresponding to a certain plane, the generation results of the first-view image and other planes are referenced simultaneously to ensure a natural transition of visual information at the seams between adjacent planes. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application;

[0023] Figure 2 This is a flowchart of a method for generating a three-dimensional scene provided in an exemplary embodiment of this application;

[0024] Figure 3 This is a schematic diagram of a denoiser for an image diffusion model provided in an exemplary embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the process of generating a three-dimensional scene according to an exemplary embodiment of this application;

[0026] Figure 5 This is a flowchart of a method for generating a three-dimensional scene provided in another exemplary embodiment of this application;

[0027] Figure 6 This is a flowchart of a training method for an image diffusion model provided in an exemplary embodiment of this application;

[0028] Figure 7 This is a flowchart illustrating the training process of an image diffusion model provided in an exemplary embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the training process of an image diffusion model provided in an exemplary embodiment of this application;

[0030] Figure 9 This is a flowchart illustrating the reasoning process of an image diffusion model provided in an exemplary embodiment of this application;

[0031] Figure 10 This is a schematic diagram illustrating the reasoning process of an image diffusion model provided in an exemplary embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the product architecture of a 3D scene generation system provided in an exemplary embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the technical architecture of a three-dimensional scene generation system provided in an exemplary embodiment of this application;

[0034] Figure 13 This is a structural block diagram of a three-dimensional scene generation apparatus provided in an exemplary embodiment of this application;

[0035] Figure 14 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In this application, the terms "first" and "second" are used to distinguish between identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on the quantity or execution order.

[0038] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with relevant laws, regulations, and standards.

[0039] Two-dimensional panoramic images are a two-dimensional representation of a three-dimensional scene, showcasing the visual content of a three-dimensional scene from different perspectives. A three-dimensional scene can be constructed by generating two-dimensional panoramic images. A two-dimensional panoramic image includes a cube map, which consists of six images from different perspectives. These six images are used to present the three-dimensional scene from six perspectives corresponding to the cube faces. In related technologies, cube maps can be generated using an image diffusion model. Specifically, a single-view image and noise data are input into the image diffusion model. The image diffusion model performs denoising processing on the noise data to obtain images from the other five perspectives in the cube map, excluding the single-view image. These five perspective images, along with the single-view image, constitute a complete cube map. However, cube maps generated by image diffusion models often suffer from inconsistencies in seams. These inconsistencies refer to discontinuities or misalignments in visual information such as color, lighting, and texture at the edge connections between the six images of the cube map. This results in a noticeable break or incongruity in the generated three-dimensional scene, indicating a lower quality of the generated 3D scene.

[0040] Based on this, this application provides a method for generating 3D scenes based on multi-plane synchronization constraints and depth information fusion. Using a first-view image and a first depth image corresponding to a certain plane of a polyhedron, a denoising process is performed on noisy data under the constraint of multi-plane synchronization conditions to obtain second-view images and second-depth images corresponding to the other planes of the polyhedron. The second-view images and the first-view images are used to construct a polyhedron texture covering the polyhedron. A 3D scene corresponding to scene elements can be generated using the polyhedron texture, the first depth image, and the second depth image. During the denoising process, the constraints of the multi-plane synchronization conditions allow for the simultaneous analysis of multiple planes corresponding to the polyhedron. This enables global optimization of the generation results of all planes during denoising, rather than isolated processing of individual planes, thereby reducing inconsistencies at seams. For example, when generating a second-view image corresponding to a certain plane, the generation results of the first-view image and other planes are referenced simultaneously to ensure a natural transition of visual information at seams between adjacent planes.

[0041] Furthermore, compared to related technologies, this application introduces a first depth image as conditional information during the denoising process. On one hand, during denoising, the first depth image, as conditional information, can constrain the generated images from other perspectives to maintain consistency with the first perspective image in terms of geometric features (such as the position and shape of scene elements), avoiding geometric inconsistencies in the generated polyhedral textures. Moreover, combining multi-plane synchronization conditions and the first depth image ensures smooth transitions at seams in the generated polyhedral textures, reducing visual artifacts caused by geometric mismatches. On the other hand, based on the first depth image, second depth images from other perspectives can be generated, providing a reference for depth information in the generated 3D scene and improving the quality of the generated 3D scene.

[0042] The computer system used to implement the three-dimensional scene generation method provided in this application is described below.

[0043] Figure 1 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application. The computer system can implement a system architecture for a method of generating a 3D scene. The computer system includes a terminal 110 and a server 120. The terminal 110 is connected to the server 120 via a wireless network or a wired network.

[0044] Terminal 110 can be an electronic device such as a mobile phone, tablet computer, vehicle terminal (vehicle system), wearable device, PC (Personal Computer), virtual reality (VR) device, augmented reality (AR) device, etc. A client for a first application can be installed and run on terminal 110. This first application can be an application for generating 3D scenes, or an application that provides 3D scene generation functionality; this application does not limit the specific form of the first application. Furthermore, this application does not limit the form of the first application, including but not limited to apps, mini-programs, etc., installed on terminal 110, and can also be in web page form. Optionally, the first application can be implemented as an application with UGC (User Generated Content) editing functionality (such as casual competitive games, party games, and sandbox games), allowing users to create 3D scenes using the UGC editing functionality provided by the first application on terminal 110. Indicatively, the first application includes a game level creation interface with UGC editing capabilities. In this interface, users can design 3D game scenes (i.e., 3D scenes) for their own use. For example, if a user uploads a scene image (i.e., a first-person view image) in the game level creation interface, the first application can generate a 3D game scene using the 3D scene generation method provided in this application, thereby improving the efficiency of users in creating game levels.

[0045] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server providing basic cloud computing services, or a node in a blockchain system. Server 120 can be the backend server of the aforementioned first application, used to provide backend services to the clients of the first application.

[0046] The method for generating a 3D scene provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. For example... Figure 1 As shown, the method for generating a 3D scene can be executed by terminal 110 (e.g., by a client of the first application installed and running on terminal 110), or by server 120, or by a combination of interaction between terminal 110 and server 120. This application does not limit the specific method.

[0047] The following explanation uses the method of generating a 3D scene by the interaction and cooperation between terminal 110 and server 120 as an example.

[0048] Terminal 110 acquires a first-person perspective image corresponding to a scene element. This first-person perspective image can be an image uploaded by the user, such as the scene image uploaded by the user in the game level creation interface, as mentioned above. The first-person perspective image covers the first plane of a polyhedron, with different planes of the polyhedron corresponding to different perspectives of the 3D scene, and the first plane corresponding to the first perspective. Terminal 110 sends the first-person perspective image to server 120.

[0049] After receiving the first-view image, server 120 analyzes the first-view image to obtain the depth information of each pixel to obtain the first depth image. The first-view image and the first depth image can be regarded as RGBD images of the three-dimensional scene in the first view, where "RGB" represents color information and "D" represents "depth information". Alternatively, the first depth image is obtained in advance. For example, if the first depth image is the depth information set by the user for the first-view image, the user can upload the set first depth image to terminal 110, and then terminal 110 sends the first-view image to server 120.

[0050] After acquiring a first-view image and a first-depth image, server 120 acquires noise data corresponding to at least two second planes, where each second plane refers to a plane in the polyhedron other than the first plane. Then, based on the first-view image and the first-depth image, server 120 performs a denoising process on the at least two noise data points under the constraint of a multi-plane synchronization condition, obtaining second-view images and second-depth images corresponding to the at least two second planes. The multi-plane synchronization condition is used to simultaneously analyze multiple planes corresponding to the polyhedron during the denoising process. The at least two second-view images and the first-view image can constitute a polyhedron texture covering the polyhedron, which is used to present scene elements from multiple perspectives corresponding to the polyhedron. Schematic, server 120 includes an image diffusion model, which inputs the first-view image, the first-depth image, and at least two noise data points to the image diffusion model and outputs the polyhedron texture.

[0051] After acquiring the polyhedral texture, server 120 generates a 3D scene corresponding to scene elements based on the polyhedral texture, a first depth image, and second depth images corresponding to at least two second planes. The first depth image and at least two second depth images constitute a multi-plane depth image corresponding to the polyhedral texture. This polyhedral texture and multi-plane depth image can be considered as an RGBD polyhedral texture of the 3D scene. The RGBD polyhedral texture is an extended version of the standard polyhedral texture, adding depth information "D" to the color information "RGB". The color information "RGB" provides the appearance information of the 3D scene, and the depth information "D" provides the geometric information of the 3D scene. Schematic, the RGBD polyhedral texture is input into the scene generator, and the 3D scene corresponding to the scene elements is output. Optionally, after generating the 3D scene, server 120 feeds the 3D scene back to terminal 110.

[0052] Those skilled in the art will understand that the number of terminals 110 described above can be more or less. For example, there may be only one terminal 110, or there may be dozens or hundreds of terminals 110, or even more. This application does not limit the number or type of terminals 110.

[0053] Next, the process of the three-dimensional scene generation method provided in the embodiments of this application will be described.

[0054] Figure 2 This is a flowchart illustrating a method for generating a three-dimensional scene according to an embodiment of this application. The method is executed by a computer device, which may be... Figure 1 The terminal 110 and / or server 120 are shown. The method includes steps 210 to 240.

[0055] Step 210: Obtain the first-view image corresponding to the scene element and the first depth image corresponding to the first-view image.

[0056] Scene elements are the basic components that make up a scene, and different scenes correspond to different scene elements. For example, in a room scene, scene elements include walls, furniture, lighting, windows, etc.

[0057] First-person perspective images are two-dimensional images that include scene elements.

[0058] In this context, the first-view image covers the first plane of the polyhedron, and different planes of the polyhedron correspond to different perspectives of the 3D scene. The first plane corresponds to the first perspective, which means that the scene elements in the first-view image can be regarded as scene elements presented from the first perspective.

[0059] A polyhedron is a three-dimensional geometric body enclosed by multiple (usually at least four) polygonal planes (triangles, quadrilaterals, etc.). Optionally, a polyhedron includes at least one of a regular tetrahedron, a cube (regular hexahedron), a regular octahedron, etc. In this application, a cube is used as an example to illustrate the polyhedron.

[0060] For a cube, the cube consists of six square planes, which correspond to six viewpoints of the three-dimensional scene. Optionally, the six square planes of the cube correspond to six orthogonal viewpoints of the three-dimensional scene. Schematic, with the center point of the cube (which can be regarded as the camera position) as the origin of the coordinate system, the six orthogonal viewpoints corresponding to the cube are: (1) Right viewpoint (corresponding to the +X face of the cube, i.e., the right side): looking from the center point towards the positive X-axis direction; (2) Left viewpoint (corresponding to the -X face of the cube, i.e., the left side): looking from the center point towards the negative X-axis direction; (3) Top viewpoint (corresponding to the +Y face of the cube, i.e., the top face): looking from the center point towards the positive Y-axis direction; (4) Bottom viewpoint (corresponding to the -Y face of the cube, i.e., the bottom face): looking from the center point towards the negative Y-axis direction; (5) Front viewpoint (corresponding to the +Z face of the cube, i.e., the front side): looking from the center point towards the positive Z-axis direction; (6) Rear viewpoint (corresponding to the -Z face of the cube, i.e., the rear side): looking from the center point towards the negative Z-axis direction. In this application, the first perspective can be implemented as at least one of the following: right perspective, left perspective, top perspective, bottom perspective, front perspective, and rear perspective. Schematic, if the first perspective is implemented as a front perspective, then the first perspective image is regarded as a two-dimensional image obtained by observing a three-dimensional scene from the front perspective.

[0061] First-person view images are used to represent the appearance information of scene elements. In a schematic way, first-person view images include the color information of each pixel, which refers to the R (Red), G (Green), and B (Blue) values ​​of the pixel.

[0062] The first depth image is used to represent the geometric information of scene elements. Schematic, the first depth image includes the depth information of each pixel, which refers to the D (Depth) value of the pixel, where the depth value represents the distance between the scene position corresponding to the pixel and the camera position.

[0063] In some embodiments, the first-view image covers a single plane of the polyhedron, and is used to present scene elements from a single perspective (i.e., the first perspective). In other embodiments, the first-view image covers at least two planes of the polyhedron, and is used to present scene elements from at least two perspectives. For example, the first-view image may consist of a first sub-image and a second sub-image, where the first sub-image is used to present scene elements from a first perspective, and the second sub-image is used to present scene elements from a second perspective. In other words, in this application, a polyhedron texture corresponding to a polyhedron can be generated using a single-view image corresponding to a scene element, or a polyhedron texture can be generated using a multi-view image corresponding to a scene element; no limitation is imposed here.

[0064] Step 220: Obtain noise data corresponding to at least two second planes respectively.

[0065] The second plane refers to any plane in a polyhedron other than the first plane. For example, if the polyhedron is a cube, and the first plane is the front face of the cube from a frontal viewpoint, then at least two second planes include the right face of the cube from a right-side viewpoint, the left face of the cube from a left-side viewpoint, the top face of the cube from an upper viewpoint, the bottom face of the cube from a lower viewpoint, and the rear face of the cube from a rear viewpoint.

[0066] Noise data refers to random noise, which is randomly sampled from a random distribution (such as a Gaussian distribution). Optionally, each second plane corresponds to a first noise image and a second noise image. The pixel values ​​in the first noise image represent color information, and these pixel values ​​are randomly sampled from a random distribution (such as a Gaussian distribution). The pixel values ​​in the second noise image represent depth information, and these pixel values ​​are also randomly sampled from a random distribution (such as a Gaussian distribution).

[0067] Optionally, at least two can also be at least three, at least five, etc., that is, at least three noise data corresponding to the second plane respectively, or at least five noise data corresponding to the second plane respectively.

[0068] Step 230: Based on the first viewpoint image and the first depth image, perform a denoising process on the noisy data under the constraint of multi-plane synchronization conditions to obtain at least two second viewpoint images and second depth images corresponding to the second planes respectively.

[0069] Optionally, using the first viewpoint image and the first depth image as denoising conditions, a denoising process is performed on the noisy data under the constraint of multi-plane synchronization conditions to obtain second viewpoint images and second depth images corresponding to at least two second planes, respectively. The denoising conditions are used to indicate guiding information in the denoising process of the noisy data. In this application, the first viewpoint image and the first depth image are used as denoising conditions; that is, the first viewpoint image and the first depth image are used as guiding information in the denoising process of the noisy data.

[0070] Multi-plane synchronization is used to simultaneously analyze multiple planes corresponding to a polyhedron during the denoising process. That is, during the denoising process, the viewpoint image data corresponding to multiple planes of the polyhedron are analyzed simultaneously. For example, when analyzing the first viewpoint image, the viewpoint image data corresponding to other planes (i.e., at least two second planes) need to be considered.

[0071] Optionally, the multi-plane synchronization condition includes at least one of the following conditions:

[0072] (1) Synchronous attention operation constraints: used to synchronously analyze the viewpoint image data corresponding to multiple planes of the polyhedron during the attention operation in the denoising process.

[0073] Optionally, attention operations include at least one of self-attention operations, cross-attention operations, etc.

[0074] This illustration demonstrates how synchronous attention operation constraints are used during denoising to consider viewpoint image data from other planes when performing attention operations on a specific plane. Optionally, for self-attention operations, features from other planes are used as references when processing features of a plane to enhance the calculation of attention weights for the current plane. For cross-attention operations, features from other planes are used as "keys" and "values," while features from the current plane are used as "queries," and attention weights are obtained through cross-plane attention calculations.

[0075] (2) Constraints for synchronous convolution operation: used to simultaneously analyze the viewpoint image data corresponding to multiple planes of the polyhedron during the convolution operation in the denoising process.

[0076] As an illustration, the synchronous convolution operation constraint is used to consider the viewpoint image data corresponding to other planes when performing a convolution operation on a certain plane during the denoising process. Optionally, when performing a convolution operation on the features of a certain plane, the features used for convolution also include features from other planes, thereby achieving cross-plane convolution.

[0077] (3) Synchronous normalization operation constraint: used to simultaneously analyze the viewpoint image data corresponding to multiple planes of the polyhedron during the normalization operation in the denoising process.

[0078] This illustrative example illustrates how synchronous normalization constraints are used during denoising to perform global normalization calculations on the viewpoint image data corresponding to multiple planes of a polyhedron, rather than independently normalizing the viewpoint image data of a single plane. Optionally, the normalization parameters of all planes are calculated synchronously to ensure consistent distribution across different planes. Alternatively, the normalization parameters of different planes can be dynamically adjusted using learnable parameters to adapt them to each other.

[0079] In some embodiments, a denoising process is performed using an image diffusion model. Optionally, a first-view image, a first-depth image, and noise data corresponding to at least two second planes are input into the image diffusion model, and the model outputs second-view images and second-depth images corresponding to at least two second planes.

[0080] The image diffusion model includes an operator based on multi-plane synchronization conditions. This multi-plane synchronization operator is used to simultaneously analyze multiple planes corresponding to a polyhedron during denoising. Illustratively, an operator refers to a basic unit that performs a specific computational task during denoising. In this application, based on multi-plane synchronization conditions, the spatial dimension of the operator in the image diffusion model is adjusted from a single-plane (or single-viewpoint) spatial dimension "H×W" to a multi-plane (or multi-viewpoint) spatial dimension "M×H×W". H represents the height of the viewpoint image (number of pixels in the vertical direction), W represents the width of the viewpoint image (number of pixels in the horizontal direction), and M represents the number of planes of the polyhedron. If the polyhedron is implemented as a cube, then M = 6. The operator with the spatial dimension "M×H×W" is also the multi-plane synchronization operator in this application.

[0081] Optionally, the multi-plane synchronization operator includes at least one of the following operators:

[0082] (1) Convolutional layer operator: The convolutional layer operator refers to the operator of the convolutional layer in the image diffusion model. The convolutional layer operator represents the basic unit that performs convolution operation in the denoising process.

[0083] Optionally, the convolutional layer operator is an operator determined based on the constraints of the synchronous convolution operation. Illustratively, in this application, the spatial dimension of the input data to the convolutional layer of the image diffusion model is expanded to "M×H×W" based on the convolutional layer operator, and pixel values ​​of the viewpoint images corresponding to other planes are filled into the convolutional layer input data based on the convolutional layer operator. For example, when performing a convolution operation on the first plane image of the first plane, the data corresponding to other planes in the convolutional layer input data of the first plane image can be filled with color values ​​of the viewpoint images corresponding to other planes, thereby replacing zero-padding in related technologies and eliminating discontinuities at seams. The convolutional layer operator is used for cross-plane feature extraction, capturing the local correlation between adjacent planes.

[0084] (2) Attention layer operators: Attention layer operators refer to the operators of the attention layer in the image diffusion model. Attention layer operators represent the basic units that perform attention operations during the denoising process.

[0085] Optionally, the attention layer operator is an operator determined based on the constraints of synchronous attention operations. Illustratively, in this application, based on the attention layer operator, the spatial dimension of the input data of the attention layer in the image diffusion model is expanded to "M×H×W". For example, when performing an attention operation on the first plane image of the first plane, the dimension of the attention layer input data of the first plane image is expanded to "M×H×W", enabling the attention layer to perform attention operations simultaneously on all planes of the polyhedron. The attention layer operator is used to globally model the dependencies between planes (such as illumination consistency), allowing the model to dynamically focus on seam-related regions.

[0086] (3) Normalization layer operator: The normalization layer operator refers to the operator of the normalization layer in the image diffusion model. The normalization layer operator represents the basic unit that performs the normalization operation in the denoising process.

[0087] Optionally, the normalization layer operator is an operator determined based on the constraints of synchronous normalization operation. Illustratively, in this application, based on the normalization layer operator, the spatial dimension of the input data of the normalization layer in the image diffusion model is expanded to "M×H×W", enabling the normalization layer to simultaneously perform normalization operations on all planes corresponding to the multi-plane model. That is, it globally normalizes and calculates the statistics of all planes, rather than processing each plane independently. The normalization layer operator is used to unify the distribution of multi-plane features, avoiding generation bias caused by differences in feature scales across different planes.

[0088] The aforementioned scheme explicitly includes a multi-plane synchronization operator based on multi-plane synchronization conditions in the image diffusion model. Through this operator, the model can simultaneously process features from multiple planes during denoising, globally optimizing the generated results (such as at least two second-plane images corresponding to the second viewpoint and second-depth images, respectively), rather than generating isolated single planes. This effectively reduces visual discontinuities at seams. Furthermore, the image diffusion model incorporates depth information as input, further constraining geometric consistency and ensuring that the generated polyhedral textures are geometrically aligned, reducing misalignment caused by viewpoint differences.

[0089] To illustrate, taking the multi-plane synchronization operators of the image diffusion model, including convolutional layer operators, attention layer operators, and normalization layer operators, as an example, the following will combine... Figure 3 This application describes the multi-plane synchronization process. Figure 3 A schematic diagram of a denoiser for an image diffusion model is shown.

[0090] like Figure 3 As shown, the denoising mechanism of the image diffusion model includes N denoising network layers, where N is an integer greater than 1. Optionally, these N denoising network layers are used to perform the denoising process, the specific denoising process of which will be discussed later. Figure 5 The embodiments shown are described in detail here and will not be repeated. Each denoising network layer 300 includes a normalization layer, a convolutional layer, and an attention layer. The spatial dimension of the input data for the denoising network layer is "M×H×W". The input data is sequentially fed into the normalization layer, convolutional layer, and attention layer. Since the spatial dimension of the operators in the normalization layer, convolutional layer, and attention layer is "M×H×W", in this denoising network layer: During the normalization operation of the normalization layer, synchronous normalization is implemented, that is, global normalization is calculated for all planar data of the polyhedron, rather than processing each plane independently, ensuring a uniform scale of multi-planar data in the feature space. During the convolution operation of the convolutional layer, synchronous convolution is implemented, that is, for the input multi-planar feature data with a spatial dimension of "M×H×W", the pixel values ​​of adjacent planes are used to replace the traditional zero padding, eliminating discontinuities at the seams. During the attention operation of the attention layer, synchronous attention operation is implemented, that is, the attention operation can be applied to all planes simultaneously.

[0091] In some embodiments, the noise data corresponding to each second plane includes a first noise image and a second noise image. Then, using the first viewpoint image and the first depth image as denoising conditions, a denoising process is performed on the noise data under the constraint of multi-plane synchronization conditions to obtain a second viewpoint image and a second depth image corresponding to at least two second planes respectively. This includes: using the first viewpoint image and the first depth image as denoising conditions, a denoising process is performed on the first noise image and the second noise image corresponding to at least two second planes respectively under the constraint of multi-plane synchronization conditions to obtain a second viewpoint image and a second depth image corresponding to at least two second planes respectively, wherein the second viewpoint image is the image obtained by denoising the first noise image, and the second depth image is the image obtained by denoising the second noise image.

[0092] In a schematic manner, a first-view image, a first-depth image, and first and second noise images corresponding to at least two second planes are input into an image diffusion model, and the model outputs second-view images and second-depth images corresponding to at least two second planes.

[0093] Among them, at least two second-view images and a first-view image are used to form a polyhedral texture covering the polyhedron, and the polyhedral texture is used to present scene elements in multiple views corresponding to the polyhedron.

[0094] Step 240: Generate a 3D scene corresponding to scene elements based on the polyhedral texture, the first depth image, and at least two second depth images.

[0095] Optionally, the first depth image and at least two second depth images are used to form a polyhedral depth image covering the polyhedron.

[0096] In some embodiments, a polyhedral depth image and a polyhedral texture are input into a scene generator, which outputs a 3D scene corresponding to scene elements. Optionally, the scene generator can be implemented as a scene encoder-scene decoder structure, where the polyhedral depth image and the polyhedral texture are input into the scene encoder, which outputs a 3D scene feature representation; the 3D scene feature representation is input into the scene decoder, which outputs 3D point cloud data. The 3D point cloud data is used to represent the 3D scene. Schematic, the 3D point cloud data includes multiple point clouds, each storing X, Y, and Z coordinate values ​​and color information.

[0097] In some embodiments, a 3D scene can be represented by a 3D Gaussian representation, which corresponds to a 3D Gaussian function. Each 3D Gaussian function has point cloud coordinates, a covariance matrix, and color information. The point cloud coordinates of a given point cloud represent the center position (i.e., mean) of the 3D Gaussian function in 3D space. The covariance matrix controls the shape, size, and orientation of the 3D Gaussian function, describing the point cloud distribution characteristics around that point cloud. For example, a narrow Gaussian (with a small covariance matrix) can capture details (such as object edges), while a wide Gaussian (with a large covariance matrix) can cover flat areas. Color information is typically represented using RGB values.

[0098] Optionally, when a 3D scene is represented using a 3D Gaussian representation, the rendering process includes: projecting the 3D Gaussian function corresponding to all point clouds onto a 2D image plane according to the rendering viewpoint (such as camera position and camera orientation). The projection data of each point cloud is implemented as a Gaussian distribution. For each pixel on the projection plane, the contributions of all Gaussian distributions projected onto that pixel are accumulated. This is done by calculating the probability density value of each Gaussian distribution at that pixel and multiplying it by the corresponding color information (such as RGB values). Finally, the accumulated contributions of all pixels are fused to obtain the final rendered image, which is the scene image of the 3D scene from the rendering viewpoint.

[0099] Optionally, generating a 3D scene corresponding to scene elements based on a polyhedral texture, a first depth image, and at least two second depth images further includes the following steps:

[0100] Step 1: Based on the first depth image and at least two second depth images, map multiple pixels in the polyhedron map. Perform an inverse projection transformation on the points to obtain the point cloud coordinates corresponding to multiple pixels in the polyhedral texture.

[0101] Point cloud coordinates are used to represent the position of a pixel in a 3D scene.

[0102] The first depth image and at least two second depth images constitute a polyhedral depth image, which provides depth information for each pixel in the polyhedral texture. Based on the first depth image and at least two second depth images, an inverse projection transformation is performed on multiple pixels (sampled) in the polyhedral texture to obtain the point cloud coordinates corresponding to each pixel. That is, using the depth information of the polyhedral pixels, the multiple pixels in the polyhedral texture are projected into the 3D scene through inverse projection transformation to obtain the point cloud coordinates corresponding to each pixel.

[0103] To illustrate, using a cube as an example, for each pixel of each face of the cube texture, the two-dimensional pixel coordinates are converted into three-dimensional coordinates. The specific formula is: x = D × (vc x )÷f x y = D × (vc y )÷f y z = D, where (x, y, z) represent the x, y, and z values ​​of the point cloud coordinates, and (c x c y ) represents the principal point coordinates of the camera (if the camera is located at the center of the cube, then the principal point coordinates are the center coordinates of a face within the cube). If the currently calculated pixel is located on the right face of the cube, then the principal point coordinates are the coordinates of the center point of the right face. (f x f y The focal length (in pixels) is represented by ). D is the depth value of the pixel. The coordinates of the 3D coordinates are transformed to the coordinates of the 3D scene to obtain the 3D point cloud coordinates.

[0104] Step 2: Based on the point cloud coordinates and polyhedral textures corresponding to multiple pixels, construct multiple pixel maps respectively. The corresponding three-dimensional Gaussian representation.

[0105] The three-dimensional Gaussian representations corresponding to multiple pixels are used to represent a three-dimensional scene. More specifically, the three-dimensional Gaussian representations corresponding to multiple pixels are used to represent a static three-dimensional scene.

[0106] The 3D Gaussian representation includes the point cloud coordinates, covariance matrix, and color information of each pixel. The covariance matrix expresses the distribution characteristics of the pixels. Illustratively, the point cloud coordinates of the pixels in the polyhedral texture are used as the mean to obtain the pixel's covariance matrix. The covariance matrix represents the point cloud distribution characteristics near the pixel's location. A 3D Gaussian function is constructed based on the mean and the mean covariance. The pixel's color information is used as an additional parameter of the 3D Gaussian function, thus forming the 3D Gaussian representation of the pixel.

[0107] Optionally, during the construction of the 3D Gaussian representation, if the pixels of each face of the polyhedral texture are uniformly sampled when generating the point cloud through inverse projection, two problems will arise: (1) flat areas and high-frequency texture areas will use the same sampling density, wasting resources; (2) geometric edges (depth abrupt change areas) and texture detail areas will be insufficiently sampled, resulting in a decrease in reconstruction accuracy. Based on this, this application proposes an adaptive sampling strategy: Optionally, the sampling density is dynamically determined according to the geometric complexity (depth change) and / or texture richness (color change) of local areas in the polyhedral texture.

[0108] To address geometric complexity (depth variation), depth map gradient analysis is performed on each face of the polyhedron texture to detect regions of abrupt depth changes (such as object edges). The gradient of each pixel on each face of the polyhedron texture is determined, with a larger gradient indicating a more drastic depth change. The sampling weight of each pixel is determined based on the gradient, with a larger gradient indicating a higher sampling weight.

[0109] To assess texture richness (color variation), texture richness region detection is performed on each face of the polyhedron texture. The variance of RGB colors within the neighborhood of a pixel (e.g., a 3×3 window) is calculated. The larger the variance, the richer the texture. Sampling weights are determined based on the variance, with higher sampling weights for larger variances.

[0110] In some embodiments, depth map gradient analysis is performed on each face of the polyhedron texture to determine a first sampling weight for each pixel in each face of the polyhedron texture; texture-rich region detection is performed on each face of the polyhedron texture to determine a second sampling weight for each pixel in each face of the polyhedron texture; a weighted summation is performed on the first sampling weight and the second sampling determination to obtain a weighted sum as the sampling weight for each pixel in each face of the polyhedron texture.

[0111] Optionally, after determining the sampling weight, the loudness is non-uniformly sampled according to the sampling weight, and the high-weight region is selected first to generate point cloud data. That is, the "multiple pixels in the polyhedral texture" mentioned above are determined according to the sampling weight.

[0112] In the above scheme, priority sampling is given to depth discontinuities and texture-rich regions to improve the reconstruction accuracy of geometric edges and visually salient areas. Simultaneously, depth consistency constraints at the connections between planes in the polyhedral depth image eliminate duplicate points and gaps caused by projection errors.

[0113] Optionally, the scene generator can be implemented as a scene generator based on 3D Gaussian representation. Illustratively, a polyhedral depth image and a polyhedral texture are input into the 3D Gaussian representation-based scene generator, which outputs 3D Gaussian representations corresponding to multiple pixels. The 3D Gaussian representation uses 3D Gaussian functions to represent the point cloud constituting the 3D scene; each Gaussian function has point cloud coordinates, a covariance matrix, and color information.

[0114] In the above scheme, a 3D scene is generated by expressing it through a 3D Gaussian representation. The 3D Gaussian representation can model the spatial distribution characteristics of point clouds in the 3D scene through the covariance matrix, thereby improving the rendering details of the 3D scene.

[0115] In some embodiments, the process of constructing the three-dimensional Gaussian representations corresponding to multiple pixels further includes an adaptive adjustment process, and the adaptive adjustment method includes at least one of the following methods:

[0116] (1) Adaptive adjustment based on rendering error.

[0117] Optionally, based on the point cloud coordinates and polyhedral textures corresponding to multiple pixels, an initial 3D Gaussian representation corresponding to each pixel is constructed; a candidate polyhedral texture corresponding to the polyhedron is generated by projecting the initial 3D Gaussian representations corresponding to multiple pixels; based on the differences between the candidate polyhedral textures and the polyhedral textures, the initial 3D Gaussian representation is adjusted to obtain the 3D Gaussian representations corresponding to multiple pixels.

[0118] Optionally, a rendering loss is determined based on the difference between the candidate polyhedral texture and the original polyhedral texture. The rendering loss refers to the pixel-level difference between the candidate polyhedral texture and the original polyhedral texture. This pixel-level loss can be determined by at least one of the pixel error loss function, structural similarity error loss function, gradient error loss function, etc.

[0119] Optionally, based on the differences between candidate polyhedral maps and polyhedral maps, the initial 3D Gaussian representation is adjusted to obtain the 3D Gaussian representation corresponding to multiple pixels. This means: for areas in the polyhedral map with large rendering loss, the density is increased, that is, the scale of the covariance matrix of the pixels sampled in that area is reduced (making the Gaussians more concentrated) to capture details; for areas in the polyhedral map with small rendering loss, the scale of the covariance matrix of the pixels sampled in that area is increased to reduce the number of redundant Gaussians.

[0120] The aforementioned adaptive adjustment scheme based on rendering errors improves the quality of the generated 3D scene by optimizing the difference between the initial 3D Gaussian representation and the polyhedral texture, adjusting the Gaussian parameters of the initial 3D Gaussian representation, correcting local errors in model generation, and improving the quality of the generated 3D scene.

[0121] (2) Adaptive adjustment based on visual importance.

[0122] Optionally, the visual weights corresponding to multiple pixels in the polyhedral texture are determined, whereby the visual weights represent the visual importance of the pixels. Based on the point cloud coordinates, visual weights, and polyhedral textures corresponding to the multiple pixels, a three-dimensional Gaussian representation of each pixel is constructed.

[0123] Visual importance can be determined by texture richness, geometric edges, and color contrast. For texture richness, the gradient magnitude of pixels in the polyhedral texture is calculated (e.g., using the Sobel operator), with high-gradient regions (such as edges and textures) assigned higher visual weight. For geometric edges, the depth difference between adjacent pixels in the polyhedral depth image is detected, with discontinuous regions (such as object boundaries) assigned higher visual weight. For color contrast, regions with high color variation in the polyhedral depth image (such as patterns and text) are assigned higher visual weight.

[0124] In some embodiments, optionally, an initial three-dimensional Gaussian representation corresponding to each of the multiple pixels is constructed based on the point cloud coordinates and polyhedral texture corresponding to each of the multiple pixels; the initial three-dimensional Gaussian representation is adjusted according to the visual weights corresponding to each of the multiple pixels to obtain the three-dimensional Gaussian representation corresponding to each of the multiple pixels.

[0125] Optionally, adjusting the initial 3D Gaussian representation based on the visual weights corresponding to multiple pixels means: for regions with larger visual weights in the polyhedral texture, increasing the density, i.e., reducing the scale of the covariance matrix of the pixels sampled in that region (making the Gaussians more concentrated), to capture details; for regions with smaller visual weights in the polyhedral texture, increasing the scale of the covariance matrix of the pixels sampled in that region, reducing the number of redundant Gaussians.

[0126] The aforementioned adaptive adjustment scheme based on visual importance assigns higher weights to visually important regions (such as edges, textured or key feature regions), making the Gaussian distribution parameters of these regions more accurate and ensuring their detailed representation in three-dimensional space; while secondary regions are given lower weights to allow for a larger distribution range, reducing computational redundancy.

[0127] The aforementioned adaptive adjustment process automatically adjusts the local precision of the scene representation (i.e., the 3D Gaussian representation) based on rendering errors and visual importance. Sparse, large-size Gaussians are used in open areas, while dense, small-size Gaussians are deployed on complex surfaces and edge regions, optimizing the balance between representation efficiency and quality.

[0128] In some embodiments, the multi-plane synchronization conditions provided in this application can also be used to generate video.

[0129] Optionally, video frames corresponding to multiple planes of the polyhedron are obtained; video noise data corresponding to multiple planes are obtained; based on the video frames corresponding to multiple planes, a denoising process is performed on the video noise data under the constraint of multi-plane synchronization conditions to obtain scene videos corresponding to multiple planes, wherein the scene videos corresponding to different planes have different viewpoints.

[0130] To illustrate, the depth image corresponding to the video frame is obtained. Based on the video frame and depth image corresponding to multiple planes respectively, a denoising process is performed on the video noise data under the constraint of multi-plane synchronization conditions to obtain the scene video corresponding to multiple planes respectively.

[0131] Here, a video frame refers to a pre-defined static video image. Schematic, it can be obtained by acquiring multiple viewpoint images taken by the user of a certain scene element as video frames corresponding to multiple planes respectively. Alternatively, a video frame refers to the video frame obtained by rendering the above-generated 3D scene. Optionally, the 3D scene is rendered based on the viewpoints corresponding to the multiple planes of the polyhedron, resulting in video frames corresponding to the multiple planes of the polyhedron, that is, video frames corresponding to the multiple viewpoints of the polyhedron.

[0132] The specific process of performing a denoising process on video noise data under the constraint of multi-plane synchronization to obtain scene videos corresponding to multiple planes can be found in the description of performing a denoising process on noise data under the constraint of multi-plane synchronization to obtain second view images and second depth images corresponding to at least two second planes in this application, which will not be repeated here.

[0133] Optionally, the video frames corresponding to multiple planes and the video noise data corresponding to multiple planes are input into the video diffusion model, and the scene videos corresponding to multiple planes are output; the operators of the video diffusion model include multi-plane synchronization operators determined based on multi-plane synchronization conditions.

[0134] In a schematic manner, the depth image corresponding to the video frame is acquired. The video frames and depth images corresponding to multiple planes, along with the video noise data corresponding to each plane, are input into the video diffusion model, which outputs the scene video corresponding to each plane. The video diffusion model includes a multi-plane synchronization operator determined based on multi-plane synchronization conditions. Optionally, the multi-plane synchronization operator includes at least one of the convolutional layer operator, normalization layer operator, and attention layer operator in the video diffusion model.

[0135] For details on how the video frames and video noise data corresponding to multiple planes are input into the video diffusion model and output scene videos corresponding to multiple planes, please refer to the description of the image diffusion model in this application. It will not be repeated here.

[0136] In the above embodiments, by using multi-plane synchronization technology and image-to-video diffusion generation method, a panoramic dynamic video covering the entire field of view (that is, scene videos corresponding to multiple planes) is obtained, avoiding inconsistencies in video dynamics under different viewpoints, ensuring that video frames from different viewpoints change synchronously on the time axis, and avoiding dynamic seam problems.

[0137] In some embodiments, the scene element generation method provided in this application is further used to generate a dynamic 3D scene. Optionally, generating a 3D scene corresponding to scene elements based on a polyhedral texture, a first depth image, and at least two second depth images further includes the following steps:

[0138] Step 1: Construct a polyhedron texture based on the polyhedron texture, the first depth image, and at least two second depth images. The multiple pixels in the image correspond to the three-dimensional Gaussian representations.

[0139] Among them, the three-dimensional Gaussian representations corresponding to multiple pixels are used to express static three-dimensional scenes.

[0140] Optionally, based on the first depth image and at least two second depth images, an inverse projection transformation is performed on multiple pixels in the polyhedral texture to obtain point cloud coordinates corresponding to the multiple pixels in the polyhedral texture. The point cloud coordinates are used to express the scene position of the pixel in the 3D scene. Based on the point cloud coordinates corresponding to the multiple pixels and the polyhedral texture, a 3D Gaussian representation corresponding to the multiple pixels is constructed. The 3D Gaussian representation includes the point cloud coordinates, covariance matrix and color information corresponding to the pixel. The covariance matrix is ​​used to express the distribution characteristics of the pixel.

[0141] Among them, the three-dimensional Gaussian representations corresponding to multiple pixels are used to express the three-dimensional scene.

[0142] Step 2: Construct multiple pixel representations based on the 3D Gaussian representation and implicit neural representation corresponding to each pixel. The initial four-dimensional Gaussian representation of each point.

[0143] The implicit neural representation is used to characterize the positional changes of pixels over time. Schematic, the implicit neural representation is used to encode the displacement and rotational bias of the point cloud corresponding to a pixel in a 3D scene over time. The 3D Gaussian representation and the implicit neural representation can be used as the initial 4D Gaussian representation.

[0144] Step 3: Adjust the initial four-dimensional Gaussian representation based on the scene video corresponding to multiple planes to obtain multiple pixels. The four-dimensional Gaussian representation of each point.

[0145] Among them, the four-dimensional Gaussian representations corresponding to multiple pixels are used to express dynamic three-dimensional scenes.

[0146] Schematic, the four-dimensional Gaussian representation is optimized using scene videos corresponding to multiple planes, so that it matches the video frame when projected onto the faces of the polyhedron at any time stamp. Optionally, the implicit neural representation in the initial four-dimensional Gaussian representation and / or the covariance matrix in the three-dimensional Gaussian representation are adjusted based on the scene videos corresponding to multiple planes to obtain four-dimensional Gaussian representations corresponding to multiple pixels.

[0147] Optionally, a dynamic loss is determined based on the difference between the four-dimensional Gaussian representations corresponding to multiple pixels and the scene video corresponding to multiple planes; the initial four-dimensional Gaussian representation is adjusted according to the dynamic loss to obtain the four-dimensional Gaussian representations corresponding to multiple pixels.

[0148] The above scheme combines multi-plane synchronization technology and a video diffusion model. By constructing panoramic dynamic video to optimize the four-dimensional Gaussian representation, it achieves dynamic 3D scene generation. In the four-dimensional Gaussian representation, an implicit neural representation with a time dimension is introduced to model the dynamic changes of pixels (such as object movement and deformation). Combined with scene video adjustments, high-fidelity dynamic 3D reconstruction is achieved.

[0149] In some embodiments, after obtaining a three-dimensional scene, the three-dimensional scene can be rendered to obtain a two-dimensional scene image.

[0150] The following example demonstrates how a polyhedron can be implemented as a cube. Figure 4 This section introduces the process of panoramic image rendering. Figure 4 The diagram illustrates a three-dimensional scene generation process provided in this application. Please refer to it. Figure 4 After inputting the first-view image 411 and the first-depth image 412 into the image diffusion model, a cube map 421 and a cube depth image 422 are generated. The cube map 421 and the cube depth image 422 are input into the scene generator to generate a three-dimensional scene 430. For the three-dimensional scene 430, the three-dimensional scene can be rendered under a free viewpoint (that is, not limited to the orthogonal viewpoint corresponding to the stereo) to obtain a two-dimensional scene image 440.

[0151] In summary, the 3D scene generation method provided in this application, based on a first-view image and a first-depth image corresponding to a certain plane of a polyhedron, performs a denoising process on noisy data under the constraint of multi-plane synchronization conditions to obtain second-view images and second-depth images corresponding to other planes of the polyhedron. The second-view images and the first-view images are used to construct a polyhedron texture covering the polyhedron. A 3D scene corresponding to scene elements can be generated through the polyhedron texture, the first-depth image, and the second-depth image. During the denoising process, the constraints of multi-plane synchronization conditions allow for the simultaneous analysis of multiple planes corresponding to the polyhedron. This enables the global optimization of the generation results of all planes during the denoising process, rather than processing a single plane in isolation, thereby reducing the problem of inconsistent seams. For example, when generating a second-view image corresponding to a certain plane, the generation results of the first-view image and other planes are referenced simultaneously to ensure that the visual information at the seams transitions naturally between adjacent planes.

[0152] In some embodiments, the denoising process described above is implemented using an image diffusion model, which includes an image encoder, a denoiser, and an image decoder. For illustrative examples, please refer to... Figure 5 The above Figure 2 The illustrated embodiment also includes steps 510 to 540.

[0153] Step 510: Obtain the first-view image corresponding to the scene element and the first depth image corresponding to the first-view image.

[0154] In this context, the first-view image covers the first plane of the polyhedron, and different planes of the polyhedron correspond to different perspectives of the 3D scene. The first plane corresponds to the first perspective, which means that the scene elements in the first-view image can be regarded as scene elements presented from the first perspective.

[0155] Step 520: Obtain noise data corresponding to at least two second planes respectively.

[0156] The second plane refers to any plane in a polyhedron other than the first plane.

[0157] Noise data refers to random noise, such as a noisy image, where each pixel is randomly sampled from a random distribution (such as a Gaussian distribution). In a diffusion model, the generation process typically begins with completely random noise data, which serves as the initial input and is progressively denoised through multiple denoising network layers. Optionally, the image diffusion model also includes a noise encoder, which takes the noise data as input and outputs a noise feature representation.

[0158] Optionally, each second plane corresponds to a first noise image and a second noise image. The value of each pixel in the first noise image represents color information. The value of each pixel in the second noise image represents depth information. The first noise image and the second noise image can be the same or different; this is not limited here.

[0159] It should be noted that the dimensions of the first noisy image, the second noisy image, the first viewpoint image, and the first depth image are all the same, for example, 640x480 pixels. The pixels of the first noisy image, the second noisy image, the first viewpoint image, and the first depth image correspond one-to-one; that is, pixels at the same position in the first noisy image, the second noisy image, the first viewpoint image, and the first depth image (such as the pixel in the i-th row and j-th column) correspond to the same position in the 3D scene.

[0160] Step 531: Input the first viewpoint image into the image encoder and output the image feature representation; input the first depth image into the image encoder and output the depth feature representation.

[0161] The image encoder can be implemented as at least one of the following: a Transformer-based encoder, a convolutional neural network-based encoder, etc., without limitation.

[0162] Optionally, the image encoder includes a multi-plane synchronization operator determined based on multi-plane synchronization conditions. The multi-plane synchronization operator includes at least one of convolutional layer operators, attention layer operators, and normalization layer operators. Schematic, the image encoder may be referred to as a synchronous image encoder.

[0163] In some embodiments, a first-view image is input to a first image encoder, which outputs an image feature representation; a first-depth image is input to a second image encoder, which outputs a depth feature representation. The first image encoder and the second image encoder may be the same or different.

[0164] Optionally, normalization is performed on the first depth image to obtain a normalized image; the normalized image is then input into an image encoder to output a depth feature representation. Optionally, performing normalization on the first depth image means normalizing the depth values ​​of each pixel in the first depth image. Illustratively, the depth values ​​of each pixel in the first depth image are scaled to the range [-s, +s] to create additional margin for depth generation in other planes and avoid out-of-domain depth values.

[0165] In the above scheme, the depth image is normalized before being input into the encoder. The normalized depth data is easier to integrate with the image features, thus improving the geometric accuracy of the generated results.

[0166] Step 532: Generate input feature representation based on image feature representation, depth feature representation, and noise data corresponding to at least two second planes.

[0167] In some embodiments, image latent variables are generated based on image feature representations and first noise images corresponding to at least two second planes respectively; depth latent variables are generated based on depth feature representations and second noise images corresponding to at least two second planes respectively; and input feature representations are generated based on image latent variables and depth latent variables.

[0168] In the above scheme, noise data is divided into image noise and depth noise. After generating latent variables, they are fused. On the one hand, the noise of the image and depth is processed independently to avoid mutual interference. On the other hand, through joint modeling of image latent variables and depth latent variables, the geometric and visual consistency between the image generated by the image diffusion model and the depth data is ensured.

[0169] Optionally, generating image latent variables based on image feature representations and first noise images corresponding to at least two second planes includes: generating image noise feature representations of the first noise images corresponding to at least two second planes; and fusing the image feature representations and the image noise feature representations corresponding to at least two second planes to obtain image latent variables. Optionally, the first noise image is input into a noise encoder to obtain image noise feature representations. Illustratively, the image latent variables are obtained by concatenating the image feature representations and the image noise feature representations corresponding to at least two second planes; or, the image latent variables are obtained by fusing the image feature representations and the image noise feature representations corresponding to at least two second planes based on an attention operation.

[0170] Optionally, generating deep latent variables based on the deep feature representations and the second noise images corresponding to at least two second planes includes: generating deep noise feature representations of the second noise images corresponding to at least two second planes; and fusing the deep feature representations and the deep noise feature representations corresponding to at least two second planes to obtain the deep latent variables. Optionally, the second noise images are input into a noise encoder to obtain the deep noise feature representations. Illustratively, the deep feature representations and the deep noise feature representations corresponding to at least two second planes are concatenated to obtain the deep latent variables, or, alternatively, the deep feature representations and the deep noise feature representations corresponding to at least two second planes are fused based on an attention operation to obtain the deep latent variables.

[0171] In some embodiments, generating an input feature representation based on image latent variables and depth latent variables includes: obtaining a positional feature representation, wherein the positional feature representation characterizes the positional information of the pixels corresponding to the polyhedron; and generating an input feature representation based on the image latent variables, depth latent variables, and positional feature representation.

[0172] In the above scheme, positional feature representation is introduced as input to the image diffusion model. Positional feature representation represents the positional information of the pixels of the polyhedron. Through explicit positional information, the image diffusion model can accurately control the spatial position of the multi-plane generation result, reduce geometric misalignment caused by viewpoint transformation, and the introduction of positional information can guide feature fusion at the seam and improve the coherence of adjacent planes.

[0173] Optionally, obtaining the position feature representation includes: obtaining the three-dimensional coordinates of the pixels corresponding to the multiple planes of the polyhedron; projecting the three-dimensional coordinates onto a unit sphere to obtain normalized coordinates; and determining the position feature representation based on the normalized coordinates of the pixels corresponding to the multiple planes of the polyhedron.

[0174] To illustrate, to ensure the geometric consistency of the generated polyhedral texture and the coherence of object relationships between different planes, the positional feature representation can be used as the input feature representation of the image diffusion model (i.e., as a denoising condition). Specifically, for each pixel on each plane of the polyhedron, its 3D coordinates are projected onto a unit sphere, and the resulting coordinate values ​​are normalized to [-1, 1]. These normalized coordinates are then added as three additional channels to the image latent variables and depth latent variables, such as by stitching together the image latent variables, depth latent variables, and positional feature representation to obtain the input feature representation.

[0175] In the above scheme, the position feature representation is obtained by projecting three-dimensional coordinates onto a unit sphere. The normalized coordinates unify the spatial reference system of multiple planes, which facilitates the fusion of cross-plane features of the model. Furthermore, the normalized coordinates also reduce the calculation error caused by the difference in coordinate systems.

[0176] In some embodiments, generating an input feature representation based on image latent variables and depth latent variables includes: obtaining a mask feature representation, the mask feature representation characterizing the position of a first plane in a polyhedron; and generating the input feature representation based on the image latent variables, depth latent variables, and the mask feature representation. Illustratively, for the mask feature representation, the feature value of the pixel corresponding to the first plane is 0, and the feature value of the pixel corresponding to other planes is 1.

[0177] In the above scheme, a mask feature representation is introduced to identify the position of the known first plane, thereby guiding the model to generate only the unknown plane (the second plane), reducing redundant calculations and improving efficiency.

[0178] Optionally, an input feature representation is generated based on image latent variables, depth latent variables, positional feature representations, and mask feature representations. Schematically, the image latent variables, depth latent variables, positional feature representations, and mask feature representations are concatenated to obtain the input feature representation.

[0179] Step 533: Input the input feature representation into the denoiser to obtain the denoising result.

[0180] Optionally, the denoiser's operators include a multi-plane synchronization operator determined based on multi-plane synchronization conditions. The multi-plane synchronization operator includes at least one of convolutional layer operators, attention layer operators, and normalization layer operators; schematically, the denoiser may be referred to as a synchronous denoiser.

[0181] In some embodiments, the denoiser includes N denoising network layers, where N is an integer greater than 1.

[0182] Optionally, the input feature representation is input into N denoising network layers to obtain the denoising result; wherein, the (i+1)th denoising network layer is used to obtain the output feature representation of the (i+1)th denoising network layer based on the output feature representation of the ith denoising network layer, where i is a positive integer and i≤N.

[0183] For N denoising network layers: The N denoising network layers progressively remove "noise" from the input feature representation. Optionally, the image diffusion model corresponds to T time steps (T is an integer greater than 1). At each time step, the following denoising process is performed: The (i+1)th denoising network layer obtains the output feature representation of the ith denoising network layer and removes the "noise" from the output feature representation of the ith denoising network layer, obtaining the output feature representation of the (i+1)th denoising network layer, and finally obtaining the denoising result of the last denoising network layer (i.e., the Nth denoising network layer). If the current denoising process corresponds to the last time step, the denoising result of the last denoising network layer is used as the final output denoising result (used as input to the image decoder); if the current denoising process does not correspond to the last time step, the denoising result of the last denoising network layer is used as a new input feature representation and re-inputted into the N denoising network layers, starting the denoising process corresponding to the next time step.

[0184] Optionally, each denoising network layer includes a convolutional layer and an attention layer. In the (i+1)th denoising network layer, the output feature representation of the ith denoising network layer is input into the convolutional layer to obtain a convolutional feature representation, and the convolutional feature representation is input into the attention layer to obtain an attention feature representation, which is then used as the output feature representation of the (i+1)th denoising network layer. Specifically, if i = 1 and the current time step corresponds to the denoising process, in the ith denoising network layer, the input feature representation is input into the convolutional layer to obtain a convolutional feature representation, and the convolutional feature representation is input into the attention layer to obtain an attention feature representation, which is then used as the output feature representation of the ith denoising network layer. If i = N and the current time step corresponds to the denoising process, in the nth denoising network layer, the output feature representation of the (N-1)th denoising network layer is input into the convolutional layer to obtain a convolutional feature representation, and the convolutional feature representation is input into the attention layer to obtain an attention feature representation, which is then used as the output feature representation of the nth denoising network layer. The output feature representation of the nth denoising network layer is also the final denoised result (used as input to the image decoder).

[0185] In some embodiments, the denoising conditions of the input denoiser further include scene cue information, which is used to indicate the scene content of the 3D scene. Optionally, the scene cue information may be in the form of at least one of text, image, audio, video, etc. Illustratively, the scene cue information can be implemented as scene description text, which is used to describe the scene content of the 3D scene in text form. Optionally, the scene description text includes description text corresponding to multiple viewpoints of a polyhedron, with the description text for different viewpoints used to describe the scene content of the 3D scene from different perspectives.

[0186] The above scheme introduces scene cue information as a denoising condition. For example, scene cue information (such as scene description text) is added to the input conditions of the denoiser to describe the overall content of the 3D scene or specify the details of the polyhedron from different perspectives. This improves the accuracy and detail richness of the generated results, while avoiding the semantic ambiguity and content limitations caused by single-view images.

[0187] Optionally, a scene feature representation corresponding to the scene prompt information is generated. For example, taking the scene prompt information as scene description text, the scene description text is input into a text encoder, and the output text feature representation is used as the scene feature representation.

[0188] Optionally, the input feature representation and scene feature representation are fed into a denoiser to obtain the denoised result. Illustratively, the scene feature representation is input into each of the N denoising network layers, such as into the attention layer of each denoising network layer. In each attention layer, the convolutional feature representation and scene feature representation are fused based on an attention operation to obtain the output feature representation.

[0189] Step 534: Input the denoising result into the image decoder and output at least two second-view images and second-depth images corresponding to the second plane respectively.

[0190] The image decoder can be implemented as at least one of the following: a Transformer-based decoder, a convolutional neural network-based decoder, etc., without limitation here.

[0191] Optionally, the image decoder includes a multi-plane synchronization operator determined based on multi-plane synchronization conditions. The multi-plane synchronization operator includes at least one of convolutional layer operators, attention layer operators, and normalization layer operators. Schematic, the image decoder may be referred to as a synchronized image decoder.

[0192] In some embodiments, if the noise data includes a first noise image and a second noise image, the final output denoising result includes the image denoising result corresponding to the first noise image and the depth denoising result corresponding to the second noise image.

[0193] Optionally, the image denoising result is input into an image decoder, which outputs at least two second-view images corresponding to the second planes respectively. The at least two output second-view images can be implemented as separate images, or the image decoder can output a complete polyhedral texture obtained by combining the at least two second-view images with the first-view image.

[0194] Optionally, the depth denoising result is input into an image decoder, which outputs at least two second depth images corresponding to the second planes respectively. The at least two output second depth images can be implemented as separate images, or the image decoder can output a complete polyhedral depth image obtained by combining the at least two second depth images with the first depth image.

[0195] Step 540: Generate a 3D scene corresponding to scene elements based on the polyhedral texture, the first depth image, and at least two second depth images.

[0196] Among them, at least two second-view images and a first-view image are used to form a polyhedral texture covering the polyhedron, and at least two second-depth images and a first-depth image are used to form a polyhedral depth image covering the polyhedron. The polyhedral texture is used to present scene elements in multiple views corresponding to the polyhedron.

[0197] In some embodiments, a polyhedral depth image and a polyhedral texture are input into a scene generator, and a 3D scene corresponding to the scene elements is output.

[0198] In summary, the 3D scene generation method provided in this application generates a second-view image and a second-depth image through an image diffusion model architecture of image encoder-denoiser-decoder. The image encoder extracts features from the image and depth respectively, avoiding interference between the two types of information. The denoiser gradually removes noise, improving the denoising effect. The image decoder realizes the process from the denoised result to the reconstructed image. In the entire image diffusion model architecture, the consistency of multiple planes is constrained by multi-plane synchronization conditions, thereby globally optimizing the generation results of all planes during the denoising process, rather than processing a single plane in isolation, reducing the problem of inconsistent seams.

[0199] In some embodiments, the above image diffusion model is a trained model. The training process of the image diffusion model is described below for illustrative purposes. Please refer to [the provided text]. Figure 6 , Figure 6 This application provides a flowchart illustrating a training method for an image diffusion model, which is executed by a computer device. The computer device can be... Figure 1 The terminal 110 and / or server 120, or computer equipment shown, may be other devices specifically designed for model training, and are not limited herein. The method includes steps 601 to 605 as follows.

[0200] Step 601: Obtain the sample polyhedron texture and the corresponding sample polyhedron depth image.

[0201] The sample polyhedral texture refers to the polyhedral texture corresponding to the sample 3D scene. It is an image covering a polyhedron, and includes sample view images corresponding to multiple planes of the polyhedron. Different planes of the polyhedron represent different viewpoints. Specifically, the sample polyhedral texture includes the sample view image corresponding to the first plane and sample view images corresponding to at least two second planes.

[0202] The sample polyhedron depth image includes the sample depth image corresponding to the first plane and the sample depth images corresponding to at least two second planes respectively.

[0203] Step 602: Perform noise addition processing on the sample view images corresponding to at least two second planes in the sample polyhedron texture to obtain latent variables of the sample images; perform noise addition processing on the sample depth images corresponding to at least two second planes in the sample polyhedron depth image to obtain latent variables of the sample depth.

[0204] Optionally, the sample polyhedral texture is input into the image encoder to obtain the sample image feature representation; the sample polyhedral depth image is input into the image encoder to obtain the sample depth feature representation; noise processing is performed on the feature representations of the sample view images corresponding to at least two second planes in the sample image feature representation to obtain the sample image latent variables; noise processing is performed on the feature representations of the sample depth images corresponding to at least two second planes in the sample polyhedral depth image to obtain the sample noise latent variables.

[0205] To illustrate, noise is progressively added to the feature representations of the sample viewpoints corresponding to at least two second planes in the sample image feature representation, resulting in image noise-added data for T time steps. Here, image noise-added data refers to the noise added to the feature representations of the sample viewpoints corresponding to at least two second planes in the sample image feature representation. To illustrate, noise is added to the feature representations of the sample viewpoints corresponding to at least two second planes in the sample image feature representation at T time steps, until the feature representations of the sample viewpoints corresponding to at least two second planes become pure noise data (Gaussian noise following a normal distribution). The noise added to the feature representations of the sample viewpoints corresponding to at least two second planes at each time step is saved, where T is an integer greater than 1.

[0206] Illustratively, noise is progressively added to the feature representations of the sample depth images corresponding to at least two second planes in the sample polyhedral depth image, resulting in depth noise-added data for T time steps. Here, depth noise-added data refers to the noise added to the feature representations of the sample depth images corresponding to at least two second planes in the sample polyhedral depth image. Illustratively, noise is added to the feature representations of the sample depth images corresponding to at least two second planes in T time steps until the feature representations of the sample depth images corresponding to at least two second planes become pure noise data (Gaussian noise conforming to a normal distribution). The noise added to the feature representations of the sample depth images corresponding to at least two second planes at each time step is saved.

[0207] Optionally, the sample polyhedral depth image is normalized to obtain a sample normalized image; the sample normalized image is then input into an image encoder to output a sample depth feature representation.

[0208] Step 603: Input the latent variables of the sample image and the latent variables of the sample depth into the sample denoiser, and output the image denoising result and the depth denoising result.

[0209] Optionally, the operators in the sample denoising unit include a multi-plane synchronization operator determined based on multi-plane synchronization conditions. The multi-plane synchronization operator includes at least one of convolutional layer operators, attention layer operators, and normalization layer operators.

[0210] Optionally, the latent variables of the sample image, the latent variables of the sample noise, the sample location feature representation, and the mask feature representation are concatenated to obtain the sample input feature representation; the sample input feature representation is then input into the sample denoiser to output the image denoising result and the depth denoising result.

[0211] The sample position feature representation characterizes the position information of the pixels corresponding to the polyhedron. Optionally, obtaining the sample position feature representation includes: obtaining the three-dimensional coordinates of the pixels corresponding to the multiple planes of the polyhedron; projecting the three-dimensional coordinates onto a unit sphere to obtain normalized coordinates; and determining the sample position feature representation based on the normalized coordinates of the pixels corresponding to the multiple planes of the polyhedron.

[0212] Mask features represent the position of the first plane in a polyhedron.

[0213] Optionally, the sample input feature representation corresponding to the g-th time step is input into the sample denoising network layer, and the image denoising result and depth denoising result at the g-th time step are output. Illustratively, the sample input feature representation corresponding to the g-th time step is input into N sample denoising network layers, and the image denoising result and depth denoising result at the g-th time step are output. g is a positive integer, and g ≤ N.

[0214] In some embodiments, the denoising conditions of the input sample denoiser further include sample scene prompts, illustratively, whereby the sample scene prompts are used to indicate the scene content of the 3D scene. Optionally, the sample scene prompts may take the form of at least one of text, images, audio, and video. Illustratively, the sample scene prompts may be implemented as scene description text, which is used to describe the scene content of the 3D scene in text form. Optionally, the sample scene description text includes sample description text corresponding to multiple viewpoints of a polyhedron, where the sample description texts from different viewpoints are used to describe the scene content of the 3D scene from different viewpoints.

[0215] Optionally, a sample scene feature representation corresponding to the sample scene prompt information is generated. For example, taking the sample scene prompt information as sample scene description text, the sample scene description text is input into a text encoder, and the output sample text feature representation is used as the sample scene feature representation.

[0216] Optionally, the sample input feature representation and the sample scene feature representation are input into the sample denoising network layer, which outputs the image denoising result and the depth denoising result. Illustratively, the sample scene feature representation is input into each of the N sample denoising network layers.

[0217] Step 604: Determine the first denoising loss based on the image denoising result, and determine the second denoising loss based on the depth denoising result.

[0218] In some embodiments, noise is added to the predicted image at the g-th time step based on the image denoising result at the g-th time step, where the noise added to the predicted image refers to the noise added to the image denoising result at the g-th time step predicted by the N-sample denoising network layer. A first denoising loss is determined based on the difference between the noise added to the predicted image at the g-th time step and the noise added data to the image at the g-th time step.

[0219] In some embodiments, the predicted depth noise at the g-th time step is determined based on the depth denoising result at the g-th time step, wherein the predicted depth noise refers to the noise added to the depth denoising result at the g-th time step predicted by the N-sample denoising network layer. A second denoising loss is determined based on the difference between the predicted depth noise at the g-th time step and the depth noise-added data at the g-th time step.

[0220] The loss function used to calculate the first denoising loss and the second denoising loss includes at least one of the following: KL loss function, mean squared error loss function, etc.

[0221] Step 605: Train the sample denoiser based on the first denoising loss and the second denoising loss to obtain the image diffusion model.

[0222] Optionally, after obtaining the first denoising loss and the second denoising loss, the parameters of the N-sample denoising network layer are updated with the goal of minimizing the first denoising loss and the second denoising loss. Illustratively, the sum or weighted sum of the first denoising loss and the second denoising loss is calculated as the denoising loss, and the parameters of the N-sample denoising network layer are updated with the goal of minimizing the denoising loss.

[0223] The above training process is executed iteratively to complete one training process for a sample data pair (i.e., the sample polyhedral texture and the sample polyhedral depth image). The above training process is iterated multiple times for multiple sample data pairs until the sum (or weighted sum) of the calculated first denoising loss and the second denoising loss is less than the preset loss, or the number of training times reaches the preset number. If it is determined that the sample denoiser meets the training requirements, then training is stopped. The N sample denoising network layers obtained at this time are the N trained denoising network layers, which are also the denoiser.

[0224] An image diffusion model is constructed based on a denoiser, an image encoder, and an image decoder.

[0225] It should be noted that, regarding Figure 6 For details not elaborated in the training process of the image diffusion model described in the article, please refer to [link / reference]. Figure 2 or Figure 5 The introduction to image diffusion models in the previous section will not be repeated here.

[0226] The training and inference processes of the image diffusion model provided in this application will be explained below, taking a cube implementation as an example.

[0227] First, it should be noted that the image diffusion model proposed in this application applies a multi-plane synchronization condition. Based on the multi-plane synchronization condition, the spatial operators of the neural network of the image diffusion model can be comprehensively modified, ensuring translational equivalence in the omnidirectional image domain. Specifically, for the image diffusion model, this application modifies the following key operators:

[0228] Attention layer operator (i.e., synchronous attention mechanism): The input label of the attention layer is shaped into (M×H×W)×C, enabling the attention mechanism to operate simultaneously on all planes. C represents the number of channels.

[0229] Convolutional layer operator (also known as synchronous 2D convolution): Replaces zero-padding with adjacent face pixels of geometric projection to eliminate discontinuities at the seams of the cube map.

[0230] Normalization layer operator (also known as synchronization group normalization): calculates statistics for all planes globally, rather than processing each view independently.

[0231] This comprehensive and systematic operator adaptation method enables pre-trained 2D diffusion models to be directly applied to omnidirectional image (i.e., cube maps and cube depth images) generation, creating high-quality panoramic content with consistent seams. By replacing the operators of all attention layers (such as self-attention layers), convolutional layers (such as 2D convolutional layers), and normalization layers in the image diffusion model with multi-plane synchronous operators, the problems of discontinuous seams and inconsistent color tones in cube map generation are effectively solved.

[0232] ●Training process

[0233] This is illustrative; please refer to it. Figure 7 , Figure 7 A flowchart illustrating the training process of an image diffusion model provided in an embodiment of this application is shown.

[0234] Step 701: Obtain the sample cube map and the corresponding sample cube depth image.

[0235] The sample cube map refers to the cube map corresponding to the sample 3D scene. It is an image covering a cube, and includes sample viewpoint images corresponding to multiple planes of the cube. Different planes of the cube represent different viewpoints. Specifically, the sample cube map includes the sample viewpoint image corresponding to the first plane and sample viewpoint images corresponding to at least two second planes.

[0236] The sample cube depth image includes the sample depth image corresponding to the first plane and the sample depth images corresponding to at least two second planes. The at least two second planes are the other planes in the cube besides the first plane.

[0237] Step 702: Input the sample cube map into the image encoder to obtain the sample image feature representation; input the sample cube depth image into the image encoder to obtain the sample depth feature representation.

[0238] Figure 8 A schematic diagram illustrating the training process of an image diffusion model provided in this application is shown.

[0239] This is illustrative; please refer to it. Figure 8 The sample cube map 801 is input into the image encoder 802 to obtain the sample image feature representation; the sample cube depth image 803 is input into the image encoder 804 to obtain the sample depth feature representation.

[0240] Optionally, normalization processing is performed on the sample cube depth image 803 to obtain a sample normalized image; the sample normalized image is then input into the image encoder 804 to obtain the sample depth feature representation.

[0241] Step 703: Perform noise addition processing on the feature representations of the sample view images corresponding to at least two second planes in the sample image feature representation to obtain the latent variables of the sample images; perform noise addition processing on the feature representations of the sample depth images corresponding to at least two second planes in the sample cube depth image to obtain the latent variables of the sample noise.

[0242] This is illustrative; please refer to it. Figure 8 Based on the first noise data 805, noise processing is performed on the feature representations of the sample view images corresponding to at least two second planes in the sample image feature representation to obtain the sample image latent variable 806. Based on the second noise data 807, noise processing is performed on the feature representations of the sample depth images corresponding to at least two second planes in the sample cube depth image to obtain the sample noise latent variable 808.

[0243] Step 704: Concatenate the latent variables of the sample image, the latent variables of the sample noise, the sample location feature representation, and the mask feature representation to obtain the sample input feature representation.

[0244] This is illustrative; please refer to it. Figure 8 The latent variable 806 of the sample image, the latent variable 808 of the sample noise, the sample position feature representation 809, and the mask feature representation 810 are spliced ​​together to obtain the sample input feature representation.

[0245] Step 705: Input the sample feature representation into the sample denoiser and output the image denoising result and the depth denoising result.

[0246] This is illustrative; please refer to it. Figure 8 The sample input feature representation is input into the sample denoiser 811, and the output image denoising result 812 and depth denoising result 813. Optionally, the sample input features and sample scene prompt information are used as denoising conditions input into the sample denoiser 811, and the output image denoising result and depth denoising result are output.

[0247] Step 706: Determine the first denoising loss based on the image denoising result, and determine the second denoising loss based on the depth denoising result.

[0248] This is illustrative; please refer to it. Figure 8 Based on the image denoising result 812, a first denoising loss 814 is determined; based on the depth denoising result 813, a second denoising loss 815 is determined. The specific process of determining the first denoising loss and the second denoising loss can be referred to step 604, and will not be elaborated here.

[0249] Step 707: Train the sample denoiser based on the first denoising loss and the second denoising loss to obtain the denoiser.

[0250] This is illustrative; please refer to it. Figure 8The model parameters in the sample denoiser 811 are updated based on the first denoising loss 814 and the second denoising loss 815. The training process is iterated multiple times until the sample denoiser 811 meets the training requirements and the training is stopped. The sample denoiser 811 obtained at this time is also the denoiser.

[0251] Step 708: Construct an image diffusion model based on the denoiser, image encoder, and image decoder.

[0252] Image encoder 802 and image encoder 804 share weights. Optionally, image encoder 802 and image encoder 804 can be the same encoder.

[0253] ● Reasoning process

[0254] This is illustrative; please refer to it. Figure 9 , Figure 9 A flowchart illustrating the inference process of an image diffusion model provided in an embodiment of this application is shown.

[0255] Step 901: Obtain the first-view image corresponding to the scene element and the first depth image corresponding to the first-view image.

[0256] The first-view image covers the first plane of the cube, and the first plane corresponds to the first viewpoint.

[0257] Step 902: Obtain the first noise image and the second noise image corresponding to the other planes of the cube.

[0258] Other planes refer to the planes in the cube other than the first plane.

[0259] Step 903: Input the first viewpoint image into the image encoder and output the image feature representation; input the first depth image into the image encoder and output the depth feature representation.

[0260] Figure 10 A schematic diagram illustrating the reasoning process of an image diffusion model provided in this application is shown.

[0261] This is illustrative; please refer to it. Figure 10 The first-view image 1001 is input into the image encoder 1002, and the image feature representation is output; the first-depth image 1003 is input into the image encoder 1004, and the depth feature representation is output.

[0262] Optionally, normalization processing is performed on the first depth image 1003 to obtain a normalized image; the normalized image is input into the image encoder 1004 to output a depth feature representation.

[0263] Step 904: Generate image latent variables based on image feature representation and the first noisy image; generate depth latent variables based on depth feature representation and the second noisy image.

[0264] This is illustrative; please refer to it. Figure 10 Based on the image feature representation and the first noise image corresponding to the other planes respectively, image latent variable 1005 is generated; based on the depth feature representation and the second noise image corresponding to the other planes respectively, depth latent variable 1006 is generated.

[0265] Step 905: Concatenate the image latent variables, depth latent variables, sample location feature representations, and mask feature representations to obtain the input feature representation.

[0266] This is illustrative; please refer to it. Figure 10 The image latent variable 1005, depth latent variable 1006, sample location feature representation 1007, and mask feature representation 1008 are spliced ​​together to obtain the input feature representation.

[0267] Step 906: Input the input feature representation into the denoiser to obtain the denoising result.

[0268] This is illustrative; please refer to it. Figure 10 The input feature representation is input to the denoiser 1009, and iterative denoising (i.e., gradually removing noise) is performed to obtain the denoising result.

[0269] Optionally, the scene prompts and input feature representations are input into the denoiser to obtain the denoising result.

[0270] Step 907: Input the denoising result into the image decoder and output the second view image and the second depth image corresponding to the other planes of the cube.

[0271] The second-view and first-view images corresponding to the other planes of the cube are used to construct a cube map covering the cube. This cube map is used to render scene elements from multiple viewpoints corresponding to the cube. The second-depth and first-depth images corresponding to the other planes of the cube are used to construct a cube depth image covering the cube.

[0272] This is illustrative; please refer to it. Figure 10 The denoising result is input into the image decoder 1010, which outputs a cube map 1011 and a cube depth image 1012.

[0273] Step 908: Generate a 3D scene corresponding to scene elements based on the cube map, the first depth image, and the second depth images corresponding to the other planes of the cube.

[0274] To illustrate, the cube map 1011 and the cube depth image 1012 are input into the scene generator to generate a 3D scene corresponding to the scene elements.

[0275] Optionally, the cube map 1011 and cube depth image 1012 generated in this application can be used to reconstruct a 3D scene expressed in a 3D Gaussian representation. Illustratively, the cube map 1011 and cube depth image 1012, using the depth information of the six faces in the cube depth image 1012, project each pixel of the cube map 1011 into 3D space through inverse projection transformation to form an initial point cloud. Then, sampling is performed on the initial point cloud. This application employs an adaptive sampling strategy, prioritizing sampling of depth discontinuities and texture-rich regions to improve the reconstruction accuracy of geometric edges and visually salient regions. The sampled point cloud represents the 3D scene, where the data of each point cloud (such as point cloud position, color information, and covariance matrix) can be stored in a 3D Gaussian representation. In addition, this application also samples the point cloud in the 3D scene with dynamic density adaptive adjustment. Schematic, the local accuracy of the 3D Gaussian representation of the point cloud is automatically adjusted according to rendering error and visual importance. Sparse large-size Gaussian is used in open areas, while dense small-size Gaussian is deployed in complex surfaces and edge areas, thus optimizing the balance between representation efficiency and quality.

[0276] In some embodiments, this application can also render a 3D scene expressed in 3D Gaussian representation as six static videos (i.e., streaming video frames) corresponding to a cube perspective, and use a multi-plane synchronization conditional image-to-video diffusion model (i.e., the aforementioned video diffusion model) to animate the six static videos, resulting in six dynamic videos from the cube perspective. This scheme combines multi-plane synchronization conditions and the image-to-video diffusion generation method to obtain panoramic dynamic videos covering the entire field of view, avoiding inconsistencies in video dynamics from different perspectives.

[0277] In some embodiments, this application can also combine 3D Gaussian representation and implicit neural representation to obtain 4D Gaussian representation, and optimize the 4D Gaussian representation using panoramic dynamic video data (such as six dynamic videos from a cube perspective generated by a video diffusion model) to obtain a dynamic 3D scene. The highly compressed implicit neural representation is used to encode the displacement and rotation bias of each 3D Gaussian representation over time, improving the storage efficiency of the dynamic 3D scene and reducing the optimization difficulty. This method combines multi-plane synchronization technology and a video diffusion model, and optimizes the 4D Gaussian representation by constructing panoramic dynamic videos, realizing the generation of a dynamic panoramic 3D environment and compensating for the shortcomings of static 3D scenes.

[0278] In some embodiments, a multimodal driven 3D scene generation system can be constructed according to the 3D model generation method provided in this application. This 3D scene generation system supports users to quickly construct personalized 3D spaces (corresponding to the 3D scenes) through natural language descriptions (corresponding to the scene description text above) and / or scene images (corresponding to the first-view images above), and realize free exploration and interaction.

[0279] Figure 11 This diagram illustrates the product architecture of a 3D scene generation system provided in this application. It is illustrative; please refer to the original text. Figure 11 The product architecture of the 3D scene generation system includes the following parts:

[0280] (1) The user inputs 1110.

[0281] like Figure 11 As shown, user input consists of scene description text and / or scene images.

[0282] (2) Scene generation 1120.

[0283] like Figure 11 As shown, the user input is evaluated for its type. If it is a scene image, image scene understanding is performed on the scene image, and then 3D scene generation is performed. The generation method in "performing 3D scene generation" is the same as the 3D scene generation method provided in this application. If it is scene description text, semantic parsing is performed on the scene description text, and then 3D scene generation is performed. If it is both a scene image and scene description text, 3D scene generation can be performed directly.

[0284] (3) Interactive system 1130.

[0285] like Figure 11 As shown, the final generated 3D scene is input into the real-time rendering engine for rendering. The rendered scene can be used for user interaction. For example, users can interact with the 3D scene through keyboard, touch screen and other interactive methods. Taking keyboard commands as an example, users can use the WASD keys to roam the 3D scene. The interactive system 1130 controls the camera based on keyboard commands, thereby presenting the 3D scene to the user from different perspectives.

[0286] The aforementioned 3D scene generation system supports indoor and outdoor, realistic and non-realistic scene generation, allowing users to construct their own 3D scene spaces and create personalized gameplay.

[0287] Figure 12 This diagram illustrates the technical architecture of a 3D scene generation system provided in this application. It is illustrative; please refer to the original text. Figure 12 The technical architecture of the 3D scene generation system includes the following parts:

[0288] (1) The user inputs 1210.

[0289] like Figure 12 As shown, user input consists of scene description text and / or scene images.

[0290] (2) Core processing flow 1220.

[0291] like Figure 12 As shown, the system performs input type determination on user input. If it is a scene image, visual understanding is performed on the scene image to obtain scene description text (e.g., using a large language model to understand the semantic information in the scene image to obtain scene description text). The scene description text and scene image are then input into the image diffusion model to output a 3D scene. If it is scene description text, natural language parsing is performed on the scene description text to generate a scene image (e.g., using a text-to-image diffusion model to generate a scene front view, where the front view is the perspective image corresponding to the front side of a cube). The scene description text and scene image are then input into the image diffusion model to output a 3D scene. If it is a scene image and scene description text, both can be directly input into the image diffusion model to output a 3D scene.

[0292] (3) Interactive rendering 1230.

[0293] like Figure 12 As shown, the final generated 3D scene is input into a real-time rendering engine for rendering. The rendered scene can be used for user interaction. For example, users can trigger control signals through keyboards, touch screens, etc., and control the rendering perspective of the 3D scene through the control signals, thereby realizing the user's scene roaming.

[0294] This is illustrative; please refer to it. Figure 13 It shows a structural block diagram of a three-dimensional scene generation device, such as Figure 13 As shown, the device includes:

[0295] The acquisition module 1310 is used to acquire a first-view image corresponding to a scene element and a first-depth image corresponding to the first-view image, wherein the first-view image covers the first plane of the polyhedron and the first plane corresponds to the first view.

[0296] The acquisition module 1310 is used to acquire noise data corresponding to at least two second planes respectively, wherein the second plane refers to the plane in the polyhedron other than the first plane;

[0297] The generation module 1320 is used to perform a denoising process on the noisy data based on the first view image and the first depth image, under the constraint of multi-plane synchronization conditions, to obtain the second view image and the second depth image corresponding to the at least two second planes respectively. The multi-plane synchronization conditions are used to synchronously analyze multiple planes corresponding to the polyhedron during the denoising process. The at least two second view images and the first view image are used to form a polyhedron texture covering the polyhedron. The polyhedron texture is used to present the scene elements under the multiple viewpoints corresponding to the polyhedron.

[0298] The generation module 1320 is used to generate a three-dimensional scene corresponding to the scene elements based on the polyhedral texture, the first depth image and at least two second depth images.

[0299] In some embodiments, the generation module 1320 is used to input the noise data corresponding to the first view image, the first depth image, and the at least two second planes into the image diffusion model, and output the second view image and the second depth image corresponding to the at least two second planes, wherein the image diffusion model includes a multi-plane synchronization operator determined based on the multi-plane synchronization condition.

[0300] In some embodiments, the multi-plane synchronization operator includes at least one of the following operators:

[0301] Convolutional layer operator, wherein the convolutional layer operator refers to the operator of the convolutional layer in the image diffusion model;

[0302] Attention layer operator, which refers to the operator of the attention layer in the image diffusion model;

[0303] The normalization layer operator refers to the operator of the normalization layer in the image diffusion model.

[0304] In some embodiments, the image diffusion model includes an image encoder, a denoiser, and an image decoder; the generation module 1320 is configured to input the first viewpoint image into the image encoder and output an image feature representation; input the first depth image into the image encoder and output a depth feature representation; generate an input feature representation based on the image feature representation, the depth feature representation, and noise data corresponding to the at least two second planes; input the input feature representation into the denoiser to obtain a denoising result; and input the denoising result into the image decoder to output a second viewpoint image and a second depth image corresponding to the at least two second planes.

[0305] In some embodiments, the noise data includes a first noise image and a second noise image; the generation module 1320 is configured to generate image latent variables based on the image feature representation and the first noise images corresponding to the at least two second planes respectively; generate depth latent variables based on the depth feature representation and the second noise images corresponding to the at least two second planes respectively; and generate the input feature representation based on the image latent variables and the depth latent variables.

[0306] In some embodiments, the acquisition module 1310 is used to acquire a position feature representation, the position feature representation representing the position information of the pixel corresponding to the polyhedron; the generation module 1320 is used to generate the input feature representation based on the image latent variable, the depth latent variable and the position feature representation.

[0307] In some embodiments, the acquisition module 1310 is used to acquire the three-dimensional coordinates of the pixels corresponding to the multiple planes of the polyhedron; project the three-dimensional coordinates onto a unit sphere to obtain normalized coordinates; and determine the position feature representation based on the normalized coordinates of the pixels corresponding to the multiple planes of the polyhedron.

[0308] In some embodiments, the acquisition module 1310 is used to acquire a mask feature representation, the mask feature representation representing the position of the first plane in the polyhedron; the generation module 1320 is used to generate the input feature representation based on the image latent variable, the depth latent variable and the mask feature representation.

[0309] In some embodiments, the acquisition module 1310 is used to acquire scene prompt information, which is used to prompt the scene content of the three-dimensional scene; generate a scene feature representation corresponding to the scene prompt information; and the generation module 1320 is used to input the input feature representation and the scene feature representation into the denoiser to obtain the denoising result.

[0310] In some embodiments, the acquisition module 1310 is configured to perform normalization processing on the first depth image to obtain a normalized image; input the normalized image into the image encoder to output the depth feature representation.

[0311] In some embodiments, the polyhedral texture includes color information corresponding to multiple pixels; the generation module 1320 is configured to perform inverse projection transformation on the multiple pixels in the polyhedral texture based on the first depth image and the at least two second depth images to obtain point cloud coordinates corresponding to the multiple pixels in the polyhedral texture, the point cloud coordinates being used to express the scene position of the pixel in the three-dimensional scene; based on the point cloud coordinates corresponding to the multiple pixels and the polyhedral texture, construct a three-dimensional Gaussian representation corresponding to the multiple pixels, the three-dimensional Gaussian representation including the point cloud coordinates, covariance matrix and color information corresponding to the pixel, the covariance matrix being used to express the distribution characteristics of the pixel; wherein, the three-dimensional Gaussian representation corresponding to the multiple pixels is used to express the three-dimensional scene.

[0312] In some embodiments, the generation module 1320 is configured to construct an initial three-dimensional Gaussian representation corresponding to each of the plurality of pixels based on the point cloud coordinates corresponding to the plurality of pixels and the polyhedron texture; project the initial three-dimensional Gaussian representation corresponding to the plurality of pixels to generate a candidate polyhedron texture corresponding to the polyhedron; and adjust the initial three-dimensional Gaussian representation based on the difference between the candidate polyhedron texture and the polyhedron texture to obtain the three-dimensional Gaussian representation corresponding to each of the plurality of pixels.

[0313] In some embodiments, the generation module 1320 is used to determine the visual weights corresponding to the plurality of pixels in the polyhedral texture, wherein the visual weights represent the visual importance of the pixels; and to construct a three-dimensional Gaussian representation corresponding to the plurality of pixels based on the point cloud coordinates corresponding to the plurality of pixels, the visual weights corresponding to the plurality of pixels, and the polyhedral texture.

[0314] In some embodiments, the acquisition module 1310 is used to render the three-dimensional scene based on the viewpoints corresponding to the multiple planes of the polyhedron, to obtain video frames corresponding to the multiple planes of the polyhedron; and to acquire video noise data corresponding to the multiple planes; the generation module 1320 is used to perform a denoising process on the video noise data based on the video frames corresponding to the multiple planes under the constraint of the multi-plane synchronization condition, to obtain scene videos corresponding to the multiple planes, wherein the viewpoints corresponding to the scene videos of different planes are different.

[0315] In some embodiments, the generation module 1320 is used to input video frames corresponding to the plurality of planes and video noise data corresponding to the plurality of planes into a video diffusion model, and output scene videos corresponding to the plurality of planes; the operators of the video diffusion model include a multi-plane synchronization operator determined based on the multi-plane synchronization conditions.

[0316] In some embodiments, the generation module 1320 is configured to: construct, based on the polyhedral texture, the first depth image, and the at least two second depth images, a three-dimensional Gaussian representation corresponding to a plurality of pixels in the polyhedral texture; wherein the three-dimensional Gaussian representation corresponding to the plurality of pixels is used to represent a static three-dimensional scene; construct an initial four-dimensional Gaussian representation corresponding to the plurality of pixels based on the three-dimensional Gaussian representation corresponding to the plurality of pixels and an implicit neural representation; wherein the implicit neural representation is used to characterize the positional change of the pixels over time; adjust the initial four-dimensional Gaussian representation based on the scene video corresponding to the plurality of planes to obtain the four-dimensional Gaussian representation corresponding to the plurality of pixels; wherein the four-dimensional Gaussian representation corresponding to the plurality of pixels is used to represent a dynamic three-dimensional scene.

[0317] In summary, the 3D scene generation apparatus provided in this application, based on a first-view image and a first-depth image corresponding to a certain plane of a polyhedron, performs a denoising process on noisy data under the constraint of multi-plane synchronization conditions to obtain second-view images and second-depth images corresponding to other planes of the polyhedron. The second-view images and the first-view images are used to construct a polyhedron texture covering the polyhedron. A 3D scene corresponding to scene elements can be generated through the polyhedron texture, the first-depth image, and the second-depth image. During the denoising process, the constraints of multi-plane synchronization conditions allow for the simultaneous analysis of multiple planes corresponding to the polyhedron. This enables the global optimization of the generation results of all planes during the denoising process, rather than processing a single plane in isolation, thereby reducing the problem of inconsistent seams. For example, when generating a second-view image corresponding to a certain plane, the generation results of the first-view image and other planes are referenced simultaneously to ensure that the visual information at the seams transitions naturally between adjacent planes.

[0318] It should be noted that the specific limitations of the embodiments of the one or more three-dimensional scene generation devices provided above can be found in the limitations of the three-dimensional scene generation method above, and will not be repeated here. Each module of the above device can be implemented entirely or partially by software, hardware, or a combination thereof. Each module can be embedded in the processor of the computer device in hardware form or independent of the processor, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0319] This application also provides a computer device, which includes: a processor and a memory, wherein the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the three-dimensional scene generation method provided in the above-described method embodiments.

[0320] For example, Figure 14 This is a structural block diagram of a computer device 1400 provided in an exemplary embodiment of this application. Typically, the computer device 1400 includes a processor 1401 and a memory 1402.

[0321] Processor 1401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1401 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0322] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 are used to store at least one instruction, which is executed by the processor 1401 to implement the three-dimensional scene generation method provided in the various method embodiments of this application.

[0323] In some embodiments, the computer device 1400 may optionally include an input interface 1403 and an output interface 1404. The processor 1401, memory 1402, and input interfaces 1403 and 1404 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interfaces 1403 and 1404 via a bus, signal lines, or a circuit board. The input interfaces 1403 and 1404 can be used to connect at least one input / output (I / O) related peripheral device to the processor 1401 and memory 1402. In some embodiments, the processor 1401, memory 1402, and input interfaces 1403 and 1404 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1401, memory 1402, and input interfaces 1403 and 1404 can be implemented on separate chips or circuit boards, and this application does not limit this.

[0324] Those skilled in the art will understand that Figure 14 The structure shown does not constitute a limitation on the computer device 1400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0325] In an exemplary embodiment, this application provides a chip including programmable logic circuits and / or program instructions, which, when run on a computer device, are used to implement the three-dimensional scene generation method provided in the above-described method embodiments.

[0326] In an exemplary embodiment, this application provides a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the three-dimensional scene generation method provided in the above-described method embodiments.

[0327] In an exemplary embodiment, this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the processor of the computer device to load and execute the method for generating a three-dimensional scene provided in the above-described method embodiment.

[0328] It should be noted that the collection and processing of relevant data (such as first-person perspective images) in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0329] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0330] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0331] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0332] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating a three-dimensional scene, characterized in that, The method includes: Obtain a first-view image corresponding to a scene element and a first-depth image corresponding to the first-view image. The first-view image covers the first plane of the polyhedron, and the first plane corresponds to the first view. Obtain noise data corresponding to at least two second planes, where the second planes refer to planes in the polyhedron other than the first plane; Based on the first viewpoint image and the first depth image, a denoising process is performed on the noisy data under the constraint of multi-plane synchronization conditions to obtain the second viewpoint image and the second depth image corresponding to the at least two second planes respectively. The multi-plane synchronization conditions are used to synchronously analyze multiple planes corresponding to the polyhedron during the denoising process. The at least two second viewpoint images and the first viewpoint image are used to form a polyhedron texture covering the polyhedron. A 3D scene corresponding to the scene elements is generated based on the polyhedral texture, the first depth image, and at least two second depth images.

2. The method according to claim 1, characterized in that, The step of performing a denoising process on the noisy data based on the first viewpoint image and the first depth image, under the constraint of multi-plane synchronization conditions, to obtain the second viewpoint image and the second depth image corresponding to the at least two second planes respectively, includes: The image diffusion model inputs the noise data corresponding to the first viewpoint image, the first depth image, and the at least two second planes respectively, and outputs the second viewpoint image and the second depth image corresponding to the at least two second planes respectively. The image diffusion model includes a multi-plane synchronization operator determined based on the multi-plane synchronization condition.

3. The method according to claim 2, characterized in that, The image diffusion model includes an image encoder, a denoiser, and an image decoder; The step of inputting the noise data corresponding to the first viewpoint image, the first depth image, and the at least two second planes into the image diffusion model, and outputting the second viewpoint image and the second depth image corresponding to the at least two second planes, includes: The first viewpoint image is input into the image encoder, and the image feature representation is output; the first depth image is input into the image encoder, and the depth feature representation is output. An input feature representation is generated based on the image feature representation, the depth feature representation, and the noise data corresponding to the at least two second planes. The input feature representation is input into the denoiser to obtain the denoising result; The denoising result is input into the image decoder, which outputs the second viewpoint image and the second depth image corresponding to the at least two second planes, respectively.

4. The method according to claim 3, characterized in that, The noise data includes a first noise image and a second noise image; The step of generating an input feature representation based on the image feature representation, the depth feature representation, and the noise data corresponding to the at least two second planes includes: Based on the image feature representation and the first noise images corresponding to the at least two second planes, image latent variables are generated; Based on the depth feature representation and the second noise images corresponding to the at least two second planes, a depth latent variable is generated; The input feature representation is generated based on the image latent variables and the depth latent variables.

5. The method according to claim 4, characterized in that, The method further includes: Obtain a position feature representation, wherein the position feature representation characterizes the position information of the pixel points corresponding to the polyhedron; The step of generating the input feature representation based on the image latent variables and the depth latent variables includes: The input feature representation is generated based on the image latent variables, the depth latent variables, and the location feature representation.

6. The method according to claim 5, characterized in that, The acquisition of location feature representation includes: Obtain the three-dimensional coordinates of the pixels corresponding to the multiple planes of the polyhedron; The three-dimensional coordinates are projected onto a unit sphere to obtain normalized coordinates; The positional feature representation is determined based on the normalized coordinates of the pixels corresponding to the multiple planes of the polyhedron.

7. The method according to claim 4, characterized in that, The method further includes: Obtain a mask feature representation, wherein the mask feature representation characterizes the position of the first plane in the polyhedron; The step of generating the input feature representation based on the image latent variables and the depth latent variables includes: The input feature representation is generated based on the image latent variables, the depth latent variables, and the mask feature representation.

8. The method according to claim 3, characterized in that, The method further includes: Obtain scene prompt information, which is used to prompt the scene content of the three-dimensional scene; Generate a scene feature representation corresponding to the scene prompt information; The step of inputting the input feature representation into the denoiser to obtain the denoising result includes: The input feature representation and the scene feature representation are input into the denoiser to obtain the denoising result.

9. The method according to claim 3, characterized in that, The step of inputting the first depth image into the image encoder and outputting a depth feature representation includes: Normalization is performed on the first depth image to obtain a normalized image; The normalized image is input into the image encoder, which outputs the depth feature representation.

10. The method according to any one of claims 1 to 9, characterized in that, The polyhedral texture includes color information corresponding to multiple pixels; The step of generating a 3D scene corresponding to the scene elements based on the polyhedral texture, the first depth image, and at least two second depth images includes: Based on the first depth image and the at least two second depth images, an inverse projection transformation is performed on the plurality of pixels in the polyhedral texture to obtain the point cloud coordinates corresponding to the plurality of pixels in the polyhedral texture, and the point cloud coordinates are used to express the scene position of the pixel in the three-dimensional scene. Based on the point cloud coordinates corresponding to the plurality of pixels and the polyhedral texture, a three-dimensional Gaussian representation corresponding to the plurality of pixels is constructed. The three-dimensional Gaussian representation includes the point cloud coordinates, covariance matrix and color information corresponding to the pixel. The covariance matrix is ​​used to express the distribution characteristics of the pixel. The three-dimensional Gaussian representations corresponding to the plurality of pixels are used to represent the three-dimensional scene.

11. The method according to claim 10, characterized in that, The step of constructing a 3D Gaussian representation of each of the multiple pixels based on the point cloud coordinates and the polyhedral texture includes: Based on the point cloud coordinates corresponding to the plurality of pixels and the polyhedral texture, an initial three-dimensional Gaussian representation corresponding to the plurality of pixels is constructed. Based on the initial 3D Gaussian representation projection corresponding to the multiple pixels, a candidate polyhedron texture is generated for the polyhedron. Based on the difference between the candidate polyhedral texture and the polyhedral texture, the initial 3D Gaussian representation is adjusted to obtain the 3D Gaussian representations corresponding to the multiple pixels respectively.

12. The method according to claim 10, characterized in that, The method further includes: Determine the visual weights corresponding to the plurality of pixels in the polyhedral texture, wherein the visual weights represent the visual importance of the pixels; The step of constructing a 3D Gaussian representation of each of the multiple pixels based on the point cloud coordinates and the polyhedral texture includes: Based on the point cloud coordinates corresponding to the plurality of pixels, the visual weights corresponding to the plurality of pixels, and the polyhedral texture, a three-dimensional Gaussian representation corresponding to the plurality of pixels is constructed.

13. The method according to any one of claims 1 to 9, characterized in that, The method further includes: The three-dimensional scene is rendered based on the viewpoints corresponding to the multiple planes of the polyhedron, and video frames corresponding to the multiple planes of the polyhedron are obtained. Obtain the video noise data corresponding to the multiple planes respectively; Based on the video frames corresponding to the multiple planes respectively, a denoising process is performed on the video noise data under the constraint of the multi-plane synchronization condition to obtain the scene videos corresponding to the multiple planes respectively, wherein the scene videos corresponding to different planes have different perspectives.

14. The method according to claim 13, characterized in that, The step of generating a 3D scene corresponding to the scene elements based on the polyhedral texture, the first depth image, and at least two second depth images includes: Based on the polyhedral texture, the first depth image, and the at least two second depth images, a three-dimensional Gaussian representation corresponding to each pixel in the polyhedral texture is constructed; wherein, the three-dimensional Gaussian representation corresponding to each pixel is used to represent a static three-dimensional scene; Based on the three-dimensional Gaussian representation and implicit neural representation corresponding to the plurality of pixels respectively, an initial four-dimensional Gaussian representation corresponding to the plurality of pixels is constructed; wherein, the implicit neural representation is used to characterize the position change of the pixel over time; The initial four-dimensional Gaussian representation is adjusted based on the scene video corresponding to the multiple planes to obtain the four-dimensional Gaussian representations corresponding to the multiple pixels respectively; wherein, the four-dimensional Gaussian representations corresponding to the multiple pixels are used to express the dynamic three-dimensional scene.

15. A device for generating a three-dimensional scene, characterized in that, The device includes: The acquisition module is used to acquire a first-view image corresponding to a scene element and a first-depth image corresponding to the first-view image. The first-view image covers the first plane of the polyhedron, and the first plane corresponds to the first view. The acquisition module is used to acquire noise data corresponding to at least two second planes, where the second planes refer to planes in the polyhedron other than the first plane. The generation module is used to perform a denoising process on the noisy data based on the first view image and the first depth image under the constraint of multi-plane synchronization conditions, to obtain the second view image and the second depth image corresponding to the at least two second planes respectively. The multi-plane synchronization conditions are used to synchronously analyze multiple planes corresponding to the polyhedron during the denoising process. The at least two second view images and the first view image are used to form a polyhedron texture covering the polyhedron. The generation module is used to generate a 3D scene corresponding to the scene elements based on the polyhedral texture, the first depth image, and at least two second depth images.

16. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the method for generating a three-dimensional scene as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the method for generating a three-dimensional scene as described in any one of claims 1 to 14.

18. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method for generating a three-dimensional scene as described in any one of claims 1 to 14.

Images