Method, device and storage medium for constructing three-dimensional scene

Abstract

Embodiments of the present application provide a three-dimensional scene construction method, device and equipment and a storage medium, comprising: generating an initial representation of a three-dimensional scene based on an equidistant view, segmenting objects in the three-dimensional scene to obtain N objects, and rendering each object using N1 preset perspectives to obtain N1 candidate images of each object; using a non-modal completion method based on a video diffusion model to complete the candidate images of the objects with occlusions; for each object, obtaining a three-dimensional structure prior representation of the object according to a complete image of the object, and generating a final three-dimensional structure of the object according to the three-dimensional structure prior representation by a native three-dimensional generation model. The method creates an initial three-dimensional scene through an equidistant view generation technology, segments the objects in the initial three-dimensional scene to obtain multiple independent objects, and then performs multi-modal completion and reconstruction based on each independent object, thereby being able to generate a three-dimensional scene that supports object instance-level editing and reasonable layout.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for constructing a three-dimensional scene. Background Technology

[0002] In the fields of Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), or digital content creation, generating high-quality, interactive 3D scenes is a crucial task.

[0003] In recent years, with the rapid development of generative artificial intelligence (AI) technology, researchers have proposed various methods for generating 3D scenes. One method utilizes a pre-trained latent diffusion model and a monocular depth estimator to generate 3D mesh or Gaussian scattering scenes by inscribing scenes from random perspectives. This method, based on image completion, typically treats the entire scene as a single unit, failing to effectively decouple individual object instances within the scene. This coupled generation prevents users from selectively editing or interacting with individual objects in the scene, significantly limiting the practicality and operability of the generated scenes. This limitation is particularly pronounced in applications requiring local scene adjustments or object replacement. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for constructing a three-dimensional scene, which can generate a three-dimensional scene that supports object instance-level editing and has a reasonable layout.

[0005] In a first aspect, embodiments of this application provide a method for constructing a three-dimensional scene, the method comprising:

[0006] A two-dimensional initial reference image is obtained, and an initial representation of a three-dimensional scene is generated based on the initial reference image, wherein the initial reference image is an isometric view;

[0007] Based on the initial representation of the three-dimensional scene, the objects in the three-dimensional scene are segmented to obtain N objects in the three-dimensional scene;

[0008] The N objects are rendered using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2;

[0009] Based on the candidate images and segmentation masks of the N objects, determine the occluded objects from the N objects;

[0010] A nonmodal completion method based on a video diffusion model is used to complete the candidate image of the occluded object;

[0011] For each of the N objects, a priori representation of the object's 3D structure is obtained based on the object's complete image and the camera parameters of the image. When the object is not occluded, the complete image of the object is the candidate image of the object; when the object is occluded, the complete image of the object is the complete image obtained by completing the object.

[0012] The prior representation of the object's three-dimensional structure, the noise of the prior representation of the object's three-dimensional structure, and the complete image of the object are input into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

[0013] In some exemplary embodiments, obtaining the two-dimensional initial reference image includes: generating the initial reference image based on a three-dimensional scene description text input by a user; or receiving the initial reference image input by a user.

[0014] In some exemplary embodiments, generating the initial reference image based on user-inputted scene description text includes:

[0015] Multiple scene candidate images are generated based on the scene description text, and the scene candidate images are isometric views;

[0016] Based on user input, an image is selected from the plurality of scene candidate images as the initial reference image.

[0017] In some exemplary embodiments, generating a plurality of scene candidate images based on the scene description text, wherein the scene candidate images are isometric views, includes:

[0018] Add a fixed prefix to the scene description text; the fixed prefix is ​​used to identify the generation of the isometric view.

[0019] The two-dimensional generative model generates multiple scene candidate images based on the scene description text after adding the fixed prefix.

[0020] In some exemplary embodiments, the step of segmenting the objects in the three-dimensional scene according to the initial representation of the three-dimensional scene to obtain N objects in the three-dimensional scene includes:

[0021] The initial representation of the three-dimensional scene is rendered using N2 preset viewpoints to obtain N2 scene images, where N2 is an integer greater than or equal to 2;

[0022] The N2 scene images are segmented using an entity segmentation network to obtain N2 two-dimensional segmentation masks, wherein each scene image corresponds to a two-dimensional segmentation mask, and the two-dimensional segmentation mask is an instance-level mask that is independent of the category.

[0023] The N2 two-dimensional segmentation masks are projected into a three-dimensional space using a semantic segmentation method to separate N objects in the three-dimensional scene.

[0024] In some exemplary embodiments, determining the occluded object from the N objects based on the candidate images and segmentation masks of the N objects includes:

[0025] For each object, N1 candidate images of the object and the segmentation mask corresponding to the N1 candidate images are input into the visual language model to obtain the occlusion detection result of the object. The occlusion detection result is used to indicate whether the object is occluded.

[0026] In some exemplary embodiments, the method of using a nonmodal completion method based on a video diffusion model to complete the candidate image of the occluded object includes:

[0027] For each candidate image of the occluded object, the candidate image is encoded by a variational autoencoder to obtain the latent features of the occluded object;

[0028] The first input feature is obtained by concatenating the latent features of the occluded object and the visible area mask;

[0029] Input the first input feature into the video diffusion model;

[0030] The occluded image of the occluded object is encoded using the CLIP model to obtain the image features of the occluded object, and the image features are sent to the cross-attention module of the video diffusion model.

[0031] The video diffusion model generates video data of the occluded object based on the first input feature and the image feature.

[0032] The last frame of the video data is a complete image obtained by filling in the occluded object.

[0033] In some exemplary embodiments, the concatenation of the latent features of the occluded object and the visible region mask to obtain the first input feature includes:

[0034] The visible region mask of the candidate image of the occluded object is downsampled to obtain the downsampled visible region mask.

[0035] The first input feature is obtained by concatenating the latent features of the occluded object and the downsampled visible area mask.

[0036] In some exemplary embodiments, obtaining a priori representation of the three-dimensional structure of each of the N objects, based on a complete image of the object and camera parameters of the image, includes:

[0037] The initial geometric features of the object are obtained by processing the complete image of the object and the camera parameters of the image using a multi-view large-scale reconstruction model.

[0038] The initial geometric features of the object are voxelized to obtain the display voxel representation of the object;

[0039] The structural encoder encodes the explicit voxel representation of the object to obtain the implicit voxel representation of the object, which is a priori representation of the three-dimensional structure of the object.

[0040] In some exemplary embodiments, the native 3D generation model includes: a structure generator, a structure decoder, a structure feature generator, and a structure feature decoder;

[0041] The step of inputting the prior representation of the object's 3D structure, the noise of the prior representation of the object's 3D structure, and the complete image of the object into the native 3D generation model to obtain the final 3D structure of the object includes:

[0042] The structure generator processes the prior representation of the object's three-dimensional structure, the noise in the prior representation of the object's three-dimensional structure, and the complete image of the object to obtain the latent features of the object's sparse structure.

[0043] The structure decoder decodes the latent features of the sparse structure of the object to obtain the low-dimensional features of the sparse structure of the object.

[0044] The structural feature generator processes the low-dimensional features of the sparse structure of the object to obtain the latent structural features of the object.

[0045] The structural feature decoder decodes the latent structural features of the object to obtain the final three-dimensional structure of the object.

[0046] In some exemplary embodiments, the noise in the prior representation of the three-dimensional structure of the object is the noise corresponding to the corrected flow model at an intermediate time.

[0047] In some exemplary embodiments, the method further includes:

[0048] A first training set is constructed, which includes multiple first training samples. The first training samples include a first sample image and a visible region mask of the first sample image. The first sample image is an image with occlusion. The first sample image and the visible region mask of the first sample image are constructed by the following steps: superimposing a randomly selected second image on a first original image to obtain the first sample image with occlusion; superimposing the visible region mask of the first original image on the second image to obtain the visible region mask of the first sample image; the first original image is an image without occlusion.

[0049] A second training set is constructed, which includes multiple second training samples. The second training samples include second sample images and visible region masks of the second sample images. The second sample images are images with shadows caused by occlusion. The second sample images are constructed through the following steps: standardizing the second original image; simulating the natural falling process of the occluder under gravity using rigid body simulation technology on the standardized second original image; adding the shadow effect caused by the occluder to the second original image to obtain an intermediate image; and performing data augmentation on the intermediate image using various lighting settings to obtain the second sample image. The second original image is an image without occlusion.

[0050] The video diffusion model is trained using the first training set and the second training set, and the output of the video diffusion model is the video data corresponding to the training samples.

[0051] In some exemplary embodiments, training the video diffusion model based on the first training set and the second training set, wherein the output of the video diffusion model is video data corresponding to the training samples, includes:

[0052] The training sample is input into the video diffusion model for training to obtain the video data corresponding to the training sample, wherein the training sample is either the first training sample or the second training sample.

[0053] Based on the training samples and the corresponding video data, the loss of the video diffusion model is calculated using the following formula:

[0054] L dm =Ε x,ε～N(0,1),t [||ε-ε θ (x t ,t,I in M vis )|| 2 ]

[0055] Among them, I in M represents the sample image.vis The visible region mask of the sample image is represented by t, where t represents time, ε represents noise, and ε ~ N(0,1) indicates that the noise follows a normal distribution. t ε represents the result of adding noise to the latent features of the sample image. θ (x t ,t,I in M vis ) represents the predicted noise obtained by the video diffusion model, and E represents the desired operation;

[0056] The parameters of the video diffusion model are updated based on the loss of the video diffusion model.

[0057] Secondly, embodiments of this application provide a three-dimensional scene construction apparatus, the apparatus comprising:

[0058] The scene initialization module is used to acquire a two-dimensional initial reference image and generate an initial representation of the three-dimensional scene based on the initial reference image, wherein the initial reference image is an isometric view.

[0059] The segmentation module is used to segment the objects in the three-dimensional scene according to the initial representation of the three-dimensional scene, so as to obtain N objects in the three-dimensional scene;

[0060] The rendering module is used to render the N objects using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2.

[0061] An occlusion determination module is used to determine the occluded objects from the N objects based on the candidate images and segmentation masks of the N objects.

[0062] The completion module is used to complete the candidate image of the occluded object using a non-modal completion method based on a video diffusion model;

[0063] The generation module is used to obtain a priori representation of the three-dimensional structure of each of the N objects based on the complete image of the object and the camera parameters of the image. When the object is not occluded, the complete image of the object is the candidate image of the object. When the object is occluded, the complete image of the object is the complete image obtained by completing the object.

[0064] The generation module is further configured to input the prior representation of the object's three-dimensional structure, the noise of the prior representation of the object's three-dimensional structure, and the complete image of the object into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

[0065] Thirdly, embodiments of this application provide an electronic device, the electronic device comprising: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in the first aspect above.

[0066] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the method described in the first aspect above.

[0067] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0068] The method for constructing a 3D scene provided in this application generates an initial representation of the 3D scene based on an isometric view. Based on this initial representation, objects in the 3D scene are segmented to obtain N objects. The N objects are rendered using N1 preset viewpoints to obtain N1 candidate images for each object. Based on the candidate images and segmentation masks, occluded objects are identified from the N objects. A non-modal completion method based on a video diffusion model is used to complete the candidate images of the occluded objects. For each of the N objects, a priori representation of the object's 3D structure is obtained based on the object's complete image and camera parameters. The priori representation of the object's 3D structure, the noise in the priori representation, and the complete image of the object are input into a native 3D generation model to obtain the final 3D structure of the object. This method creates an initial 3D scene using isometric view generation technology and segments the objects in the initial 3D scene to obtain multiple independent objects. Subsequent multimodal completion and reconstruction can be performed based on each independent object, thereby generating a 3D scene that supports object instance-level editing and has a reasonable layout. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart of a method for constructing a three-dimensional scene provided in Embodiment 1 of this application;

[0071] Figure 2 This is a schematic diagram of the processing flow in the layered scene parsing stage;

[0072] Figure 3 A schematic diagram generated for a hierarchical scene;

[0073] Figure 4 This is a schematic diagram of a 3D scene and its attributes generated using the method described in the embodiments of the application.

[0074] Figure 5 A flowchart illustrating the training method for the video diffusion model provided in Embodiment 2 of this application;

[0075] Figure 6 This is a schematic diagram of the first training sample, the second training sample, and the video data output by the model.

[0076] Figure 7 A flowchart of the nonmodal completion method based on a video diffusion model provided in Embodiment 3 of this application;

[0077] Figure 8 This is a network architecture diagram for a nonmodal completion method based on a video diffusion model.

[0078] Figure 9 A flowchart of the spatial alignment method provided in Embodiment 4 of this application;

[0079] Figure 10 This is a schematic diagram of the spatial alignment process;

[0080] Figure 11 This is a schematic diagram of the objects and scenes regenerated based on the method of this embodiment;

[0081] Figure 12 A schematic diagram of the structure of the three-dimensional scene construction device provided in Embodiment 5 of this application;

[0082] Figure 13 This is a schematic diagram of a terminal device provided in Embodiment Six of this application. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0085] This application provides a method for constructing a three-dimensional scene, which is of paramount importance in the fields of VR, AR, MR, or digital content creation, where generating high-quality, interactive three-dimensional scenes is crucial.

[0086] To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts involved in all embodiments of this application will be appropriately explained as follows:

[0087] 1) Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It defines and generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of interactive three-dimensional dynamic visual scenes and simulation of physical behavior in the virtual environment, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0088] 2) Virtual reality devices (VR devices) are terminals that realize virtual reality effects. They can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged according to actual needs.

[0089] Optionally, the virtual reality devices described in the embodiments of this application may include, but are not limited to, the following types:

[0090] 2.1) PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions. External PC-based virtual reality devices use the data output from the PC to achieve virtual reality effects.

[0091] 2.2) Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0092] 2.3) All-in-one virtual reality devices have processors for performing virtual functions, thus having independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal, and have a high degree of freedom of use.

[0093] 3) Virtual field of view: The area in the virtual environment that a user can perceive through the lens in a virtual reality device. The perceived area is represented by the field of view (FOV).

[0094] 4) Augmented Reality (AR): A technology that calculates the camera's pose parameters in the real world (or 3D world, real world) in real time during image acquisition, and adds virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0095] 5) Mixed Reality (MR): A simulated scene that integrates computer-created sensory input (e.g., virtual objects) with sensory input or its representation from a physical setting. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0096] 6) Extended Reality (XR) refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices, including virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0097] 7) A virtual scene is a virtual scene displayed (or provided) by an application while running on an electronic device. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual scene, or a purely fictional virtual scene. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application does not limit the dimension of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities. Users can control virtual objects to move within this virtual scene.

[0098] Native 3D generative models refer to a type of model that generates a 3D model directly without going through intermediate multi-view 2D images or multi-view iterative optimization.

[0099] The diffusion model is based on two processes: forward diffusion and backward denoising. The forward diffusion process is a Markov chain that progressively adds Gaussian noise to the data at each time step, eventually transforming the original data distribution into a known prior (usually a standard Gaussian distribution). The backward process, on the other hand, denoises and reconstructs the data from pure noise, simulating the data distribution through iterative denoising. Its purpose is to learn how to recover the original 3D scene data from the noise.

[0100] Figure 1 This is a flowchart illustrating a method for constructing a three-dimensional scene according to Embodiment 1 of this application, applied in a terminal device, including but not limited to AR devices, VR devices, MR devices, mobile phones, tablets, laptops, desktop computers, etc. Figure 1 As shown, the method provided in this embodiment includes the following steps.

[0101] S101. Obtain a two-dimensional initial reference image and generate an initial representation of the three-dimensional scene based on the initial reference image. The initial reference image is an isometric view.

[0102] The terminal device runs 3D scene generation software. This 3D scene generation software can be an application (APP) program running on the terminal device, or it can be implemented as a mini-program or application module in the host application, or a web-based generation client, etc.

[0103] The user interface (UI) of 3D scene generation software includes an information input box, through which the user inputs an initial reference image or information used to generate the initial reference image.

[0104] In one implementation, the user inputs a 3D scene description text, and an initial reference image is generated based on the user-input 3D scene description text. In another implementation, the user inputs an initial reference image.

[0105] The 3D scene description text is used to describe some characteristics of the 3D scene to be generated, such as the size, position, and color of the objects to be included in the 3D scene. For example, the 3D scene description text could be: Minimalist Coffee Corner.

[0106] In this embodiment, the initial reference image is an isometric view, also called an orthographic view, which is a method of representing a three-dimensional object in two dimensions. During drawing, each side of the object is scaled according to the drawing ratio, all parallel lines on the object remain parallel, and the displayed size of the object is independent of the viewpoint distance. In an isometric view, the included angle between the three axes is typically 120 degrees, and a three-dimensional image effect is constructed on a two-dimensional plane using line segments of equal length.

[0107] Compared to perspective view, isometric view has no perspective distortion, can show multiple faces of an object, which facilitates understanding the object's structure and the spatial relationship between its parts during subsequent processing. It also facilitates the segmentation of the object and the determination of occlusion relationships between objects, effectively reducing occlusion between objects in the scene and making the overall scene presentation more comprehensive and natural.

[0108] Optionally, in some scenarios, the initial reference image can also be a perspective image, and the image captured by the camera is a perspective image.

[0109] Optionally, generating an initial reference image based on the scene description text input by the user includes: generating multiple scene candidate images based on the scene description text, wherein the scene candidate images are isometric views; and selecting one image from the multiple scene candidate images as the initial reference image based on the user's operation.

[0110] Multiple generated scene candidates are displayed to the user for selection. The user can choose the image they need or are satisfied with as the initial reference image from the multiple scene candidate images, thereby improving the user's satisfaction with the final generated 3D scene.

[0111] Optionally, multiple scene candidate images can be generated by adding a fixed prefix to the scene description text, which is used to identify the generated isometric view. The two-dimensional generation model generates multiple scene candidate images based on the scene description text with the fixed prefix added.

[0112] For example, the added fixed prefix is ​​"isometric view of".

[0113] The initial reference image can be converted into an initial representation of the 3D scene using a pre-trained native 3D generation model. For example, the native 3D generation model can be a Trellis model, which uses 3D Gaussian scattering as the scene representation method to ensure rendering quality and processing efficiency.

[0114] S102. Based on the initial representation of the 3D scene, the objects in the 3D scene are segmented to obtain N objects in the 3D scene.

[0115] The method for constructing a 3D scene in this application mainly includes three stages: layered scene parsing, non-modal completion, and spatial alignment. Figure 2 This is a schematic diagram of the processing flow in the layered scene parsing stage, as shown below. Figure 2 As shown in the figure, scene initialization refers to converting the initial reference image into an initial representation of the 3D scene. The layered scene parsing stage corresponds to the processing steps S101-104 in this embodiment.

[0116] The layered scene parsing stage begins by initializing the entire scene as a whole. Subsequently, objects in the 3D scene are segmented to obtain multiple independent objects. Each object is then processed separately, and each object is treated as a whole. This forms a layered "scene-object" structure, and the scene is then generated based on the objects obtained from the layering. Figure 3 A schematic diagram generated for a hierarchical scene, such as Figure 3 As shown, multiple incomplete objects are first obtained by segmenting based on the isometric view, and complete objects are regenerated based on the incomplete objects. These multiple reconstructed complete objects form a spatially aligned 3D scene.

[0117] For example, objects in a 3D scene are segmented as follows: an initial representation of the 3D scene is rendered using N2 preset viewpoints to obtain N2 scene images, where N2 is an integer greater than or equal to 2; an entity segmentation network is used to segment the N2 scene images to obtain N2 two-dimensional segmentation masks, where each scene image corresponds to one two-dimensional segmentation mask, and the two-dimensional segmentation mask is an instance-level mask that is independent of category; a semantic segmentation method is used to project the N two-dimensional segmentation masks onto 3D space to separate N objects in the 3D scene.

[0118] The terminal device renders the 3D scene using N² preset viewpoints, obtaining a multi-view image of the 3D scene. This multi-view image includes N² scene images, and the N² preset viewpoints can be represented as follows: The multi-view image of the rendered 3D scene can then be represented as:

[0119] For each rendered scene image, an entity segmentation network is used to generate a class-independent instance-level 2D segmentation mask. The resulting 2D mask image can be represented as follows:

[0120] Category-independent segmentation means that the segmentation process does not distinguish the specific category of the object; it only focuses on segmenting each independent instance (single object) without considering whether the object belongs to a category such as cat or dog. Instance-level segmentation emphasizes that the granularity of segmentation is at each individual object instance in the image, rather than dividing the entire image into regions or only identifying the object's category. For example, when a scene image includes multiple tables, instance-level segmentation will segment each table as an object, rather than segmenting multiple tables as a single object.

[0121] A 2D segmentation mask is a 2D matrix of the same size as the original scene image, where each element corresponds to a pixel in the original scene image. In a segmentation mask, different values ​​(such as 0 and 1, or different grayscale values) are typically used to indicate whether a pixel belongs to a certain object.

[0122] Optionally, the semantic segmentation method can be OmniSeg3D-GS, a 3D Gaussian scattering semantic segmentation method based on contrastive learning. OmniSeg3D-GS projects these 2D segmentation masks into 3D space to represent the semantic segments of objects in the scene. Effective separation.

[0123] S103. Render N objects using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2.

[0124] Because objects in a scene often occlude each other, directly segmented objects are usually incomplete. To address this issue, this embodiment introduces an object-centric multi-view rendering technique. Each segmented object Oi in the scene is a 3D object. For each object, it can be rendered from N1 preset viewpoints in its local coordinate system, resulting in N1 candidate images. The N1 preset viewpoints can be represented as follows: The N1 candidate images can be represented as: Candidate images can be used as input for subsequent reconstruction processes. Candidate images obtained from multi-view rendering are two-dimensional images.

[0125] S104. Based on the candidate images and segmentation masks of N objects, determine the occluded objects from the N objects.

[0126] Among N objects, some objects may be occluded. In this embodiment, it is necessary to identify the occluded objects. In one optional implementation, for each object, the N1 candidate images of the object and the segmentation mask corresponding to the N1 candidate images are input into the visual language model to obtain the occlusion detection result of the object. The occlusion detection result is used to indicate whether the object is occluded.

[0127] The segmentation mask for each object can be obtained from the 2D segmentation mask of the corresponding scene image. Each 2D segmentation mask corresponding to the scene image includes the segmentation masks for multiple objects.

[0128] Optionally, when an object is detected to be occluded and the occlusion ratio exceeds a preset occlusion threshold, the object is determined to be occluded; if the occlusion ratio is less than the occlusion threshold, the object is determined to be occluded.

[0129] Optionally, after determining that an object is occluded, candidate images of the object are marked, and the marked candidate images of the object need to be processed by nonmodal completion.

[0130] S105. A nonmodal completion method based on a video diffusion model is used to complete the candidate image of an occluded object.

[0131] To recover the occluded object, the occluded image must first be completed. When there is occlusion in the scene, existing non-modal completion methods are difficult to accurately restore the details of the occluded part and cannot effectively remove visual traces such as shadows and reflections, resulting in visually inconsistent generated results.

[0132] The visual diffusion model takes as input a candidate image of the occluded object and a visible region mask of the candidate image. The output is a video dataset containing multiple frames, where the last frame is the complete image obtained by padding the occluded object. Each object has N1 candidate images. For each of the N1 candidate images of an occluded object, padding is performed sequentially to obtain the complete image after padding each candidate image.

[0133] The nonmodal completion method based on a video diffusion model provided in this embodiment treats the completion process as a gradual time-varying process, enabling the model to better understand the overall structure of the object and improving the completion effect. To train this visual diffusion model, this embodiment also specifically constructs a training dataset for training the visual diffusion model, and the training process is described in detail in the following embodiments.

[0134] S106. For each of the N objects, obtain the prior representation of the object's three-dimensional structure based on the complete image of the object and the camera parameters of the image.

[0135] After completing the candidate image of each object, the object needs to be reconstructed. To ensure that the reconstructed object is consistent with the original scene in terms of spatial layout, this application proposes a spatial alignment mechanism with shape prior injection. This mechanism cleverly combines the high-fidelity texture generation capability of the native 3D generative model with the advantage of maintaining spatial relationships in the multi-view large-scale reconstruction model.

[0136] This step is to obtain the three-dimensional structural prior representation of the object. The three-dimensional structural prior representation of the object is also called the shape prior representation of the object. The three-dimensional structural prior representation can also be described as three-dimensional structural prior features or three-dimensional structural prior vectors. This three-dimensional structural prior representation is used to represent the three-dimensional structure or shape of the object.

[0137] Among N objects, some objects may not be occluded, while others are occluded. For objects that are not occluded, the complete image of the object is a candidate image of the object. In step S104, N1 candidate images of each object are obtained through multi-view rendering. All or part of the N1 candidate images of the object can be used as the complete image of the object.

[0138] When an object is occluded, step S105 completes all N1 candidate images of the occluded object. The complete image obtained by completing all or part of the N1 candidate images of the object is then taken as the complete image of the object.

[0139] S107. Input the prior representation of the object's three-dimensional structure, the noise of the prior representation of the object's three-dimensional structure, and the complete image of the object into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

[0140] This native 3D generation model guides the generation of the object's 3D structure by incorporating prior knowledge. Guided by the prior representation of the object's 3D structure, the final generated 3D structure not only retains the object's initial geometric scale and pose characteristics, but also significantly improves geometric details and texture quality, ensuring spatial consistency between the reconstructed object and the original scene.

[0141] Optionally, the noise in the prior representation of the object's 3D structure is the noise corresponding to the intermediate time t of the corrected flow model, typically chosen to be t∈[0.2,0.4]. The noise corresponding to the intermediate time t of the corrected flow model usually has some correlation with the original data. The corrected flow model considers information such as the data's distribution characteristics during noise generation, so the noise generated at the intermediate time t will retain some statistical properties or potential structural information of the data to a certain extent. By introducing different levels of noise at different times, the model can learn the robust characteristics of the data, enhancing its adaptability to various input conditions.

[0142] Optionally, the noise in the prior representation of the object's three-dimensional structure is random noise, which is generally generated independently according to a certain random distribution (such as Gaussian distribution, uniform distribution, etc.). It usually has no direct correlation with the original data and is generated completely randomly. Random noise can also be used for data augmentation.

[0143] The method in this embodiment can generate 3D scenes that support object instance-level editing, have a natural and reasonable layout, effectively handle occlusion relationships, and maintain spatial consistency. Figure 4 This is a schematic diagram of a 3D scene and its attributes generated using the method described in the application embodiments. Figure 4 It can be seen that the final generated 3D scene has the same layout as the objects in the initial reference image, and the objects in the generated 3D scene can be edited and interacted with.

[0144] In this embodiment, an initial representation of the 3D scene is generated based on an isometric view. Based on this initial representation, objects in the 3D scene are segmented to obtain N objects. N1 preset viewpoints are used to render the N objects, resulting in N1 candidate images for each object. Based on the candidate images and segmentation masks, occluded objects are identified from the N objects. A non-modal completion method based on a video diffusion model is used to complete the candidate images of the occluded objects. For each of the N objects, a priori representation of the object's 3D structure is obtained based on the object's complete image and camera parameters. The priori representation of the object's 3D structure, the noise in the priori representation, and the complete image of the object are input into the native 3D generation model to obtain the final 3D structure of the object. This method creates an initial 3D scene using isometric view generation technology and segments the objects in the initial 3D scene to obtain multiple independent objects. Subsequently, multimodal completion and reconstruction can be performed based on each independent object, thereby generating a 3D scene that supports object instance-level editing and has a reasonable layout. The nonmodal completion method based on the video diffusion model can effectively handle the occlusion relationship between objects. During reconstruction, the prior representation of the 3D structure of the object is used as a guide for the native 3D generation model, which can generate a 3D scene that maintains spatial consistency.

[0145] Based on Embodiment 1, Embodiment 2 of this application provides a training method for a video diffusion model, used to train the video diffusion model used in Embodiment 1. Figure 5 This is a flowchart of the training method for the video diffusion model provided in Embodiment 2 of this application, as shown below. Figure 5 As shown, the training method for this video diffusion model includes the following steps.

[0146] S201. Construct a first training set, which includes multiple first training samples. Each first training sample includes a first sample image and a visible region mask of the first sample image, wherein the first sample image is an image with occlusion.

[0147] Optionally, the first sample image and the visible region mask of the first sample image are constructed by the following steps: superimposing a randomly selected second image on the first original image to obtain the first sample image with occlusion, and superimposing the visible region mask of the first original image on the second image to obtain the visible region mask of the first sample image, wherein the first original image is an image without occlusion.

[0148] Figure 6 This is a diagram illustrating the first training sample, the second training sample, and the video data output by the model, as shown below. Figure 6 As shown, (a) is the first original image, (b) is the first sample image, and (c) is the visible region mask of the first sample image. The small car in the first sample image is an occlusion. The visible region mask of the first original image is not shown in the figure. The white area in image (c) is the visible region of the first sample image. The white area plus the area of ​​the occlusion in image (c) constitutes the visible region of the first original image.

[0149] Inputting images (b) and (c) into the video diffusion model yields the video data shown in (d). This video data includes four frames, which are the completed images from left to right. The last frame in this video data is the complete image after completion of the first sample image. As shown in (d), the image completion process is similar to the effect of a time-continuous video transition.

[0150] S202. Construct a second training set, which includes multiple second training samples. The second training samples include second sample images and visible region masks of the second sample images. The second sample images are images with shadows caused by occlusion.

[0151] Occlusion can cast shadows on the surface of an object. To address this shadow problem, embodiments of this application construct a second sample image that includes a shadow effect. Optionally, the second sample image is constructed through the following steps: standardizing a second original image; simulating the natural fall of an occluding object under gravity using rigid body simulation technology on the standardized second original image; adding a shadow effect from the occluding object to the second original image to obtain an intermediate image; and performing data augmentation on the intermediate image using various lighting settings to obtain the second sample image.

[0152] The second original image is an unoccluded image. It is scaled to a fixed size through normalization. Then, rigid body simulation techniques in Blender can be used to simulate the natural fall of occluded objects under gravity, adding shadow effects from the occluded objects to the second original image to ensure accurate shadow rendering. Optionally, data augmentation can be performed using various lighting settings to improve the model's adaptability to complex lighting conditions, enabling it to effectively remove visual artifacts such as shadows and reflections caused by occlusion.

[0153] refer to Figure 6 As shown, (e) is a data example considering shadows. (e) shows two sets of data. In the first set, the left image is the second original image, and the right image is the second sample image. The building is the occluded object, the area indicated by the small white box in the middle of the building is the occluding object, and the dashed box represents the shadow cast by the occluding object. In the second set, the left image is the second original image, and the right image is the second sample image. The drum is the occluded object, the area indicated by the white box on the drum is the occluding object, and the dashed box represents the shadow cast by the occluding object.

[0154] The visible area masks for the two second sample images are not shown in the figure. In the latter set of examples, the visible area is the part of the outer contour of the building excluding the occluder, and the visible area is the part of the outer contour of the drum excluding the occluder.

[0155] S203. Train the video diffusion model based on the first training set and the second training set. The output of the video diffusion model is the video data corresponding to the training samples.

[0156] The first sample image in the first training set is an image that includes occlusions but does not produce shadows, and the second sample image in the second training set is an image that includes occlusions but does not produce shadows. By training the diffusion video model using these two training sets, the resulting model can not only handle occlusions but also eliminate visual artifacts such as shadows and reflections caused by occlusions, thus improving the model's adaptability to complex lighting conditions.

[0157] For example, the video diffusion model is trained by the following steps: inputting training samples into the video diffusion model for training, obtaining video data corresponding to the training samples, which are either the first training sample or the second training sample; calculating the loss of the video diffusion model based on the training samples and the corresponding video data; and updating the parameters of the video diffusion model based on the loss of the video diffusion model.

[0158] Optionally, the loss of the video diffusion model can be calculated using the following formula:

[0159] L dm =Ε x,ε～N(0,1),t [||ε-εθ (x t ,t,I in M vis )|| 2 ]

[0160] Among them, I in M represents the sample image. vis The visible region mask of the sample image is represented by t, where t represents time, ε represents noise, and ε ~ N(0,1) indicates that the noise follows a normal distribution. t ε represents the result of adding noise to the latent features of the sample image. θ (x t ,t,I in M vis ) represents the predicted noise obtained from the video diffusion model, and E represents the desired operation.

[0161] The non-modal completion process during model training is the same as the non-modal completion process during model inference. The non-modal completion process during model training is similar to the non-modal completion process during model inference described below, and will not be repeated here.

[0162] It should be noted that the model training method in Embodiment 2 can be decoupled from the 3D scene construction method in Embodiment 1. That is, the training device used to train the video diffusion model and the terminal device used to perform the 3D scene construction can be different devices. Typically, the training device is used to train the video diffusion model, and the model trained by the training device is used by the terminal device to construct the 3D scene.

[0163] Embodiment 3 of this application provides a nonmodal completion method based on a video diffusion model, used to illustrate a possible implementation of step S105 in Embodiment 1. Figure 7 The flowchart of the nonmodal completion method based on the video diffusion model provided in Embodiment 3 of this application is as follows: Figure 7 As shown, the nonmodal completion method based on the video diffusion model includes the following steps.

[0164] S1051. For each candidate image of an occluded object, the candidate image is encoded by a variational autoencoder to obtain the latent features of the occluded object.

[0165] Figure 8 The network architecture diagram for a nonmodal completion method based on a video diffusion model is shown below. Figure 8As shown, the candidate image is first encoded using a Variational Autoencoder (VAE) to obtain the latent features x0 of occluded objects, which serve as the initial frame of the video latent sequence. Since the objects in this candidate image are incomplete, they need to be padded. For example, Figure 8 The position of the obstruction on the chair becomes a blank area. The obstruction is invisible to the object. Therefore, the obstruction can be set to white. Thus, the object in the candidate image consists of a visible area and an invisible area. The visible area is the image of the object itself, and the invisible area is the white part, which is the part that needs to be filled in.

[0166] VAE is a powerful generative model that combines variational inference and autoencoders. An autoencoder is an unsupervised learning model consisting of an encoder and a decoder. The encoder maps the input data to a low-dimensional latent space (encoding), and the decoder reconstructs the original input from the representation in the latent space (decoding). The goal of an autoencoder is to minimize the difference between the input and the reconstructed output.

[0167] S1052. The potential features of the occluded object and the visible area mask are concatenated to obtain the first input feature, and the first input feature is input into the video diffusion model.

[0168] In one implementation, the latent features of occluded objects and the visible region mask can be directly concatenated using the Concat() function to obtain the first input feature.

[0169] In another implementation, refer to Figure 8 As shown, the visible region mask of the candidate image containing occluded objects is first downsampled to obtain a downsampled visible region mask. Then, the latent features of the occluded objects and the downsampled visible region mask are concatenated to obtain the first input feature. By downsampling the visible region mask of the candidate image, the resolution of the candidate image and the visible region mask is ensured to be consistent.

[0170] S1053. The occluded image containing the occluded object is encoded using the CLIP model to obtain the image features of the occluded object, and these image features are sent to the cross-attention module of the video diffusion model.

[0171] Optionally, the occluded image of the occluded object can be encoded using a Contrastive Language-Image Pretraining (CLIP) model to obtain the image features of the occluded object.

[0172] The occlusion image includes the occlusion object, see reference. Figure 8As shown, the table is an occluded object, and the bread and coffee on the table are also occluded. The CLIP model is a model architecture for multimodal learning that maps images and text to a common feature space, allowing the similarity between images and text to be measured by their distance in the feature space.

[0173] The CLIP model comprises a text encoder and an image encoder. The text encoder processes text data, while the image encoder processes image data. The image encoder converts the input image into a vector representation. It extracts features from the image through operations such as convolutional layers and pooling layers, progressively extracting local and global features to ultimately obtain a vector that describes the image content. In this embodiment, the CLIP model's image encoder encodes occluded images containing occluded objects and injects them into the model via a cross-attention mechanism. The image features injected by the CLIP model help the model understand the object input by the first input feature, enhancing the video diffusion model's ability to understand semantic content and ensuring that the generated complete object is semantically consistent with the original object.

[0174] Cross-attention is a technique in neural networks used to handle the correlation between data from different modalities. Unlike self-attention, which calculates attention weights between elements within the same sequence, cross-attention calculates attention weights between two different sequences. In this way, the model can capture the correlation between data from different modalities and understand the connections between different information sources.

[0175] S1054. The video diffusion model generates video data of occluded objects based on the first input features and image features. The last frame of the video data is a complete image obtained by piecing together the occluded objects.

[0176] Reference Figure 8 As shown, the video diffusion model has two processes: forward diffusion and backward diffusion (also known as backward denoising). During forward diffusion, noise is gradually added to the model's input features. This added noise can be random noise following a normal distribution. During backward diffusion, the noise is gradually removed to recover the image. (Refer to...) Figure 8 The occluded object is a chair. The video data obtained after processing by the video diffusion model includes multiple frames of images. In these multiple frames, the occlusion of the chair is gradually eliminated. The last frame is the image with the occlusion completely eliminated. The image with the occlusion completely eliminated is the completed image.

[0177] In this embodiment, image features of occluded objects are injected into the video diffusion model through a cross-attention mechanism. The video diffusion model includes a cross-attention module with two inputs: one is the input obtained by processing the first input features, and the other is the image features of the occluded object. The cross-attention module calculates the attention weight between the two inputs. By injecting image features from the CLIP model, the video diffusion model's ability to understand semantic content is enhanced, ensuring that the generated complete object is semantically consistent with the original object.

[0178] For an occluded object, it can include multiple candidate images. For each candidate image, it is completed according to the methods in S1051-S1054 to obtain a complete candidate image of the object.

[0179] Embodiment 4 of this application provides a spatial alignment method to illustrate a possible implementation of steps S106 and S107 in Embodiment 1. Figure 9 A flowchart of the spatial alignment method provided in Embodiment 4 of this application is shown below. Figure 9 As shown, the spatial alignment method includes the following steps.

[0180] S1061. For each of the N objects, the complete image of the object and the camera parameters of the image are processed by a multi-view large-scale reconstruction model to obtain the initial geometric features of the object.

[0181] In this embodiment, steps S1061-S1063 are an optional implementation of obtaining a priori representation of the three-dimensional structure of the object. Figure 10 This is a schematic diagram of the spatial alignment process, see reference. Figure 10 As shown, a complete image of an object can be one or more images. A complete image of an object refers to an image without occlusion. In the figure, [R,T] represents the camera parameters of the corresponding image, where R represents the camera rotation matrix and T represents the camera translation matrix.

[0182] S1062. Perform voxelization on the initial geometric features of the object to obtain the display voxel representation of the object.

[0183] The initial three-dimensional geometric shape of the object is obtained by reconstructing a large-scale reconstruction model with multiple views. The initial geometric features are then voxelized and transformed into explicit voxel representations. Explicit voxel representations use discrete voxel units to describe the three-dimensional structure of the object.

[0184] S1063. The structure encoder encodes the explicit voxel representation of the object to obtain the implicit voxel representation of the object. The implicit voxel representation is the prior representation of the three-dimensional structure of the object.

[0185] The explicit voxel representation is processed using a structural encoder and compressed into a low-dimensional implicit voxel representation, compared with... Figure 9 As shown, the resolution of the display voxel is R=64 and the number of channels is C=1, while the resolution of the implicit voxel is R=16 and the number of channels is C=8. It can be understood that a structure encoder can decode the implicit voxel representation into the display voxel representation.

[0186] S1071. The prior representation of the three-dimensional structure of the object, the noise of the prior representation of the three-dimensional structure of the object, and the complete image of the object are processed by the structure generator to obtain the latent features of the sparse structure of the object.

[0187] In this embodiment, steps S1071-S1074 are an implementation method for generating the final three-dimensional structural features of an object using a native 3D generation model. For example, the native 3D generation model includes: a structure generator, a structure decoder, a structural feature generator, and a structural feature decoder.

[0188] Reference Figure 10 As shown in the figure, the cube-like structure represents the noise of the prior representation of the object's three-dimensional structure. The prior representation of the object's three-dimensional structure and the noise of the prior representation of the object's three-dimensional structure can be concatenated and used as an input to the structure generator.

[0189] S1072. The latent features of the sparse structure of the object are decoded by the structure decoder to obtain the low-dimensional features of the sparse structure of the object.

[0190] The latent features of the sparse structure generated by the structure generator are high-dimensional features (also known as high-dimensional implicit features), while the structure decoder decodes the low-dimensional features of the sparse structure (also known as low-dimensional explicit features). Compared with high-dimensional features, the low-dimensional features of the sparse structure can clearly present the three-dimensional structure of the object. The low-dimensional features of the sparse structure can also be understood as features that can be used to display the three-dimensional structure of the object, while the latent features of the sparse structure cannot be directly used to display the three-dimensional structure of the object.

[0191] S1073. The low-dimensional features of the sparse structure of the object are processed by the structural feature generator to obtain the potential structural features of the object.

[0192] Based on the sparse structure of an object, the structural feature generator can generate the object's latent structural features (or latent structural variables). The latent structural features generated by the structural feature generator are high-dimensional features, which can be understood as more refined features such as geometric details, color, and texture in the structure.

[0193] S1074. The structural potential features of the object are decoded by the structural feature decoder to obtain the final three-dimensional structure of the object.

[0194] The structural latent features generated by the structural feature generator are high-dimensional features, while the structural feature decoder decodes the final three-dimensional structure, which is a low-dimensional feature. Compared with high-dimensional features, low-dimensional features can clearly present the three-dimensional structure of an object. The final three-dimensional structural features of an object can be understood as features that can be used to display the three-dimensional structure of an object, while structural latent features cannot be directly used to display the three-dimensional structure of an object.

[0195] Optionally, the final 3D structure of the object is represented by a mesh and Gaussian splats. A mesh is a common form of 3D data representation, consisting of vertices, edges, and faces. Gaussian splats are a point-based 3D representation method where each point is treated as a Gaussian function, also known as a splat. These Gaussian functions have certain properties such as radius and color. By superimposing multiple Gaussian splats, the surface and shape of the object can be approximately represented.

[0196] Figure 11 This is a schematic diagram of the objects and scenes regenerated based on the method of this embodiment, such as... Figure 11 As shown, the regenerated objects are complete, and these regenerated objects form a spatially aligned 3D scene. Optionally, the user can edit the 3D scene generated in this application embodiment. When editing the scene, each independent object in the 3D scene can be edited at the object instance level.

[0197] The spatial alignment method provided in this embodiment obtains the prior representation of the object's three-dimensional structure. Guided by the prior representation of the object's three-dimensional structure, the structure of the object generated by the native three-dimensional generative model not only retains the scale and pose characteristics of the initial geometry, but also significantly improves the geometric details and texture quality, ensuring the spatial consistency between the reconstructed object and the original scene.

[0198] To facilitate better implementation of the three-dimensional scene construction method of this application embodiment, this application embodiment also provides a three-dimensional scene construction device. Figure 12 This is a schematic diagram of the structure of the three-dimensional scene construction device provided in Embodiment 5 of this application, as shown below. Figure 12 As shown, the 3D scene construction device 100 may include:

[0199] Scene initialization module 11 is used to acquire a two-dimensional initial reference image and generate an initial representation of a three-dimensional scene based on the initial reference image, wherein the initial reference image is an isometric view;

[0200] The segmentation module 12 is used to segment the objects in the three-dimensional scene according to the initial representation of the three-dimensional scene, so as to obtain N objects in the three-dimensional scene;

[0201] The rendering module 13 is used to render the N objects using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2.

[0202] The occlusion determination module 14 is used to determine the occluded objects from the N objects based on the candidate images and segmentation masks of the N objects.

[0203] The completion module 15 is used to complete the candidate image of the occluded object using a non-modal completion method based on a video diffusion model;

[0204] The generation module 16 is used to obtain a priori representation of the three-dimensional structure of each of the N objects based on the complete image of the object and the camera parameters of the image. When the object is not occluded, the complete image of the object is the candidate image of the object. When the object is occluded, the complete image of the object is the complete image obtained by completing the object.

[0205] The generation module 16 is further configured to input the prior representation of the three-dimensional structure of the object, the noise of the prior representation of the three-dimensional structure of the object, and the complete image of the object into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

[0206] In some implementations, the scene initialization module 11 is specifically used for:

[0207] The initial reference image is generated based on the 3D scene description text input by the user;

[0208] Alternatively, the initial reference image input by the user can be received.

[0209] In some implementations, generating the initial reference image based on the scene description text input by the user includes:

[0210] Multiple scene candidate images are generated based on the scene description text, and the scene candidate images are isometric views;

[0211] Based on user input, an image is selected from the plurality of scene candidate images as the initial reference image.

[0212] In some implementations, generating multiple scene candidate images based on the scene description text, wherein the scene candidate images are isometric views, includes:

[0213] Add a fixed prefix to the scene description text; the fixed prefix is ​​used to identify the generation of the isometric view.

[0214] The two-dimensional generative model generates multiple scene candidate images based on the scene description text after adding the fixed prefix.

[0215] In some implementations, the segmentation module 12 is specifically used for:

[0216] The initial representation of the three-dimensional scene is rendered using N2 preset viewpoints to obtain N2 scene images, where N2 is an integer greater than or equal to 2;

[0217] The N2 scene images are segmented using an entity segmentation network to obtain N2 two-dimensional segmentation masks, wherein each scene image corresponds to a two-dimensional segmentation mask, and the two-dimensional segmentation mask is an instance-level mask that is independent of the category.

[0218] The N2 two-dimensional segmentation masks are projected into a three-dimensional space using a semantic segmentation method to separate N objects in the three-dimensional scene.

[0219] In some implementations, the occlusion determination module 14 is specifically used to: for each object, input N1 candidate images of the object and the segmentation mask corresponding to the N1 candidate images into a visual language model to obtain the occlusion detection result of the object, and the occlusion detection result is used to indicate whether the object is occluded.

[0220] In some implementations, the completion module 15 is specifically used for:

[0221] For each candidate image of the occluded object, the candidate image is encoded by a variational autoencoder to obtain the latent features of the occluded object;

[0222] The first input feature is obtained by concatenating the latent features of the occluded object and the visible area mask;

[0223] Input the first input feature into the video diffusion model;

[0224] The occluded image of the occluded object is encoded by the contrastive language-image pre-trained CLIP model to obtain the image features of the occluded object, and the image features are sent to the cross-attention module of the video diffusion model.

[0225] The video diffusion model generates video data of the occluded object based on the first input feature and the image feature.

[0226] The last frame of the video data is a complete image obtained by filling in the occluded object.

[0227] In some implementations, the concatenation of the latent features of the occluded object and the visible region mask to obtain the first input feature includes:

[0228] The visible region mask of the candidate image of the occluded object is downsampled to obtain the downsampled visible region mask.

[0229] The first input feature is obtained by concatenating the latent features of the occluded object and the downsampled visible area mask.

[0230] In some implementations, the generation module 16 is specifically used for:

[0231] The initial geometric features of the object are obtained by processing the complete image of the object and the camera parameters of the image using a multi-view large-scale reconstruction model.

[0232] The initial geometric features of the object are voxelized to obtain the display voxel representation of the object;

[0233] The structural encoder encodes the explicit voxel representation of the object to obtain the implicit voxel representation of the object, which is a priori representation of the three-dimensional structure of the object.

[0234] In some implementations, the native 3D generation model includes: a structure generator, a structure decoder, a structure feature generator, and a structure feature decoder;

[0235] The step of inputting the prior representation of the object's 3D structure, the noise of the prior representation of the object's 3D structure, and the complete image of the object into the native 3D generation model to obtain the final 3D structure of the object includes:

[0236] The structure generator processes the prior representation of the object's three-dimensional structure, the noise in the prior representation of the object's three-dimensional structure, and the complete image of the object to obtain the latent features of the object's sparse structure.

[0237] The structure decoder decodes the latent features of the sparse structure of the object to obtain the low-dimensional features of the sparse structure of the object.

[0238] The structural feature generator processes the low-dimensional features of the sparse structure of the object to obtain the latent structural features of the object.

[0239] The structural feature decoder decodes the latent structural features of the object to obtain the final three-dimensional structure of the object.

[0240] In some implementations, the noise in the prior representation of the object's three-dimensional structure is the noise corresponding to the corrected flow model at an intermediate time.

[0241] In some implementations, the apparatus further includes a training module, which is used for:

[0242] A first training set is constructed, which includes multiple first training samples. The first training samples include a first sample image and a visible region mask of the first sample image. The first sample image is an image with occlusion. The first sample image and the visible region mask of the first sample image are constructed by the following steps: superimposing a randomly selected second image on a first original image to obtain the first sample image with occlusion; superimposing the visible region mask of the first original image on the second image to obtain the visible region mask of the first sample image; the first original image is an image without occlusion.

[0243] A second training set is constructed, which includes multiple second training samples. The second training samples include second sample images and visible region masks of the second sample images. The second sample images are images with shadows caused by occlusion. The second sample images are constructed through the following steps: standardizing the second original image; simulating the natural falling process of the occluder under gravity using rigid body simulation technology on the standardized second original image; adding the shadow effect caused by the occluder to the second original image to obtain an intermediate image; and performing data augmentation on the intermediate image using various lighting settings to obtain the second sample image. The second original image is an image without occlusion.

[0244] The video diffusion model is trained using the first training set and the second training set, and the output of the video diffusion model is the video data corresponding to the training samples.

[0245] In some implementations, the training module specifically includes:

[0246] The training sample is input into the video diffusion model for training to obtain the video data corresponding to the training sample, wherein the training sample is either the first training sample or the second training sample.

[0247] Based on the training samples and the corresponding video data, the loss of the video diffusion model is calculated using the following formula:

[0248] L dm =Ε x,ε～N(0,1),t [||ε-ε θ (x t ,t,I in M vis )||2 ]

[0249] Among them, I in M represents the sample image. vis The visible region mask of the sample image is represented by t, where t represents time, ε represents noise, and ε ~ N(0,1) indicates that the noise follows a normal distribution. t ε represents the result of adding noise to the latent features of the sample image. θ (x t ,t,I in M vis ) represents the predicted noise obtained by the video diffusion model, and E represents the desired operation;

[0250] The parameters of the video diffusion model are updated based on the loss of the video diffusion model.

[0251] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0252] The apparatus 100 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0253] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0254] This application also provides a terminal device. Figure 13 This is a schematic diagram of the structure of a terminal device provided in Embodiment Six of this application, as shown below. Figure 13As shown, the terminal device 200 may include: a processor 201 with one or more processing cores, a memory 202 with one or more computer-readable storage media, and a computer program stored on the memory 202 and executable on the processor. The processor 201 and the memory 202 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0255] The processor 201 is the control center of the terminal device 200. It connects various parts of the terminal device 200 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 202, and calling data stored in the memory 202, it executes various functions of the terminal device 200 and processes data, thereby performing overall processing of the terminal device 200.

[0256] In this embodiment, the processor 201 in the terminal device 200 loads the instructions corresponding to the processes of one or more applications into the memory 202 according to the following steps, and the processor 201 runs the applications stored in the memory 202 to achieve the following functions:

[0257] A two-dimensional initial reference image is obtained, and an initial representation of a three-dimensional scene is generated based on the initial reference image, wherein the initial reference image is an isometric view;

[0258] Based on the initial representation of the three-dimensional scene, the objects in the three-dimensional scene are segmented to obtain N objects in the three-dimensional scene;

[0259] The N objects are rendered using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2;

[0260] Based on the candidate images and segmentation masks of the N objects, determine the occluded objects from the N objects;

[0261] A nonmodal completion method based on a video diffusion model is used to complete the candidate image of the occluded object;

[0262] For each of the N objects, a priori representation of the object's 3D structure is obtained based on the object's complete image and the camera parameters of the image. When the object is not occluded, the complete image of the object is the candidate image of the object; when the object is occluded, the complete image of the object is the complete image obtained by completing the object.

[0263] The prior representation of the object's three-dimensional structure, the noise of the prior representation of the object's three-dimensional structure, and the complete image of the object are input into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

[0264] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0265] In some embodiments, the processor 201 may include, but is not limited to: a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0266] In some embodiments of this application, the memory 202 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0267] Optional, such as Figure 13As shown, the terminal device 200 also includes: a display screen 203, a radio frequency circuit 204, an audio circuit 205, an input unit 206, and a power supply 207. The processor 201 is electrically connected to the display screen 203, the radio frequency circuit 204, the audio circuit 205, the input unit 206, and the power supply 207. Those skilled in the art will understand that... Figure 13 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0268] Display screen 203 is used to display images, videos, etc. Display screen 203 includes a display panel, which can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of computer devices. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof.

[0269] Optionally, the display screen 203 may also include a touch panel, and correspondingly, the display screen 203 can be a touch display screen. The touch panel can be used to collect user touch operations on or near it (such as user operations using fingers, styluses, or any suitable object or accessory on or near the touch panel), and generate corresponding operation instructions, which then execute the corresponding program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 201, and can also receive and execute commands from the processor 201. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 201 to determine the type of touch event, and then the processor 201 provides corresponding visual output on the display panel according to the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen to realize input and output functions. However, in some embodiments, the touch panel and the display panel can be implemented as two separate components to perform input and output functions. That is, the touch display screen can also be used as part of the input unit 206 to perform input functions.

[0270] The radio frequency circuit 204 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0271] Audio circuitry 205 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 205 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 205, converted back into audio data, and then processed by processor 201 before being transmitted via radio frequency circuitry 204 to, for example, another computer device, or output to memory 202 for further processing. Audio circuitry 205 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0272] The input unit 206 can be used to receive input numbers, characters, or object feature information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0273] Power supply 207 is used to supply power to various components of terminal device 200. Optionally, power supply 207 can be logically connected to processor 201 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 207 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The power management system can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance).

[0274] although Figure 13 As not shown in the diagram, the terminal device 200 may also include a camera, sensor components, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here. Sensors may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, etc.

[0275] It should be understood that the various components in the terminal device are connected through a bus system, which includes not only a data bus, but also a power bus, a control bus, and a status signal bus.

[0276] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0277] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the corresponding flow in the user position control method in the virtual scene of this application embodiment. For simplicity, further details are omitted here.

[0278] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0279] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0280] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing a three-dimensional scene, characterized in that, The method includes: A two-dimensional initial reference image is obtained, and an initial representation of a three-dimensional scene is generated based on the initial reference image, wherein the initial reference image is an isometric view; Based on the initial representation of the three-dimensional scene, the objects in the three-dimensional scene are segmented to obtain N objects in the three-dimensional scene; The N objects are rendered using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2; Based on the candidate images and segmentation masks of the N objects, determine the occluded objects from the N objects; A nonmodal completion method based on a video diffusion model is used to complete the candidate image of the occluded object; For each of the N objects, a priori representation of the object's 3D structure is obtained based on the object's complete image and the camera parameters of the image. When the object is not occluded, the complete image of the object is the candidate image of the object; when the object is occluded, the complete image of the object is the complete image obtained by completing the object. The prior representation of the object's three-dimensional structure, the noise of the prior representation of the object's three-dimensional structure, and the complete image of the object are input into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

2. The method according to claim 1, characterized in that, The acquisition of the initial two-dimensional reference image includes: The initial reference image is generated based on the 3D scene description text input by the user; Alternatively, the initial reference image input by the user can be received.

3. The method according to claim 2, characterized in that, The step of generating the initial reference image based on the scene description text input by the user includes: Multiple scene candidate images are generated based on the scene description text, and the scene candidate images are isometric views; Based on user input, an image is selected from the plurality of scene candidate images as the initial reference image.

4. The method according to claim 3, characterized in that, The step of generating multiple scene candidate images based on the scene description text, wherein the scene candidate images are isometric views, includes: Add a fixed prefix to the scene description text; the fixed prefix is ​​used to identify the generation of the isometric view. The two-dimensional generative model generates multiple scene candidate images based on the scene description text after adding the fixed prefix.

5. The method according to claim 1, characterized in that, The step of segmenting the objects in the three-dimensional scene according to the initial representation of the three-dimensional scene to obtain N objects in the three-dimensional scene includes: The initial representation of the three-dimensional scene is rendered using N2 preset viewpoints to obtain N2 scene images, where N2 is an integer greater than or equal to 2; The N2 scene images are segmented using an entity segmentation network to obtain N2 two-dimensional segmentation masks, wherein each scene image corresponds to a two-dimensional segmentation mask, and the two-dimensional segmentation mask is an instance-level mask that is independent of the category. The N2 two-dimensional segmentation masks are projected into a three-dimensional space using a semantic segmentation method to separate N objects in the three-dimensional scene.

6. The method according to claim 1, characterized in that, The step of determining the occluded object from the N objects based on the candidate images and segmentation masks of the N objects includes: For each object, N1 candidate images of the object and the segmentation mask corresponding to the N1 candidate images are input into the visual language model to obtain the occlusion detection result of the object. The occlusion detection result is used to indicate whether the object is occluded.

7. The method according to any one of claims 1-6, characterized in that, The method of using a nonmodal completion method based on a video diffusion model to complete the candidate image of the occluded object includes: For each candidate image of the occluded object, the candidate image is encoded by a variational autoencoder to obtain the latent features of the occluded object; The first input feature is obtained by concatenating the latent features of the occluded object and the visible area mask; Input the first input feature into the video diffusion model; The occluded image of the occluded object is encoded by the contrastive language-image pre-trained CLIP model to obtain the image features of the occluded object, and the image features are sent to the cross-attention module of the video diffusion model. The video diffusion model generates video data of the occluded object based on the first input feature and the image feature. The last frame of the video data is a complete image obtained by filling in the occluded object.

8. The method according to claim 7, characterized in that, The first input feature is obtained by concatenating the latent features of the occluded object and the visible region mask, including: The visible region mask of the candidate image of the occluded object is downsampled to obtain the downsampled visible region mask. The first input feature is obtained by concatenating the latent features of the occluded object and the downsampled visible area mask.

9. The method according to any one of claims 1-6, characterized in that, For each of the N objects, obtaining a priori representation of the object's 3D structure based on the object's complete image and the image's camera parameters includes: The initial geometric features of the object are obtained by processing the complete image of the object and the camera parameters of the image using a multi-view large-scale reconstruction model. The initial geometric features of the object are voxelized to obtain the display voxel representation of the object; The structural encoder encodes the explicit voxel representation of the object to obtain the implicit voxel representation of the object, which is a priori representation of the three-dimensional structure of the object.

10. The method according to claim 9, characterized in that, The native 3D generation model includes: a structure generator, a structure decoder, a structure feature generator, and a structure feature decoder; The step of inputting the prior representation of the object's 3D structure, the noise of the prior representation of the object's 3D structure, and the complete image of the object into the native 3D generation model to obtain the final 3D structure of the object includes: The structure generator processes the prior representation of the object's three-dimensional structure, the noise in the prior representation of the object's three-dimensional structure, and the complete image of the object to obtain the latent features of the object's sparse structure. The structure decoder decodes the latent features of the sparse structure of the object to obtain the low-dimensional features of the sparse structure of the object. The structural feature generator processes the low-dimensional features of the sparse structure of the object to obtain the latent structural features of the object. The structural feature decoder decodes the latent structural features of the object to obtain the final three-dimensional structure of the object.

11. The method according to claim 9, characterized in that, The noise in the prior representation of the object's three-dimensional structure is the noise corresponding to the intermediate time step of the corrected flow model.

12. The method according to claim 7, characterized in that, Also includes: A first training set is constructed, which includes multiple first training samples. The first training samples include a first sample image and a visible region mask of the first sample image. The first sample image is an image with occlusion. The first sample image and the visible region mask of the first sample image are constructed by the following steps: superimposing a randomly selected second image on a first original image to obtain the first sample image with occlusion; superimposing the visible region mask of the first original image on the second image to obtain the visible region mask of the first sample image; the first original image is an image without occlusion. A second training set is constructed, which includes multiple second training samples. The second training samples include second sample images and visible region masks of the second sample images. The second sample images are images with shadows caused by occlusion. The second sample images are constructed through the following steps: standardizing the second original image; simulating the natural falling process of the occluder under gravity using rigid body simulation technology on the standardized second original image; adding the shadow effect caused by the occluder to the second original image to obtain an intermediate image; and performing data augmentation on the intermediate image using various lighting settings to obtain the second sample image. The second original image is an image without occlusion. The video diffusion model is trained using the first training set and the second training set, and the output of the video diffusion model is the video data corresponding to the training samples.

13. The method according to claim 12, characterized in that, The step of training the video diffusion model based on the first training set and the second training set, wherein the output of the video diffusion model is the video data corresponding to the training samples, includes: The training sample is input into the video diffusion model for training to obtain the video data corresponding to the training sample, wherein the training sample is either the first training sample or the second training sample. Based on the training samples and the corresponding video data, the loss of the video diffusion model is calculated using the following formula: L dm =E x,ε～N(0,1),t [||e-e θ (x t ,t,I in ,M vis )|| 2 ] Among them, I in M represents the sample image. vis The visible region mask of the sample image is represented by t, where t represents time, ε represents noise, and ε ~ N(0,1) indicates that the noise follows a normal distribution. t ε represents the result of adding noise to the latent features of the sample image. θ (x t ,t,I in M vis ) represents the predicted noise obtained by the video diffusion model, and E represents the desired operation; The parameters of the video diffusion model are updated based on the loss of the video diffusion model.

14. A device for constructing a three-dimensional scene, characterized in that, The device includes: The scene initialization module is used to acquire a two-dimensional initial reference image and generate an initial representation of the three-dimensional scene based on the initial reference image, wherein the initial reference image is an isometric view. The segmentation module is used to segment the objects in the three-dimensional scene according to the initial representation of the three-dimensional scene, so as to obtain N objects in the three-dimensional scene; The rendering module renders the N objects using N1 preset viewpoints to obtain N1 candidate images for each object, where N1 is an integer greater than or equal to 2. An occlusion determination module is used to determine the occluded objects from the N objects based on the candidate images and segmentation masks of the N objects. The completion module is used to complete the candidate image of the occluded object using a non-modal completion method based on a video diffusion model; The generation module is used to obtain a priori representation of the three-dimensional structure of each of the N objects based on the complete image of the object and the camera parameters of the image. When the object is not occluded, the complete image of the object is the candidate image of the object. When the object is occluded, the complete image of the object is the complete image obtained by completing the object. The generation module is further configured to input the prior representation of the object's three-dimensional structure, the noise of the prior representation of the object's three-dimensional structure, and the complete image of the object into the native three-dimensional generation model to obtain the final three-dimensional structure of the object.

15. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method of any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 13.

Images