3D scene generation method and device, electronic equipment, storage medium and product

Abstract

Embodiments of the present disclosure disclose a 3D scene generation method and device, electronic equipment, storage medium and product, wherein the method comprises: determining an incomplete image and an incomplete depth map corresponding to a second pose based on a first global 3D scene graph corresponding to a first pose; the second pose is determined based on processing of the first pose; processing the incomplete image and received text description information through a first network model to determine a complete image; processing the incomplete depth map and the complete image through a second network model to determine a complete depth map; processing the second pose, the complete depth map and the complete image through a third network model to determine a second global 3D scene graph; the second global 3D scene graph at least includes the first global 3D scene graph.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a 3D scene generation method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] Existing technologies have disclosed various methods for offline generation of 3D scenes from single images. These methods typically involve generating multiple views or panoramas of the scene and then converting them into a 3D representation. For example, methods like Text2Room and LucidDreamer start with an input image and a user's text description, generate multiple scene images, and then use 3D optimization to refine the scene and create a more accurate and consistent 3D representation. Methods like GenEx and Dreamscene360 utilize pre-trained text-to-panorama diffusion models to synthesize coherent panoramas, which are then upscaled to 3D, ultimately producing an explorable 3D world. However, existing 3D scene generation methods generally suffer from time consumption and low efficiency. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a 3D scene generation method, apparatus, electronic device, storage medium, and product.

[0004] According to one aspect of the present disclosure, a 3D scene generation method is provided, comprising:

[0005] Based on the first global 3D scene map corresponding to the first pose, the incomplete image and incomplete depth map corresponding to the second pose are determined; the second pose is determined based on the first pose after processing.

[0006] The incomplete image and the received text description information are processed by the first network model to determine the complete image;

[0007] The incomplete depth map and the complete image are processed by a second network model to determine the complete depth map;

[0008] The second pose, the complete depth map, and the complete image are processed by a third network model to determine a second global 3D scene map; the second global 3D scene map includes at least the first global 3D scene map.

[0009] Optionally, determining the incomplete image and incomplete depth map corresponding to the second pose based on the first global 3D scene map corresponding to the first pose includes:

[0010] Obtain the first global 3D scene map corresponding to the first pose;

[0011] The first pose is processed based on the received rotation and translation matrix to obtain the second pose;

[0012] The first global 3D scene map is processed according to the rotation and translation matrix to determine the incomplete image and the incomplete depth map corresponding to the second pose.

[0013] Optionally, obtaining the first global 3D scene map corresponding to the first pose includes:

[0014] The received first pose, preset image, and preset depth map are processed using a third network model to determine the first global 3D scene map; or...

[0015] The second global 3D scene graph is used as the first global 3D scene graph.

[0016] Optionally, the step of processing the second pose, the complete depth map, and the complete image through a third network model to determine the second global 3D scene map includes:

[0017] The complete image is feature extracted through the backbone sub-network in the third network model to obtain matching features and image features;

[0018] Based on the matching features and the second pose, the complete depth map is optimized to determine the optimized depth map;

[0019] Based on the image features and the optimized depth map, a second local 3D scene map is determined;

[0020] The second global 3D scene map is determined based on the second local 3D scene map and the first global 3D scene map.

[0021] Optionally, the step of optimizing the complete depth map based on the matching features and the second pose to determine the optimized depth map includes:

[0022] Based on the rotation and translation matrix corresponding to the second pose, the historical matching features that are nearest neighbors to the second pose are obtained from the feature memory; the feature memory contains multiple historical matching features corresponding to different poses.

[0023] The optimized depth map is determined by processing the historical matching features, the matching features, and the complete depth map through the cost volume sub-network in the third network model.

[0024] Optionally, the step of processing the historical matching features, the matching features, and the complete depth map through the cost volume sub-network in the third network model to determine the optimized depth map includes:

[0025] Multiple depth candidate values ​​are generated based on the complete depth map;

[0026] Calculate the similarity between the matching feature and the historical matching feature in the depth space corresponding to the plurality of depth candidate values, and determine the probability value of each depth candidate value among the plurality of depth candidate values;

[0027] The optimized depth map is determined by performing a weighted summation on the plurality of depth candidate values ​​using the probability values ​​as weights.

[0028] Optionally, determining the second local 3D scene map based on the image features and the optimized depth map includes:

[0029] The target depth map and Gaussian model are obtained by processing the target depth value and the image features through the segmentation sub-network in the third network model.

[0030] Based on the target depth map, the Gaussian model is projected into 3D space to obtain the second local 3D scene map.

[0031] Optionally, determining the second global 3D scene map based on the second local 3D scene map and the first global 3D scene map includes:

[0032] Project the second local 3D scene map and the first global 3D scene map onto the pixel coordinate system;

[0033] In the pixel coordinate system, filter out 3D image information in the second local 3D scene map and the first global 3D scene map that does not meet the depth consistency constraint;

[0034] The second local 3D scene map and the 3D image information that meets the depth consistency constraint after being filtered in the first global 3D scene map are stitched together to obtain the second global 3D scene map.

[0035] Optionally, before processing the incomplete image and the received text description information through the first network model to determine the complete image, the method further includes:

[0036] The first initial network model is trained using a training dataset to obtain the trained first initial network model; the training dataset includes 3D images of various scenes, and each 3D image has a known scene category;

[0037] The first initial network model is obtained by compressing and accelerating the first network model using model distillation technology.

[0038] Optionally, the step of processing the incomplete depth map and the complete image using a second network model to determine the complete depth map includes:

[0039] Based on the incomplete depth map, determine the binary validity mask;

[0040] The incomplete depth map, the binary validity mask, and the complete image are input into the second network model; the second network model is obtained through training.

[0041] According to another aspect of the embodiments of this disclosure, a 3D scene generation apparatus is provided, comprising:

[0042] The preprocessing module is used to determine the incomplete image and incomplete depth map corresponding to the second pose based on the first global 3D scene map corresponding to the first pose; the second pose is determined based on the first pose after processing.

[0043] The first processing module is used to process the incomplete image and the received text description information through a first network model to determine the complete image;

[0044] The second processing module is used to process the incomplete depth map and the complete image through a second network model to determine the complete depth map;

[0045] The third processing module is used to process the second pose, the complete depth map, and the complete image through a third network model to determine a second global 3D scene map; the second global 3D scene map includes at least the first global 3D scene map.

[0046] Optionally, the preprocessing module is specifically used to obtain a first global 3D scene map corresponding to the first pose; process the first pose based on the received rotation and translation matrix to obtain a second pose; process the first global 3D scene map according to the rotation and translation matrix to determine the incomplete image and the incomplete depth map corresponding to the second pose.

[0047] Optionally, when the preprocessing module obtains the first global 3D scene map corresponding to the first pose, it is used to process the received first pose, preset image and preset depth map through a third network model to determine the first global 3D scene map; or, the second global 3D scene map is used as the first global 3D scene map.

[0048] Optionally, the third processing module includes:

[0049] The feature extraction unit is used to extract features from the complete image through the backbone sub-network in the third network model to obtain matching features and image features.

[0050] A depth optimization unit is used to optimize the complete depth map based on the matching features and the second pose to determine an optimized depth map;

[0051] A local scene generation unit is used to determine a second local 3D scene map based on the image features and the optimized depth map;

[0052] The incremental fusion unit is used to determine the second global 3D scene map based on the second local 3D scene map and the first global 3D scene map.

[0053] Optionally, the depth optimization unit is specifically used to obtain the historical matching features that are nearest neighbors to the second pose from the feature memory based on the rotation and translation matrix corresponding to the second pose; the feature memory pre-stores the historical matching features corresponding to multiple different poses; and processes the historical matching features, the matching features and the complete depth map through the cost volume sub-network in the third network model to determine the optimized depth map.

[0054] Optionally, when the depth optimization unit processes the historical matching features, the matching features, and the complete depth map through the cost volume sub-network in the third network model to determine the optimized depth map, it generates multiple depth candidate values ​​based on the complete depth map; calculates the similarity between the matching features and the historical matching features in the depth space corresponding to the multiple depth candidate values, and determines the probability value of each of the multiple depth candidate values; and performs a weighted summation on the multiple depth candidate values ​​with the probability value as the weight to determine the optimized depth map.

[0055] Optionally, the local scene generation unit is specifically used to process the target depth value and the image features through the segmentation sub-network in the third network model to obtain a target depth map and a Gaussian model; and to project the Gaussian model onto 3D space based on the target depth map to obtain the second local 3D scene map.

[0056] Optionally, the incremental fusion unit is specifically used to project the second local 3D scene map and the first global 3D scene map into a pixel coordinate system; filter 3D image information in the second local 3D scene map and the first global 3D scene map that does not meet the depth consistency constraint in the pixel coordinate system; and perform stitching on the filtered 3D image information in the second local 3D scene map and the first global 3D scene map that meets the depth consistency constraint to obtain the second global 3D scene map.

[0057] Optionally, the device further includes:

[0058] The model training module is used to train a first initial network model using a training dataset to obtain the trained first initial network model; the training dataset includes 3D images of various scenes, and each 3D image has a known scene category; the first initial network model is compressed and accelerated using model distillation technology to obtain the first network model.

[0059] Optionally, the second processing module is specifically used to determine a binary validity mask based on the incomplete depth map; input the incomplete depth map, the binary validity mask, and the complete image into the second network model; and the second network model is obtained through training.

[0060] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0061] Memory, used to store computer program products;

[0062] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the 3D scene generation method described in any of the above embodiments.

[0063] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the 3D scene generation method described in any of the above embodiments.

[0064] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the 3D scene generation method described in any of the above embodiments.

[0065] Based on the 3D scene generation method, apparatus, electronic device, storage medium, and product provided in the above embodiments of this disclosure, an incomplete image and an incomplete depth map corresponding to a second pose are determined based on a first global 3D scene map corresponding to a first pose; the second pose is determined by processing the first pose; a complete image is determined by processing the incomplete image and received text description information through a first network model; a complete depth map is determined by processing the incomplete depth map and the complete image through a second network model; a second global 3D scene map is determined by processing the second pose, the complete depth map, and the complete image through a third network model; the second global 3D scene map includes at least the first global 3D scene map; this embodiment simplifies the 3D scene generation process and improves the speed of 3D scene generation through the first network model, the second network model, and the third network model, and allows user interaction during the 3D scene generation process, realizing diverse and coherent 3D scene generation.

[0066] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0067] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0068] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0069] Figure 1 This is a flowchart illustrating a 3D scene generation method provided in an exemplary embodiment of this disclosure;

[0070] Figure 2 This is a public announcement Figure 1 A flowchart illustrating step 102 in the illustrated embodiment;

[0071] Figure 3 This is a public announcement Figure 1 A flowchart illustrating step 108 in the illustrated embodiment;

[0072] Figure 4 This is a schematic diagram of the structure of the third network model in the 3D scene generation method provided in an exemplary embodiment of this disclosure;

[0073] Figure 5 This is a schematic diagram illustrating an optional example of a 3D scene generation method provided in another exemplary embodiment of this disclosure;

[0074] Figure 6 This is a schematic diagram of the structure of a 3D scene generation apparatus provided in an exemplary embodiment of this disclosure;

[0075] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0076] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0077] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0078] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0079] It should also be understood that in the embodiments of this disclosure, "a plurality of" can refer to two or more, and "at least one" can refer to one, two or more.

[0080] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0081] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.

[0082] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0083] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0084] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0085] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0086] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0087] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0088] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0089] Application Overview

[0090] In developing this disclosure, the inventors discovered that existing 3D scene generation methods suffer from at least the following problems: they typically operate offline, preventing user interaction during the generation process. For example, WonderJourney uses LLM to generate scene descriptions, employs a text-driven process to achieve coherent 3D scene generation, and utilizes a visual-language model for result verification. This process takes several minutes and is unsuitable for interactive use. WonderJourney has reduced reconstruction time by using FastLAyered Gaussian Surfels and generates geometrically consistent scenes through guided diffusion depth estimation, but each scene still takes a considerable amount of time. To improve the efficiency of 3D scene generation, the inventors propose the following 3D scene generation method.

[0091] Exemplary methods

[0092] Figure 1 This is a schematic flowchart of a 3D scene generation method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0093] Step 102: Based on the first global 3D scene map corresponding to the first pose, determine the incomplete image and incomplete depth map corresponding to the second pose.

[0094] The second pose is determined by processing the first pose.

[0095] In this embodiment, the 3D scene generation process is progressively expanded. Each time, a local 3D scene map is generated based on a simulated camera (viewpoint) at one pose. The 3D scene range is expanded by continuously changing the simulated camera from one pose to another. Therefore, in this embodiment, the first pose can be understood as the previous pose, and the second pose can be understood as the next pose. When the current local scene map is the i-th iteration, the second global 3D scene map obtained in the (i-1)-th iteration is used as the first global 3D scene map in the i-th iteration. Optionally, the second pose can be obtained by rotating and / or translating the first pose. The corresponding rotation and translation matrix can be preset (by using the preset rotation and translation matrix to expand the scene by a preset size each time, realizing the progressive generation of a complete 3D scene map) or it can be user-inputted. When the user-input rotation and translation matrix is ​​received, real-time interactive 3D scene map generation is realized.

[0096] Step 104: The incomplete image and the received text description information are processed by the first network model to determine the complete image.

[0097] In one embodiment, the text description information can be user-inputted or randomly matched preset text description information; when the user-inputted text description information is received, real-time interactive 3D scene graph generation is realized; in this embodiment, the incomplete image is a 2D image with only partial content based on the second pose, which is abstracted from the first global 3D scene graph. The first network model combines the partial 2D image and the text description information to generate a complete image that completes the 2D image with the corresponding text description content. For example, if the incomplete image includes a partial tree canopy and the user-inputted text description information is "trees", the first network model will output the complete tree image content.

[0098] Step 106: Process the incomplete depth map and the complete image using the second network model to determine the complete depth map.

[0099] In this embodiment, since the second pose is obtained by rotating and translating the first pose, the depth map corresponding to the second pose is an incomplete depth map when only the first global 3D scene map is known. That is, the depth of the part of the incomplete depth map that does not correspond to the first global 3D scene map is unknown. This embodiment uses the second network model in combination with the complete image to fill in the depth part of the incomplete depth map, which can effectively fill in the depth information and enhance the consistency and integrity of the 3D scene.

[0100] Step 108: Process the second pose, complete depth map, and complete image using the third network model to determine the second global 3D scene map.

[0101] The second global 3D scene graph includes at least the first global 3D scene graph.

[0102] In this embodiment, after obtaining a complete image based on the first network model and a complete depth map based on the second network model, the local 3D scene map corresponding to the second pose can be determined by combining the second pose with the third network model. The local 3D scene map is then stitched together according to the rotation and translation relationship between the first pose and the second pose to obtain a second global 3D scene map with an expanded field of view.

[0103] The 3D scene generation method provided in the above embodiments of this disclosure determines an incomplete image and an incomplete depth map corresponding to a second pose based on a first global 3D scene map corresponding to a first pose. The second pose is determined by processing the first pose. A complete image is determined by processing the incomplete image and received text description information through a first network model. A complete depth map is determined by processing the incomplete depth map and the complete image through a second network model. A second global 3D scene map is determined by processing the second pose, the complete depth map, and the complete image through a third network model. The second global 3D scene map includes at least the first global 3D scene map. This embodiment simplifies the 3D scene generation process and improves the speed of 3D scene generation through the first, second, and third network models. Furthermore, it allows user interaction during the 3D scene generation process, realizing diverse and coherent 3D scene generation.

[0104] This embodiment implements a novel framework specifically designed for real-time interactive 3D scene generation. To address the key challenge of inference efficiency, optimizations have been made in both geometric representation and appearance modeling.

[0105] like Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 102 may include the following steps:

[0106] Step 1021: Obtain the first global 3D scene map corresponding to the first pose.

[0107] Optionally, the received first pose, preset image, and preset depth map are processed by a third network model to determine a first global 3D scene map; or, a second global 3D scene map is used as the first global 3D scene map.

[0108] In this embodiment, the first global 3D scene map is usually the second global 3D scene map output from the previous iteration during the scene expansion iteration process. In addition, there is a special scenario where, before the iteration starts, when the first global 3D scene map is obtained for the first time, the preset image of the received plane, the first pose (initial pose) input by the user, and the preset depth map (which can be determined or randomly generated according to the depth information input by the user) are input into the third network model, and the first global 3D scene map can be generated through the third network model.

[0109] Step 1022: Process the first pose based on the received rotation and translation matrix to obtain the second pose.

[0110] Optionally, in this embodiment, the rotation and translation matrix is ​​an n*n matrix, for example, a 4*4 matrix; the rotation and translation matrix represents the rotation angle and translation distance required to move from the first pose to the second pose; in this embodiment, the rotation and translation matrix input by the user can be received through a preset interface.

[0111] Step 1023: Process the first global 3D scene map according to the rotation and translation matrix to determine the incomplete image and incomplete depth map corresponding to the second pose.

[0112] In this embodiment, the first global 3D scene map corresponds to the first pose. To obtain the portion of the first global 3D scene map corresponding to the second pose, the content of the first global 3D scene map can be filtered using a rotation and translation matrix. For example, the selection frame (3D image display area) corresponding to the first global 3D scene map can be rotated and translated using a rotation and translation matrix, retaining only the portion still displayed in the selection frame after the rotation and translation matrix processing. This results in an incomplete 3D scene map. At this point, decomposing the incomplete 3D scene map yields an incomplete image and an incomplete depth map. In this embodiment, the pose of the simulated camera is changed using a rotation and translation matrix to obtain the incomplete image and incomplete depth map corresponding to the first global 3D scene map when the simulated camera is in the second pose, thus providing the basis for subsequently obtaining the second global 3D scene map corresponding to the second pose.

[0113] like Figure 3 As shown above, in the above Figure 1 Based on the illustrated embodiment, step 108 may include the following steps:

[0114] Step 1081: Extract features from the complete image using the backbone sub-network in the third network model to obtain matching features and image features.

[0115] Optionally, the third network model can be StepSplat, which includes at least a backbone sub-network (the network structure of which may include several convolutional layers), through which feature extraction is performed on the complete image to obtain matching features and image features.

[0116] Step 1082: Optimize the complete depth map based on matching features and the second pose to determine the optimized depth map.

[0117] Optionally, the complete depth map can be optimized based on the depth-guided cost volume. In this embodiment, since the complete image and the complete depth map come from the outputs of different network models, there may be a mismatch. This embodiment improves the correspondence between the optimized depth map and the complete image through optimization, thus solving the problem of mismatch between the generated 3D scene image and the depth.

[0118] Step 1083: Determine the second local 3D scene map based on image features and optimized depth map.

[0119] In this embodiment, given an optimized depth map, the image features can be projected into 3D space by optimizing the depth information included in the depth map to obtain a second local 3D scene map.

[0120] Step 1084: Determine the second global 3D scene map based on the second local 3D scene map and the first global 3D scene map.

[0121] In this embodiment, the third network model solves the geometric optimization in 3D scene generation, enabling rapid 3D scene expansion (e.g., 3D scene expansion within 0.26 seconds). Unlike traditional 3DGS methods that rely on iterative training to update 3D representations, StepSplat borrows ideas from recent feedforward methods to directly perform 3DGS inference, significantly improving the efficiency of 3D scene expansion.

[0122] In some alternative embodiments, step 1082 may include:

[0123] Based on the rotation and translation matrix corresponding to the second pose, historical matching features of the nearest neighbor to the second pose are obtained from the feature memory.

[0124] The feature memory stores multiple historical matching features corresponding to different poses, which can be called by the cost volume subnetwork when calculating other poses.

[0125] The cost volume sub-network in the third network model is used to process historical matching features, matching features, and the complete depth map to determine the optimized depth map.

[0126] Optionally, multiple depth candidate values ​​are generated based on the complete depth map; the similarity between the matching features and historical matching features is calculated in the depth space corresponding to the multiple depth candidate values ​​to determine the probability value of each depth candidate value; and a weighted summation is performed on the multiple depth candidate values ​​with the probability value as the weight to determine the optimized depth map.

[0127] Optionally, this embodiment uses a Depth-Guided Cost Volume subnetwork to retrieve historical matching features from the feature memory corresponding to the historical pose (e.g., the first pose) that is closest to the second pose corresponding to the matching feature. Multiple depth candidate values ​​are generated based on the input complete depth map to form a cost volume. That is, by calculating the normalized dot product correlation (similarity value) between the matching feature and the historical matching feature in the space corresponding to different depth candidate values, this similarity value reflects the probability value of different depth candidate values. The multiple depth candidate values ​​are weighted and summed using this probability value to determine the optimized depth map (cost volume), which is more accurate than the multiple depth candidate values ​​randomly generated in the prior art.

[0128] In this embodiment, a feature memory is maintained to store the matching features of each iteration's pose, enabling the third network model to extend the feedforward paradigm to interactive 3D geometric representation and ensuring consistency during dynamic viewpoint changes. This module adaptively builds the cost volume as the viewpoint changes.

[0129] In terms of network structure, the third network model also includes a segmentation subnetwork, which receives the target depth value output by the bulk subnetwork and the image features output by the backbone subnetwork. Correspondingly, step 1083 may include:

[0130] The target depth map and Gaussian model are obtained by processing the target depth value and image features through the segmentation sub-network in the third network model.

[0131] The Gaussian model is projected onto the 3D space based on the target depth map to obtain the second local 3D scene map.

[0132] In this embodiment, the segmentation subnetwork can be a 2DU-Net network. Through this segmentation subnetwork, the Gaussian model can be projected onto the 3D space based on the target depth map, thereby obtaining the second local 3D scene map corresponding to the second pose.

[0133] In some optional embodiments, based on the second local 3D scene map corresponding to the second pose obtained in the above steps, step 1084 may include:

[0134] Project the second local 3D scene map and the first global 3D scene map onto the pixel coordinate system;

[0135] In the pixel coordinate system, filter out 3D image information that does not conform to the depth consistency constraint in the second local 3D scene image and the first global 3D scene image;

[0136] The 3D image information that meets the depth consistency constraint after being filtered from the second local 3D scene map and the first global 3D scene map is stitched together to obtain the second global 3D scene map.

[0137] In this embodiment, both the first global 3D scene graph and the second local 3D scene graph can be represented using a Gaussian model.

[0138] Optionally, the fusion of local 3D scene maps implemented in this embodiment can be based on an incremental fusion module. The incremental fusion module updates the local Gaussian model to the global Gaussian model through depth constraints (depth constraint refers to calculating the depth difference between corresponding parts in the second local 3D scene map and the first global 3D scene map, identifying parts with smaller depth differences as redundant parts; optionally, the corresponding parts can be determined based on the relationship between the first pose and the second pose to identify parts with similar content in the two 3D scene maps; the 3D scene maps of the similar parts are projected onto the second pose to obtain a 2D image; potentially redundant points are identified, and then the depth difference is calculated) to ensure a continuous and consistent 3D scene representation. This process includes projecting all global Gaussian models to the current pixel coordinate system, filtering Gaussian models that violate depth consistency constraints, and merging only valid local Gaussian models.

[0139] Figure 4 This is a schematic diagram of the structure of the third network model in the 3D scene generation method provided in an exemplary embodiment of this disclosure. For example... Figure 4 As shown, the third network model includes a backbone sub-network, a cost volume sub-network, a feature memory, and a segmentation sub-network.

[0140] The backbone subnetwork receives the complete image, extracts features, and outputs matching features and image features. The obtained matching features are input into the cost volume subnetwork and the feature memory; the image features are input into the segmentation subnetwork.

[0141] The feature memory receives and saves matching features. It receives the second pose, finds the historical matching features corresponding to the poses that are nearest neighbors of the second pose, and inputs the historical matching features into the cost volume subnetwork.

[0142] The cost volume sub-network receives matching features, historical matching features, and a complete depth map; it then obtains an optimized depth map, which is input into the segmentation sub-network.

[0143] The segmentation sub-network receives image features and optimizes the depth map to obtain a second local 3D scene map.

[0144] In addition, the third network model may also include an incremental fusion module. The incremental fusion module receives a second local 3D scene graph and a first global 3D scene graph, performs depth matching on the second local 3D scene graph and the first global 3D scene graph, filters out the parts in the second local 3D scene graph that violate depth consistency constraints, and concatenates the other parts with the first global 3D scene graph to obtain the second global 3D scene graph.

[0145] The first, second, and third network models provided in the above embodiments of this disclosure are pre-trained before application. To train the first and second network models, this disclosure provides a training set specifically for interactive 3D scene generation training. This embodiment constructs a dataset based on various existing 3D scene generation methods and uses this dataset to train all network models (the first, second, and third network models). Optionally, multiple 3D scene generation methods (which can be any existing 3D scene generation method) are used to create 3D scene graphs that each method excels at, and a VLM model is used to verify whether the generated data conforms to the defined scene. The final dataset contains over 6 million frames rendered through simulated interactive trajectories, including rotating paths, linear movement, and mixed trajectories. The dataset provided in this embodiment covers most 3D task scenes, enabling more comprehensive training of the network models. For example, it covers four categories: indoor environments, urban landscapes, natural terrain, and stylized art scenes.

[0146] In some optional embodiments, before processing the incomplete image and the received text description information through a first network model to determine the complete image, the method further includes:

[0147] The first initial network model is trained using a training dataset to obtain the trained first initial network model; the training dataset includes 3D images of various scenes, and each 3D image has a known scene category;

[0148] The first initial network model is obtained by compressing and accelerating the first network model using model distillation technology.

[0149] The first network model (FastPaint) proposed in this embodiment does not restrict the network structure and can adopt any network model structure for image completion. This embodiment reduces the inference steps of the first network model to two steps through model distillation, and enhances the repair capability of the pre-trained model through distillation and fine-tuning, making it suitable for interactive 3D generation. Optionally, by synergistically utilizing the advantages of ODE trajectory preservation and ODE trajectory reconstruction, knowledge distillation is performed on the first network model, reducing the required inference steps while maintaining the quality of appearance modeling.

[0150] In this embodiment, to address the issue of the inpainting region differing from the fine-tuning stage in 3D scene generation, a dedicated dataset (including datasets of various 3D scenes) was constructed for training FastPaint. Simulated camera poses and partial image occlusion using masks were designed to simulate the interactive 3D generation process. By acquiring depth maps and images and using projection to obtain masks, the dataset was ensured to meet the specific requirements of the inpainting task within this context. The construction of this dataset is similar to the methods used in third-party and second-party network models, particularly in simulating camera trajectories, helping the model adapt to interactive 3D scene generation.

[0151] In some alternative embodiments, the process of determining the complete depth map using a second network model may include:

[0152] Determine the binary validity mask based on the incomplete depth map;

[0153] The binary validity mask indicates which coordinates in the complete image have depth and which do not, and is determined based on the incomplete depth map.

[0154] The incomplete depth map, binary validity mask, and complete image are input into the second network module.

[0155] The second network model was obtained through training.

[0156] In this embodiment, the second network model (QuickDepth) is trained based on a lightweight depth estimation model. This embodiment does not limit the network structure of the second network model and can use existing network structures. The input to the second network model includes the complete image corresponding to the second pose (output of the first network model), an incomplete depth map, and a binary validity mask. To meet the needs of interactive 3D scene generation, a dataset containing diverse scenes from indoor and outdoor environments, comics, and artworks was created, and multiple camera poses were set to simulate the realistic 3D scene generation process. In the processing of each frame, the depth map of the previous frame is transformed to the coordinate system of the current frame through relative pose transformation, thereby generating an incomplete depth map, and marking the binary validity mask of the area to be filled. During training, sometimes the true depth map of the target frame is completely obscured, and sometimes the deformed depth map and the corresponding mask are used as input. The prediction results are supervised by the L1 loss function to ensure that the prediction results are close to the true depth. Figure 1 This approach enhances the performance and accuracy of the second network model in complex and varied scenes. It enables the second network model to effectively fill in depth information, thereby improving the consistency and integrity of the 3D scene.

[0157] Figure 5 This is a schematic diagram illustrating an optional example of a 3D scene generation method provided in another exemplary embodiment of this disclosure. For example... Figure 5 As shown, the network structure in this embodiment includes a first network model, a second network model, and a third network model. This embodiment only shows the process of the i-th iteration.

[0158] The input data is the second global 3D scene map output from the (i-1)th iteration, which is used as the first global 3D scene map for the current iteration. The rotation and translation matrix of the i-th iteration is received from the user (to realize user interaction), and the second pose is determined. The first global 3D scene map is processed according to the second pose, and the resulting incomplete image is input into the first network model, and the resulting incomplete depth map is input into the second network model.

[0159] The first network model receives an incomplete image and text description information input by the user (to enable user interaction), completes the incomplete image using the text description information, and then inputs the complete image into the second and third network models.

[0160] The second network model receives a complete image and an incomplete depth map. Based on the incomplete depth map, it determines a binary validity mask to complete the depth of the incomplete depth map, obtains a complete depth map, and inputs the complete depth map into the third network model.

[0161] The third network model receives the complete image, the complete depth map, and the second pose corresponding to the i-th iteration, and determines the second local 3D scene map corresponding to the second pose.

[0162] It may also include an incremental fusion module (the function of which can be integrated into the third network model or independent of the third network model), which merges the second local 3D scene graph with the first global 3D scene graph to obtain the second global 3D scene graph after the i-th iteration expansion. By continuing to iterate, the global 3D scene graph can be finally obtained.

[0163] To further enhance depth consistency, this embodiment integrates a lightweight depth completion function (QuickDepth) into a second network model, providing a consistent depth prior for the third network model to construct the cost volume. Regarding appearance, a first network model (FastPaint) is proposed as an efficient method for real-time appearance refinement. Compared to traditional diffusion-based repair methods (sd) that require dozens of inference steps to refine the appearance model, the first network model achieves similar results in only two inference steps while maintaining spatial appearance consistency. This method significantly improves processing speed, enabling real-time interactive experiences.

[0164] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0165] Exemplary device

[0166] Figure 6 This is a schematic diagram of the structure of a 3D scene generation apparatus provided in an exemplary embodiment of this disclosure. For example... Figure 6 As shown, the apparatus provided in this embodiment includes:

[0167] The preprocessing module 61 is used to determine the incomplete image and incomplete depth map corresponding to the second pose based on the first global 3D scene map corresponding to the first pose.

[0168] The second pose is determined by processing the first pose.

[0169] The first processing module 62 is used to process the incomplete image and the received text description information through the first network model to determine the complete image.

[0170] The second processing module 63 is used to process the incomplete depth map and the complete image through the second network model to determine the complete depth map.

[0171] The third processing module 64 is used to process the second pose, the complete depth map, and the complete image through the third network model to determine the second global 3D scene map.

[0172] The second global 3D scene graph includes at least the first global 3D scene graph.

[0173] The 3D scene generation apparatus provided in the above embodiments of this disclosure determines an incomplete image and an incomplete depth map corresponding to a second pose based on a first global 3D scene map corresponding to a first pose. The second pose is determined by processing the first pose. A complete image is determined by processing the incomplete image and received text description information through a first network model. A complete depth map is determined by processing the incomplete depth map and the complete image through a second network model. A second global 3D scene map is determined by processing the second pose, the complete depth map, and the complete image through a third network model. The second global 3D scene map includes at least the first global 3D scene map. This embodiment simplifies the 3D scene generation process and improves the speed of 3D scene generation through the first, second, and third network models. Furthermore, it allows user interaction during the 3D scene generation process, realizing diverse and coherent 3D scene generation.

[0174] In some optional embodiments, the preprocessing module 61 is specifically used to obtain the first global 3D scene map corresponding to the first pose; process the first pose based on the received rotation and translation matrix to obtain the second pose; process the first global 3D scene map according to the rotation and translation matrix to determine the incomplete image and incomplete depth map corresponding to the second pose.

[0175] Optionally, when the preprocessing module obtains the first global 3D scene map corresponding to the first pose, it processes the received first pose, preset image and preset depth map through the third network model to determine the first global 3D scene map; or, it uses the second global 3D scene map as the first global 3D scene map.

[0176] In some optional embodiments, the third processing module 64 includes:

[0177] The feature extraction unit is used to extract features from the complete image through the backbone sub-network in the third network model to obtain matching features and image features.

[0178] The depth optimization unit is used to optimize the complete depth map based on matching features and the second pose to determine the optimized depth map.

[0179] A local scene generation unit is used to determine a second local 3D scene map based on image features and an optimized depth map;

[0180] The incremental fusion unit is used to determine the second global 3D scene map based on the second local 3D scene map and the first global 3D scene map.

[0181] Optionally, the depth optimization unit is specifically used to obtain the historical matching features that are the nearest neighbors of the second pose from the feature memory based on the rotation and translation matrix corresponding to the second pose; the feature memory pre-stores multiple historical matching features corresponding to different poses; and processes the historical matching features, matching features and complete depth map through the cost volume sub-network in the third network model to determine the optimized depth map.

[0182] Optionally, when the depth optimization unit processes historical matching features, matching features, and the complete depth map through the cost volume sub-network in the third network model to determine the optimized depth map, it generates multiple depth candidate values ​​based on the complete depth map; calculates the similarity between the matching features and historical matching features in the depth space corresponding to the multiple depth candidate values, and determines the probability value of each depth candidate value among the multiple depth candidate values; and performs a weighted summation on the multiple depth candidate values ​​with the probability values ​​as weights to determine the optimized depth map.

[0183] Optionally, the local scene generation unit is specifically used to process the target depth value and image features through the segmentation sub-network in the third network model to obtain a target depth map and a Gaussian model; and to project the Gaussian model onto the 3D space based on the target depth map to obtain a second local 3D scene map.

[0184] Optionally, the incremental fusion unit is specifically used to project the second local 3D scene map and the first global 3D scene map into a pixel coordinate system; filter out 3D image information in the second local 3D scene map and the first global 3D scene map that does not meet the depth consistency constraint in the pixel coordinate system; and perform stitching on the filtered 3D image information in the second local 3D scene map and the first global 3D scene map that meets the depth consistency constraint to obtain the second global 3D scene map.

[0185] In some optional embodiments, the apparatus provided in this embodiment may further include:

[0186] The model training module is used to train the first initial network model using a training dataset to obtain the trained first initial network model. The training dataset includes 3D images of various scenes, and the scene category of each 3D image is known. The first initial network model is compressed and accelerated using model distillation technology to obtain the first network model.

[0187] In some optional embodiments, the second processing module 63 is specifically used to determine a binary validity mask based on an incomplete depth map; input the incomplete depth map, the binary validity mask, and the complete image into a second network model; and the second network model is obtained through training.

[0188] Exemplary electronic devices

[0189] Below, for reference Figure 7 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0190] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0191] like Figure 7 As shown, the electronic device includes one or more processors and memory.

[0192] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0193] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the 3D scene generation methods of the various embodiments of this disclosure described above and / or other desired functions.

[0194] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0195] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0196] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0197] Of course, for the sake of simplicity, Figure 7Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0198] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the 3D scene generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0199] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0200] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the 3D scene generation methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0201] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0202] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0203] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0204] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0205] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0206] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0207] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0208] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating a 3D scene, characterized in that, include: Based on the first global 3D scene map corresponding to the first pose, determine the incomplete image and incomplete depth map corresponding to the second pose; The second pose is determined by processing the first pose; The incomplete image and the received text description information are processed by the first network model to determine the complete image; The incomplete depth map and the complete image are processed by a second network model to determine the complete depth map; The second pose, the complete depth map, and the complete image are processed using a third network model to determine a second global 3D scene map; the second global 3D scene map includes at least the first global 3D scene map. The process of processing the second pose, the complete depth map, and the complete image using a third network model to determine the second global 3D scene map includes: The complete image is feature extracted through the backbone sub-network in the third network model to obtain matching features and image features; Based on the matching features and the second pose, the complete depth map is optimized to determine the optimized depth map; Based on the image features and the optimized depth map, a second local 3D scene map is determined; The second global 3D scene map is determined based on the second local 3D scene map and the first global 3D scene map.

2. The method according to claim 1, characterized in that, The step of determining the incomplete image and incomplete depth map corresponding to the second pose based on the first global 3D scene map corresponding to the first pose includes: Obtain the first global 3D scene map corresponding to the first pose; The first pose is processed based on the received rotation and translation matrix to obtain the second pose; The first global 3D scene map is processed according to the rotation and translation matrix to determine the incomplete image and the incomplete depth map corresponding to the second pose.

3. The method according to claim 2, characterized in that, The step of obtaining the first global 3D scene map corresponding to the first pose includes: The received first pose, preset image, and preset depth map are processed using a third network model to determine the first global 3D scene map; or... The second global 3D scene graph is used as the first global 3D scene graph.

4. The method according to any one of claims 1-3, characterized in that, The step of optimizing the complete depth map based on the matching features and the second pose to determine the optimized depth map includes: Based on the rotation and translation matrix corresponding to the second pose, the historical matching features that are nearest neighbors to the second pose are obtained from the feature memory; the feature memory contains multiple historical matching features corresponding to different poses. The optimized depth map is determined by processing the historical matching features, the matching features, and the complete depth map through the cost volume sub-network in the third network model.

5. The method according to claim 4, characterized in that, The step of processing the historical matching features, the matching features, and the complete depth map through the cost volume subnetwork in the third network model to determine the optimized depth map includes: Multiple depth candidate values ​​are generated based on the complete depth map; Calculate the similarity between the matching feature and the historical matching feature in the depth space corresponding to the plurality of depth candidate values, and determine the probability value of each depth candidate value among the plurality of depth candidate values; The optimized depth map is determined by performing a weighted summation on the plurality of depth candidate values ​​using the probability values ​​as weights.

6. The method according to any one of claims 1-3, characterized in that, The step of determining the second local 3D scene map based on the image features and the optimized depth map includes: The target depth map and Gaussian model are obtained by processing the target depth value and the image features through the segmentation sub-network in the third network model. Based on the target depth map, the Gaussian model is projected into 3D space to obtain the second local 3D scene map.

7. The method according to any one of claims 1-3, characterized in that, The step of determining the second global 3D scene map based on the second local 3D scene map and the first global 3D scene map includes: Project the second local 3D scene map and the first global 3D scene map onto the pixel coordinate system; In the pixel coordinate system, filter out 3D image information in the second local 3D scene map and the first global 3D scene map that does not meet the depth consistency constraint; The second local 3D scene map and the 3D image information that meets the depth consistency constraint after being filtered in the first global 3D scene map are stitched together to obtain the second global 3D scene map.

8. The method according to any one of claims 1-3, characterized in that, Before processing the incomplete image and the received text description information through the first network model to determine the complete image, the process further includes: The first initial network model is trained using a training dataset to obtain the trained first initial network model; the training dataset includes 3D images of various scenes, and each 3D image has a known scene category; The first initial network model is obtained by compressing and accelerating the first network model using model distillation technology.

9. The method according to any one of claims 1-3, characterized in that, The step of processing the incomplete depth map and the complete image using a second network model to determine the complete depth map includes: Based on the incomplete depth map, determine the binary validity mask; The incomplete depth map, the binary validity mask, and the complete image are input into the second network model; the second network model is obtained through training.

10. A 3D scene generation device, characterized in that, include: The preprocessing module is used to determine the incomplete image and incomplete depth map corresponding to the second pose based on the first global 3D scene map corresponding to the first pose. The second pose is determined by processing the first pose; The first processing module is used to process the incomplete image and the received text description information through a first network model to determine the complete image; The second processing module is used to process the incomplete depth map and the complete image through a second network model to determine the complete depth map; The third processing module is used to process the second pose, the complete depth map, and the complete image through a third network model to determine the second global 3D scene map; The second global 3D scene graph includes at least the first global 3D scene graph; The third processing module includes: The feature extraction unit is used to extract features from the complete image through the backbone sub-network in the third network model to obtain matching features and image features. A depth optimization unit is used to optimize the complete depth map based on the matching features and the second pose to determine an optimized depth map; A local scene generation unit is used to determine a second local 3D scene map based on the image features and the optimized depth map; The incremental fusion unit is used to determine the second global 3D scene map based on the second local 3D scene map and the first global 3D scene map.

11. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein when the computer program product is executed, it implements the 3D scene generation method according to any one of claims 1-9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the 3D scene generation method according to any one of claims 1-9.

13. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the 3D scene generation method according to any one of claims 1-9.

Images