A hierarchical three-dimensional scene generation method and system based on spatial super-resolution
By constructing a hierarchical scene tree and a voxel super-resolution model, the problems of geometric consistency and detail generation in the existing 3D scene generation are solved, and efficient and automatic high-quality 3D scene reconstruction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-01-22
- Publication Date
- 2026-06-19
Smart Images
Figure CN121564244B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer graphics technology, specifically relating to a hierarchical 3D scene generation method and system based on spatial super-resolution. Background Technology
[0002] With the rapid development of computer graphics, virtual reality (VR), and augmented reality (AR) technologies, the technology of quickly reconstructing and generating 3D scenes from a single 2D image has demonstrated enormous application potential and market value in fields such as game development, film and television special effects production, industrial digital twins, and metaverse content construction. This technology can significantly reduce the cost of manual modeling and improve the efficiency of content production. However, despite the progress made in existing technologies, significant technical bottlenecks still exist in practical applications.
[0003] Currently, to meet the need for rapid reconstruction of 3D scenes from single 2D images, academia and industry have mainly developed three mainstream technical approaches. The first is retrieval-based methods, whose core logic is "retrieval and assembly." This involves pre-building an offline template library containing a massive number of CAD models, analyzing object features in the input image, searching for the most matching 3D model in the library, and arranging it according to the inferred spatial layout. The second is end-to-end methods, which attempt to establish a direct mapping from the 2D pixel domain to the 3D geometric domain using deep neural networks. This aims to jointly infer the scene's geometry, object layout, and surface texture in one go, achieving automated generation of the entire scene. The third is the more advanced compositional methods. To address the challenge of reconstructing complex scenes, this method employs a "divide and conquer" strategy. It first decomposes the complex 3D scene into multiple independent single-object generation subtasks, uses single-object generation models to generate each component, and finally assembles them back into a coherent 3D layout.
[0004] While the aforementioned existing technologies have achieved 3D scene generation to some extent, significant technical bottlenecks remain when generating high-quality, high-resolution scenes with complex layouts. Retrieval-based methods are severely limited by the size and diversity of pre-built template libraries, resulting in retrieved models that are often only "roughly similar," failing to achieve precise geometric alignment with object shapes in the input image and typically failing to generate realistic textures consistent with the image. End-to-end methods, while capable of generating textures, attempt to generate the entire large scene in a single network inference iteration. Limited by memory capacity and computational complexity, their output spatial resolution is often low, with significant loss of detail. Furthermore, due to the extreme scarcity of high-quality 3D scene training data, these models exhibit poor generalization ability and struggle to handle unseen, personalized objects. Combinatorial methods, while leveraging the advantages of single-object generation, typically only perform "one-time" generation and assembly of each component, lacking a hierarchical detail processing mechanism (i.e., failing to distinguish the detail requirements of foreground core objects from background objects). An even more serious problem is that when attempting to improve details through image super-resolution or other means, the generated fine structure often deviates from the original coarse geometric structure due to the randomness of the generative model. This results in the "refined object" being inconsistent with the "original scene layout." Ultimately, although the resolution of the generated scene is improved, the original correct spatial structure and geometric consistency are destroyed.
[0005] Therefore, a new method is needed that can balance geometric consistency, high-resolution detail generation, and hierarchical scene construction, so as to improve the overall scene detail while ensuring that local objects and the global layout maintain spatial and structural consistency. Summary of the Invention
[0006] In view of the above, the purpose of this invention is to provide a hierarchical 3D scene generation method and system based on spatial super-resolution. Through hierarchical decomposition and a coarse-to-fine generation mechanism, it can significantly reduce the cost of manual modeling while efficiently and automatically reconstructing high-quality 3D scenes with high visual fidelity, fine geometric details and global structural consistency from a single RGB image, so as to meet the urgent needs of high-precision 3D content in fields such as games, film and television, digital twins and metaverse.
[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:
[0008] In a first aspect, the embodiments of the present invention provide a hierarchical 3D scene generation method based on spatial super-resolution, comprising the following steps:
[0009] Target detection, segmentation, and enhancement processing are performed on scene images to obtain high-resolution object images;
[0010] An initial coarse scene voxel is generated from the scene image, and coarse voxels and structural latent variables aligned with the high-resolution object image are selected to construct a hierarchical scene tree.
[0011] High-resolution object images and coarse voxels are input into a voxel super-resolution model to generate fine voxels that maintain geometric consistency with the coarse voxels.
[0012] Based on the decoding of coarse voxels and fine voxels into 3D Gaussian sputtering representations, scale alignment and pose registration are performed to achieve geometric registration of fine voxels in the original scene.
[0013] Based on a hierarchical scene tree, the fine voxels of the sub-components are generated recursively with the coarse voxels of the current node as conditions, and finally the fine voxels of all the sub-components are assembled to generate a high-resolution 3D scene.
[0014] Preferably, the step of performing target detection, segmentation, and enhancement processing on the scene image to obtain a high-resolution object image includes:
[0015] The object detection algorithm is used to obtain the two-dimensional bounding boxes of various objects in the scene image, and the object segmentation algorithm is used to generate pixel-level object masks based on the two-dimensional bounding boxes of various objects.
[0016] The target object image is cropped from the scene image based on the object mask, and the target object image is enhanced by super-resolution reconstruction to obtain a high-resolution object image.
[0017] Preferably, the step of generating initial coarse scene voxels from the scene image and filtering out coarse voxels and structural latent variables aligned with the high-resolution object image to construct a hierarchical scene tree includes:
[0018] Using a basic 3D generative model, an initial coarse scene voxel containing a coherent spatial structure is generated from the scene image.
[0019] Based on the cross-attention correspondence between the features of the two-dimensional scene image and the voxels of the initial three-dimensional rough scene in the basic three-dimensional generative model, rough voxels aligned with the high-resolution object image and their additional structural latent variables are selected from the voxels of the initial rough scene.
[0020] Using the initial coarse scene voxel as the root node and the selected coarse voxels, i.e., the sub-components, as the child nodes, a hierarchical scene tree containing scene object representations of different granularities is constructed.
[0021] Preferably, the step of inputting a high-resolution object image and coarse voxels into a voxel super-resolution model to generate fine voxels that maintain geometric consistency with the coarse voxels includes:
[0022] In the voxel super-resolution model, the coarse voxels are encoded into voxel latent variables by the VAE encoder, and the voxel latent variables are transformed by the linear projection network to make them consistent with the feature dimensions of the image conditions, thus obtaining geometric features. The image features of the high-resolution object image are then extracted by the visual feature encoder.
[0023] Geometric features and image features are respectively input into the first cross-attention layer and the second cross-attention layer for feature fusion, and then diffused generation is performed in the stream transformer based on the fused features to obtain fine voxels that maintain geometric consistency with the coarse voxels.
[0024] Preferably, the step of decoding coarse voxels and fine voxels into a 3D Gaussian sputtering representation, and then performing scale alignment and pose registration accordingly, to achieve geometric registration of fine voxels in the original scene, includes:
[0025] The structural latent variables added to rough voxels and the structural latent variables added to fine voxels are decoded into the first three-dimensional Gaussian sputtering representation and the second three-dimensional Gaussian sputtering representation, respectively.
[0026] The average distance from the uniform sampling point on the surface of the second 3D Gaussian sputtering representation to its center is calculated as a scale proxy, and scaled with the corresponding scale proxy of the first 3D Gaussian sputtering representation.
[0027] The scale-aligned second 3D Gaussian sputtering representation is matched with the first 3D Gaussian sputtering representation to estimate the rigid transformation matrix, which is used to register the fine voxels back to the original scene.
[0028] Preferably, the step of recursively generating fine voxels of sub-components based on a hierarchical scene tree, using the coarse voxel of the current node as a condition, and finally assembling the fine voxels of all sub-components to generate a high-resolution 3D scene, includes:
[0029] Based on a hierarchical scene tree, a top-down recursive strategy is adopted starting from the root node. At each level, the coarse voxel corresponding to the current node is used as the geometric condition to drive the voxel super-resolution model to generate the fine voxels of its next level child nodes.
[0030] Perform geometric registration on the fine voxels of the generated child nodes, iterate this process until the preset level of detail is reached, and finally assemble all the registered fine voxel components under the spatial layout defined by the initial coarse scene voxels to obtain a high-resolution 3D scene.
[0031] Preferably, when generating fine voxels of child nodes, the Amodal3R framework is integrated to handle occlusion relationships between child node components and generate complete nonmodal 3D reconstruction results.
[0032] Secondly, embodiments of the present invention also provide a hierarchical 3D scene generation system based on spatial super-resolution, which is implemented using the above-mentioned hierarchical 3D scene generation method based on spatial super-resolution, including: an image preprocessing module, a scene tree construction module, a voxel super-resolution module, an instance registration module, and a scene generation module;
[0033] The image preprocessing module is used to perform target detection, segmentation and enhancement processing on the scene image to obtain a high-resolution object image;
[0034] The scene tree construction module is used to generate initial coarse scene voxels from the scene image, and filter out coarse voxels and structural latent variables that are aligned with the high-resolution object image to construct a hierarchical scene tree.
[0035] The voxel super-resolution module is used to input high-resolution object images and coarse voxels into the voxel super-resolution model, and use coarse voxels as geometric conditions to guide the diffusion process to generate fine voxels.
[0036] The instance registration module is used to decode coarse voxels and fine voxels into three-dimensional Gaussian sputtering representations, respectively, and perform scale alignment and pose registration accordingly to achieve geometric registration of fine voxels in the original scene.
[0037] The scene generation module is used to recursively generate fine voxels of sub-components based on a hierarchical scene tree, using the coarse voxel of the current node as a condition, and finally assembles the fine voxels of all sub-components to generate a high-resolution 3D scene.
[0038] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described hierarchical three-dimensional scene generation method based on spatial super-resolution when executing the computer program.
[0039] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described hierarchical 3D scene generation method based on spatial super-resolution.
[0040] Compared with the prior art, the beneficial effects of the present invention include at least the following:
[0041] (1) By constructing a hierarchical scene tree, the present invention decomposes the complex scene into components from coarse to fine, and adopts a top-down recursive generation strategy, which naturally takes into account both the correctness of the macro layout and the fineness of the micro objects, overcomes the limitations of the end-to-end method in terms of insufficient resolution and loss of details when generating the whole, and achieves the unity of global structure and local details.
[0042] (2) This invention constructs a voxel super-resolution model and innovatively uses coarse voxels as geometric conditions to guide the voxel super-resolution generation process. By constraining the randomness of the diffusion model through the cross attention mechanism, it ensures that the refined components are precisely aligned with the original scene structure. This solves the core problem of detail enhancement destroying overall consistency in the combinatorial method and ensures strict geometric consistency while improving details.
[0043] (3) Based on the robust alignment and registration of three-dimensional Gaussian representation, and combined with non-modal reconstruction technology, this invention realizes an efficient and automated high-fidelity generation process of components from generation, registration to assembly, which significantly reduces the dependence on pre-made model library and manual modeling, and provides a reliable solution for the large-scale production of high-quality three-dimensional scenes. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 This is a flowchart illustrating the hierarchical 3D scene generation method based on spatial super-resolution provided in an embodiment of the present invention.
[0046] Figure 2 This is a schematic diagram of the framework of the hierarchical 3D scene generation method based on spatial super-resolution provided in the embodiments of the present invention;
[0047] Figure 3 This is a schematic diagram of the architecture of the voxel super-resolution model provided in an embodiment of the present invention;
[0048] Figure 4 This is a schematic diagram of the three-dimensional scene generation result provided in an embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram of the structure of the hierarchical 3D scene generation system based on spatial super-resolution provided in an embodiment of the present invention. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0051] The inventive concept of this invention is as follows: Addressing the problem that existing 3D scene generation methods struggle to balance global structural consistency with local geometric details, and are prone to disrupting the original spatial layout during detail enhancement, this invention provides a hierarchical 3D scene generation method and system based on spatial super-resolution. By constructing a progressive generation architecture from coarse to fine, it avoids aiming for a perfect scene generation in one go. Instead, it first uses a basic model to quickly generate an initial scene with globally consistent geometry and layout but coarse details, serving as a "foundation." Subsequently, it gradually refines the scene through a layered mechanism. Furthermore, it innovatively employs a 2D-to-3D mapping strategy to construct a hierarchical scene tree containing parent-child node relationships, achieving precise management of scene components from the overall to the local. Crucially, by constructing a voxel super-resolution model with coarse voxels and high-resolution object images as dual input conditions, it not only improves the resolution and geometric details of voxels during generation but also enforces strong consistency in features between newly generated fine voxels and the original coarse voxels through a constraint mechanism. Finally, the enhanced components are re-registered and assembled, thereby synthesizing a high-quality 3D scene rich in local details while ensuring global layout accuracy.
[0052] like Figure 1 and Figure 2 As shown, the embodiment provides a hierarchical 3D scene generation method based on spatial super-resolution, which can reconstruct a high-quality 3D scene from a single RGB image, including the following steps:
[0053] S1 performs target detection, segmentation, and enhancement processing on the scene image to obtain a high-resolution object image.
[0054] In this embodiment, the target is first analyzed and extracted in high quality from the RGB image of the three-dimensional scene.
[0055] Specifically, the process begins by using a scene image as input and employing an object detection algorithm to identify the 2D bounding boxes of various objects within the scene image. These bounding boxes are then used as key spatial location cues in the object segmentation algorithm, guiding the model to generate pixel-level object masks. Based on these masks, RGB image regions containing only the target objects are cropped from the original scene. Finally, the segmented and extracted target object image is centered and enlarged, and the HYPIR (High-Performance Image Restoration) super-resolution algorithm is introduced to perform super-resolution reconstruction of the target object image, resulting in a high-resolution object image that provides high-quality input for subsequent processing.
[0056] S2 generates initial coarse scene voxels from the scene image and filters out coarse voxels and structural latent variables that are aligned with the high-resolution object image to construct a hierarchical scene tree.
[0057] In this embodiment, the hierarchical structure of the scene image is decomposed to construct a scene tree, so as to achieve fine-grained processing of local objects while maintaining the consistency of the global spatial layout. Each node of the scene tree represents a scene component containing voxel representation and cropped image, and as the node depth increases, the scene component focuses on a smaller local area and presents more refined geometric shapes and texture features.
[0058] Specifically, firstly, using a scene image as input, an initial coarse scene voxels (represented as voxels with structural latent variables) containing coherent spatial structure are generated based on the scene image using a basic 3D generative model (TRELLIS model). Subsequently, based on the cross-attention correspondence between the 2D scene image features and the 3D initial coarse scene voxels in the Flow Transformer of the basic 3D generative model, the complex 3D voxel segmentation task is transformed into a more manageable 2D object segmentation task. Specifically, based on the masked regions in the 2D scene image, the corresponding row indices in the cross-attention matrix are determined; the cross-attention matrix is subjected to Softmax normalization, and the attention scores of the corresponding rows are accumulated along the column direction; the sum of the accumulated attention scores is compared with a preset threshold (set to 0.3 in this embodiment), and column indices with scores greater than the preset threshold are selected as the target voxel indices. This process filters out coarse voxels aligned with the 2D high-resolution object image and their additional structural latent variables from the initial coarse scene voxels. Finally, the structural latent variables attached to these rough voxels are decoded into 3D Gaussian sputtering (3DGS) representations, thereby constructing a hierarchical scene tree containing representations of scene objects at different granularities, with the initial rough scene voxels as the root node and the selected rough voxels, i.e., sub-components, as the child nodes, thus laying a structured foundation for subsequent voxel super-resolution enhancement.
[0059] S3 takes a high-resolution object image and coarse voxels as input to a voxel super-resolution model and generates fine voxels that maintain geometric consistency with the coarse voxels.
[0060] In this embodiment, in order to solve the problem that the reconstructed object structure deviates from the original coarse structure due to the randomness and uncontrollability of the generation model in traditional image enhancement strategies, after obtaining the hierarchical scene components and their corresponding images by using the 2D to 3D scene tree construction method in step S2, the designed voxel super-resolution model is used to refine the target instance while maintaining the geometric consistency of cross-level components between it and the coarse voxel.
[0061] Specifically, such as Figure 3As shown, in the voxel super-resolution model, coarse voxels are encoded into voxel latent variables through a sparse structure VAE (Variational Auto-Encoder). A trainable linear projection network then transforms the dimensions of these latent variables to match the feature dimensions of the image conditions, yielding geometric features. These geometric features are then extracted using a visual feature encoder (DINO-V2 model) to extract image features from the high-resolution object image. Subsequently, the geometric and image features are input into trainable first and second cross-attention layers for feature fusion. Finally, a diffusion-based stream transformer generates voxels based on the fused features. During training, the original generation model parameters are frozen, and only the newly added linear projection network and the first cross-attention layer are optimized. This results in the generation of fine voxels that possess both high resolution and rich geometric details while maintaining geometric consistency with the coarse voxels.
[0062] S4 decodes coarse and fine voxels into 3D Gaussian sputtering representations, and performs scale alignment and pose registration accordingly to achieve geometric registration of fine voxels in the original scene.
[0063] In this embodiment, in order to address the scale and pose deviations between high-resolution fine objects and initial coarse objects caused by the randomness of the generated model, this step uses coarse voxels, fine voxels, and additional structural latent variables as inputs to perform scale alignment and pose registration, so as to ensure the spatial consistency of the final scene.
[0064] Specifically, firstly, the structural latent variables attached to coarse voxels and fine voxels are decoded into first and second 3D Gaussian sputtering representations, respectively. Then, for scale alignment, the average distance from the uniformly sampled points on the surface of the second 3D Gaussian sputtering representation to its center is calculated as a scale proxy, and this is scaled with the corresponding scale proxy of the first 3D Gaussian sputtering representation. Finally, for pose registration, addressing the significant imbalance in the number of Gaussian points and the large number of mismatched points between fine and coarse objects, the ICP (Iterative Closest Point) method, which relies on precise nearest neighbor matching, is abandoned. Instead, the robustness of the RANSAC (Random Sample Consensus) algorithm is utilized to match the scale-aligned second 3D Gaussian sputtering representation with the first 3D Gaussian sputtering representation to estimate the rigid transformation matrix (including rotation and translation). Based on the rigid transformation matrix, the fine voxels are registered back to the original scene.
[0065] S5, based on a hierarchical scene tree, recursively generates fine voxels of sub-components using the coarse voxel of the current node as a condition, and finally assembles the fine voxels of all sub-components to generate a high-resolution 3D scene.
[0066] In this embodiment, a hierarchical scene generation process is performed from coarse to fine based on a hierarchical scene tree and a voxel super-resolution model. The target components are gradually refined and the refined components are reassembled back into the scene to generate a high-resolution and high-quality 3D scene.
[0067] Specifically, a top-down strategy is adopted, performing a recursive generation process from coarse to fine based on the constructed hierarchical scene tree: Using the initial coarse scene voxels as input, the root node representing the overall initial scene is first divided into several sub-component nodes layer by layer according to the scene's relationships. At each level, the coarse voxel corresponding to the current node is used as the geometric condition to drive the voxel super-resolution model to generate higher-resolution fine voxels for the next level of sub-nodes to represent fine-grained sub-objects. Furthermore, to address potential occlusion issues in the segmented sub-objects, the Amodal3R framework is integrated to generate complete non-modal reconstruction results. This process continues until all objects in the scene reach the expected level of detail. Geometric registration is performed on the generated fine voxels of the sub-nodes, iterating this process until all objects in the scene reach the expected level of detail. Finally, under the spatial layout defined by the initial coarse scene voxels, all registered fine voxel components are assembled to obtain a high-resolution 3D scene, as shown in the image. Figure 4 As shown.
[0068] Based on the same inventive concept, such as Figure 5 As shown, this embodiment of the invention also provides a hierarchical 3D scene generation system 500 based on spatial super-resolution, including: an image preprocessing module, a scene tree construction module, a voxel super-resolution module, an instance registration module, and a scene generation module;
[0069] The image preprocessing module is used to perform target detection, segmentation and enhancement processing on the scene image to obtain a high-resolution object image;
[0070] The scene tree construction module is used to generate initial coarse scene voxels from the scene image, and filter out coarse voxels and structural latent variables that are aligned with the high-resolution object image to construct a hierarchical scene tree.
[0071] The voxel super-resolution module is used to input high-resolution object images and coarse voxels into the voxel super-resolution model, and use coarse voxels as geometric conditions to guide the diffusion process to generate fine voxels.
[0072] The instance registration module is used to decode coarse voxels and fine voxels into three-dimensional Gaussian sputtering representations, respectively, and perform scale alignment and pose registration accordingly to achieve geometric registration of fine voxels in the original scene.
[0073] The scene generation module is used to recursively generate fine voxels of sub-components based on a hierarchical scene tree, using the coarse voxel of the current node as a condition, and finally assembles the fine voxels of all sub-components to generate a high-resolution 3D scene.
[0074] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described hierarchical three-dimensional scene generation method based on spatial super-resolution when executing the computer program.
[0075] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described hierarchical 3D scene generation method based on spatial super-resolution.
[0076] It should be noted that the hierarchical 3D scene generation system, electronic device, and computer-readable storage medium based on spatial super-resolution provided in the above embodiments all belong to the same inventive concept as the hierarchical 3D scene generation method based on spatial super-resolution. For details of their specific implementation process, please refer to the embodiments of the hierarchical 3D scene generation method based on spatial super-resolution, which will not be repeated here.
[0077] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A hierarchical 3D scene generation method based on spatial super-resolution, characterized in that, Includes the following steps: Target detection, segmentation, and enhancement processing are performed on scene images to obtain high-resolution object images; An initial coarse scene voxel is generated from the scene image, and coarse voxels and structural latent variables aligned with the high-resolution object image are selected to construct a hierarchical scene tree. High-resolution object images and coarse voxels are input into a voxel super-resolution model to generate fine voxels that maintain geometric consistency with the coarse voxels. Based on the decoding of coarse voxels and fine voxels into 3D Gaussian sputtering representations, scale alignment and pose registration are performed to achieve geometric registration of fine voxels in the original scene. This includes: decoding the structural latent variables attached to coarse voxels and fine voxels into a first 3D Gaussian sputtering representation and a second 3D Gaussian sputtering representation, respectively; calculating the average distance from the uniform sampling point on the surface of the second 3D Gaussian sputtering representation to its center as a scale proxy, and performing scale alignment with the corresponding scale proxy of the first 3D Gaussian sputtering representation; matching the scale-aligned second 3D Gaussian sputtering representation with the first 3D Gaussian sputtering representation to estimate the rigid transformation matrix, which is used to register the fine voxels back to the original scene. Based on a hierarchical scene tree, the fine voxels of the sub-components are generated recursively with the coarse voxels of the current node as conditions, and finally the fine voxels of all the sub-components are assembled to generate a high-resolution 3D scene.
2. The hierarchical 3D scene generation method based on spatial super-resolution according to claim 1, characterized in that, The process of performing target detection, segmentation, and enhancement on the scene image to obtain a high-resolution object image includes: The object detection algorithm is used to obtain the two-dimensional bounding boxes of various objects in the scene image, and the object segmentation algorithm is used to generate pixel-level object masks based on the two-dimensional bounding boxes of various objects. The target object image is cropped from the scene image based on the object mask, and the target object image is enhanced by super-resolution reconstruction to obtain a high-resolution object image.
3. The hierarchical 3D scene generation method based on spatial super-resolution according to claim 1, characterized in that, The process of generating initial coarse scene voxels from the scene image and selecting coarse voxels and structural latent variables aligned with the high-resolution object image to construct a hierarchical scene tree includes: Using a basic 3D generative model, an initial coarse scene voxel containing a coherent spatial structure is generated from the scene image. Based on the cross-attention correspondence between the features of the two-dimensional scene image and the voxels of the initial three-dimensional rough scene in the basic three-dimensional generative model, rough voxels aligned with the high-resolution object image and their additional structural latent variables are selected from the voxels of the initial rough scene. Using the initial coarse scene voxel as the root node and the selected coarse voxels, i.e., the sub-components, as the child nodes, a hierarchical scene tree containing scene object representations of different granularities is constructed.
4. The hierarchical 3D scene generation method based on spatial super-division of claim 1, wherein, The step of inputting a high-resolution object image and coarse voxels into a voxel super-resolution model to generate fine voxels that maintain geometric consistency with the coarse voxels includes: In the voxel super-resolution model, the coarse voxels are encoded into voxel latent variables by the VAE encoder, and the voxel latent variables are transformed by the linear projection network to make them consistent with the feature dimensions of the image conditions, thus obtaining geometric features. The image features of the high-resolution object image are then extracted by the visual feature encoder. Geometric features and image features are respectively input into the first cross-attention layer and the second cross-attention layer for feature fusion, and then diffused generation is performed in the stream transformer based on the fused features to obtain fine voxels that maintain geometric consistency with the coarse voxels.
5. The hierarchical 3D scene generation method based on spatial super-division of claim 1, wherein, The process, based on a hierarchical scene tree, recursively generates fine voxels for sub-components using the coarse voxel of the current node as a condition, and finally assembles the fine voxels of all sub-components to generate a high-resolution 3D scene, including: Based on a hierarchical scene tree, a top-down recursive strategy is adopted starting from the root node. At each level, the coarse voxel corresponding to the current node is used as the geometric condition to drive the voxel super-resolution model to generate the fine voxels of its next level child nodes. Perform geometric registration on the fine voxels of the generated child nodes, iterate this process until the preset level of detail is reached, and finally assemble all the registered fine voxel components under the spatial layout defined by the initial coarse scene voxels to obtain a high-resolution 3D scene.
6. The hierarchical 3D scene generation method based on spatial super-division according to claim 1 or 5, characterized in that, When generating fine voxels of child nodes, the Amodal3R framework is integrated to handle occlusion relationships between child node components and generate complete nonmodal 3D reconstruction results.
7. A hierarchical 3D scene generation system based on spatial super-resolution, implemented using the hierarchical 3D scene generation method based on spatial super-resolution as described in any one of claims 1 to 6, characterized in that, include: Image preprocessing module, scene tree construction module, voxel super-resolution module, instance registration module, and scene generation module; The image preprocessing module is used to perform target detection, segmentation and enhancement processing on the scene image to obtain a high-resolution object image; The scene tree construction module is used to generate initial coarse scene voxels from the scene image, and filter out coarse voxels and structural latent variables that are aligned with the high-resolution object image to construct a hierarchical scene tree. The voxel super-resolution module is used to input high-resolution object images and coarse voxels into the voxel super-resolution model, and use coarse voxels as geometric conditions to guide the diffusion process to generate fine voxels. The instance registration module is used to decode coarse voxels and fine voxels into three-dimensional Gaussian sputtering representations, respectively, and perform scale alignment and pose registration accordingly to achieve geometric registration of fine voxels in the original scene. The scene generation module is used to recursively generate fine voxels of sub-components based on a hierarchical scene tree, using the coarse voxel of the current node as a condition, and finally assembles the fine voxels of all sub-components to generate a high-resolution 3D scene.
8. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the hierarchical three-dimensional scene generation method based on spatial super-resolution as described in any one of claims 1 to 6 when executing a computer program.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a computer, it implements the hierarchical three-dimensional scene generation method based on spatial super-resolution as described in any one of claims 1 to 6.