3D model generation methods, apparatus, electronic devices, computer programs, and storage media

By using multi-scale voxel partitioning and contextual completion prompts, combined with visual language model to optimize pose, the problem of 3D reconstruction of occluded areas is solved, achieving efficient and accurate 3D model generation, and avoiding the computational complexity and structural distortion of traditional methods.

CN122676118BActive Publication Date: 2026-11-17SHENZHEN SMARTCITY TECH DEV GRP CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611143277.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-07-30
Publication Date
2026-11-17
Estimated Expiration
2046-07-30

AI Technical Summary

Technical Problem

Existing 3D reconstruction methods are ineffective when dealing with occluded areas, failing to fully complete the geometric information of the occluded areas. They also have high computational complexity, large memory consumption, and cannot distinguish the importance differences between the foreground and background.

Method used

By using multi-scale voxel partitioning and adaptively allocating voxel resolution, combined with contextual completion cues generated by occlusion states, a pre-trained generative model is driven to generate 3D models. A visual language model is used to quantify spatial relationships and optimize poses to ensure physical consistency.

Benefits of technology

It achieves efficient and accurate 3D reconstruction, balancing reconstruction quality and efficiency, avoiding structural distortions in the completion results, and generating models with visual coherence and physical rationality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122676118B_ABST
    Figure CN122676118B_ABST
Patent Text Reader

Abstract

This application discloses a method, apparatus, electronic device, computer program, and storage medium for generating a 3D model. The method involves acquiring an original image, performing target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; determining the occlusion relationship between each target object and generating a depth map; determining the range of the global 3D physical space based on the depth map; dividing the global 3D physical space into multi-scale voxels according to the image feature information to construct a multi-scale voxel mesh containing spatial occupancy and occlusion states; extracting the state information of the multi-scale voxel mesh as a geometric prior, and combining it with contextual completion prompts generated based on the occlusion states to drive a pre-trained generative model to generate a 3D model. This application can efficiently reconstruct a structurally complete and spatially reasonable 3D model from single or multiple 2D images, solving the problems of geometric missingness and physical inconsistency in occluded areas.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data security, specifically to a method, apparatus, electronic device, computer program, and storage medium for generating three-dimensional models. Background Technology

[0002] Reconstructing 3D models from 2D images is a core problem in computer vision, with wide applications in augmented reality, robot navigation, autonomous driving, digital twins, and film and television production. However, high-quality 3D reconstruction from single or multiple RGB images still faces many challenges.

[0003] Existing 3D reconstruction methods can be mainly classified into the following categories: The first category is based on multi-view stereo vision methods, such as COLMAP and MVSNet. These methods require multiple images from different angles as input, and recover the 3D structure through feature matching and triangulation. Their drawback is that the reconstruction quality drops sharply when the number of input images is limited or the viewpoint variations are insufficient, and they cannot effectively fill in occluded areas.

[0004] The second category comprises monocular depth estimation and voxel reconstruction methods, such as NeRF and its variants, DeepSDF, etc. These methods use neural networks to directly predict depth or implicit representations from a single image. However, existing methods typically discretize the entire scene using a fixed-resolution voxel grid, resulting in high computational complexity, large memory consumption, and an inability to distinguish the importance differences between foreground and background. Furthermore, when severe occlusion exists in the scene, the geometric information of the occluded area is completely lost, and existing methods can only perform coarse filling based on data-driven priors, easily leading to structural distortions or spatial conflicts with adjacent objects.

[0005] In summary, how to efficiently and accurately reconstruct 3D scenes with occlusion, and ensure the rationality of the reconstructed model in terms of geometric structure and physical spatial relationships, is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, electronic device, computer program, and storage medium for generating three-dimensional models, which can efficiently and accurately reconstruct three-dimensional scenes with occlusion.

[0007] This application provides a method for generating a three-dimensional model, including: Acquire the original image, and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; Determine the occlusion relationship between each target object and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map; Based on the image feature information, the global three-dimensional physical space is divided into multi-scale voxels to construct a multi-scale voxel mesh that includes space occupancy status and occlusion status. The state information of the multi-scale voxel mesh is extracted as a geometric prior, and combined with contextual completion prompts generated based on the occlusion state, the pre-trained generative model is driven to generate a 3D model.

[0008] This application embodiment also provides a three-dimensional model generation apparatus, including: The segmentation unit is used to acquire the original image and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; The determination unit is used to determine the occlusion relationship between each target object and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map. The building unit is used to divide the global three-dimensional physical space into multi-scale voxels based on image feature information, and to build a multi-scale voxel mesh that includes spatial occupancy and occlusion states. The generation unit is used to extract the state information of multi-scale voxel mesh as geometric priors and combine it with context completion prompts generated based on occlusion state to drive the pre-trained generative model to generate a 3D model.

[0009] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads the instructions from the memory to execute the steps in any of the three-dimensional model generation methods provided in this application.

[0010] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the three-dimensional model generation methods provided in this application.

[0011] The 3D model generation method, apparatus, electronic device, computer program, and storage medium provided in this application, through multi-scale voxel partitioning, can adaptively allocate voxel resolution according to the importance of the target, resulting in rich foreground details and low background computational overhead, thus balancing reconstruction quality and efficiency. By extracting geometric and texture features around the occluded area to generate structured contextual completion prompts, the generation model is guided to perform occlusion completion with visual coherence and physical rationality, avoiding the problem of structural distortion in the completion results of traditional methods. By using a visual language model, abstract spatial relationships (support, contact, stacking) are quantified into voxel-level constraint rules and computable cost functions, and gradient descent is used to optimize the target pose, so that the correction of physical consistency has clear and repeatable judgment criteria. The entire process starts from a single RGB image, without the need for multi-view input or complex sensor equipment, which facilitates cost control. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the three-dimensional model generation method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a three-dimensional model generation method provided in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of the structure of the three-dimensional model generation device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] This application provides a method, apparatus, electronic device, computer program, and storage medium for generating three-dimensional models.

[0016] Specifically, the 3D model generation device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0017] In some embodiments, the three-dimensional model generation device can also be integrated into multiple electronic devices. For example, the three-dimensional model generation device can be integrated into multiple servers, and the three-dimensional model generation method of this application can be implemented by multiple servers.

[0018] In some embodiments, the terminal may also be used as a server to perform some or all of the functions of a server.

[0019] For example, refer to Figure 1The electronic device can be a terminal that can acquire the original image and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; determine the occlusion relationship between each target object and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map; divide the global three-dimensional physical space into multi-scale voxels according to the image feature information, and construct a multi-scale voxel mesh that includes the space occupancy state and occlusion state; extract the state information of the multi-scale voxel mesh as a geometric prior, and combine it with the context completion prompts generated based on the occlusion state to drive the pre-trained generative model to generate a three-dimensional model.

[0020] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0021] In this embodiment, a method for generating a three-dimensional model is provided, such as... Figure 1 As shown, the specific process of this 3D model generation method can be as follows: Step 110: Obtain the original image and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information.

[0022] The original image refers to one or more two-dimensional images input by the user for 3D reconstruction. In a typical application scenario of this application, the original image is usually a single RGB image, but the technical solution of this application is also applicable to the case of multiple image inputs, such as a sequence of images of the same scene taken from different angles.

[0023] The purpose of object recognition and segmentation is to separate individual object instances from the original image and extract the feature information of each instance. Image feature information includes, but is not limited to, the contour information of the target object (e.g., the set of boundary pixels of the segmentation mask), texture information (e.g., local texture histograms or Gabor features), color histograms, and semantic labels (e.g., category labels such as "table", "cup", "book").

[0024] In some implementations, pre-trained visual segmentation models can be used to process the original image. For example, Grounded-SAM (combining the object detection capabilities of Grounding DINO and the segmentation capabilities of SAM) or instance segmentation models such as Mask R-CNN can be used. Specifically, the model first identifies different instances in the image, outputting the bounding box coordinates [x_min, y_min, x_max, y_max] and confidence score for each instance, then segments each instance, generating a binary segmentation mask (background pixels are 0, target pixels are 1). For scenarios requiring open vocabulary recognition, open vocabulary detection models such as YOLO-World can be used, allowing users to specify the target category to be segmented via text prompts.

[0025] To facilitate subsequent determination of occlusion relationships, an "enhanced segmentation mask" can be further generated. The enhanced segmentation mask adds a third state—the occluded state—to the ordinary binary mask. That is, the mask value for each pixel can be 0 (background), 1 (target visible), or 2 (target occluded). The construction of this enhanced mask requires incorporating depth information, which will be explained in detail in step 120.

[0026] Step 120: Determine the occlusion relationship between each target object and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map.

[0027] A depth map is an image containing information about the distances of objects in a scene from the observer. The depth value of each pixel represents the distance from the corresponding 3D point to the camera's optical center. The global 3D physical space refers to the maximum bounding box of the reconstructed scene in the 3D world coordinate system, defined by the minimum and maximum values ​​on the X, Y, and Z coordinate axes.

[0028] In this embodiment, a monocular depth estimation model can be used to infer depth from the original image and generate a depth map. The monocular depth estimation model can be MiDaS, DPT (Dense Prediction Transformer), or ZoeDepth, among others. These models, trained on large-scale depth datasets, are capable of predicting relative or absolute depth from a single RGB image.

[0029] To obtain more accurate absolute depth, the MoGe (Monocular Geometry) model can be prioritized. The MoGe model can output pixel-level absolute depth values ​​and has good generalization ability for both indoor and outdoor scenes. After the depth map is generated, it can be combined with the camera's intrinsic parameter matrix (including focal lengths f_x, f_y and optical center coordinates c_x, c_y) to backproject each pixel (u, v) and its depth value z into 3D space, obtaining the 3D point coordinates (x, y, z). The backprojection formula is: z = depth(u, v); x = (u - c_x) × z / f_x; y = (v - c_y) × z / f_y; By traversing all pixels, a dense 3D point cloud of the scene can be obtained. Based on the extreme points of the point cloud on the X, Y, and Z axes, a minimum cubic bounding box that can contain all target objects is defined. To allow for subsequent voxel subdivision, the bounding box can typically be extended outward by 5% to 10%. This bounding box serves as the extent of the global 3D physical space.

[0030] After obtaining the depth map, occlusion relationships can be further determined. Specifically, for each target segmentation mask obtained in step 110, check if there are overlapping regions between the masks of different targets. If overlap exists, compare the depth values ​​of each pixel in the overlapping region. Targets with smaller depth values ​​(closer to the camera) are identified as occluders, and targets with larger depth values ​​(farther from the camera) are identified as the occluded targets. The mask values ​​of the overlapping regions of the occluded targets are marked as 2, thereby generating an enhanced segmentation mask.

[0031] Step 130: Divide the global three-dimensional physical space into multiple scales based on the image feature information, and construct a multiple scale voxel mesh that includes the space occupancy state and the occlusion state.

[0032] In this context, a voxel is the smallest discrete unit in three-dimensional space, similar to a pixel in a two-dimensional image. Voxel partitioning refers to dividing a continuous three-dimensional space into a regularly arranged grid of small cubes. Multi-scale voxel partitioning refers to discretizing different regions using voxels of different sizes (i.e., different resolutions) based on spatial location or semantic importance. Spatial occupancy status is used to indicate whether each voxel is occupied by geometry, typically including three states: "empty" (no object), "occupied" (object present), and "occluded" (object present but blocked by other objects, information missing).

[0033] In this embodiment, the specific implementation of multi-scale voxel partitioning includes: First, the semantic labels and spatial location information extracted in step 110 are analyzed to classify the target object into foreground objects and background objects. The classification criteria may include: the target's area proportion in the image (e.g., ≥5%), whether its semantic category belongs to a predefined set of core objects (e.g., furniture, props, people, etc.), and whether the target is located in the center of the image. Typically, objects closer to the camera and relevant to the user's query intent are classified as foreground; while objects that serve as part of the environment, such as walls, ground, and sky, are classified as background.

[0034] Secondly, allocate a higher voxel resolution (i.e., a smaller voxel edge length), such as 1cm to 3cm, to the physical space region where the foreground target is located, to preserve fine geometric details; allocate a lower voxel resolution (i.e., a larger voxel edge length), such as 5cm to 20cm, to the physical space region where the background environment target is located. It should be noted that the voxel meshes between different regions need to be aligned at the boundaries to ensure that the voxel mesh of the entire scene is spatially continuous. The alignment method can be: using the smallest voxel edge length as a reference, match larger voxel regions to integer multiples of the smallest voxel edge length by rounding up.

[0035] After dividing the resolution, a multi-scale voxel mesh can be constructed. The specific steps include: calculating the number of voxels in each dimension based on the global 3D physical space and the resolution of each region. For example, for the X-axis direction, the number of voxels W = ceil((x_max - x_min) / voxel_size_x), where voxel_size_x is the voxel side length in the X-axis direction (since voxels are usually cubes, voxel_size_x = voxel_size_y = voxel_size_z). Then, each voxel is traversed, and the corresponding pixel region is determined through coordinate transformation. The mask state (0 / 1 / 2) and depth value within that region are assigned to the voxel. For cases where multiple pixels map to the same voxel, a majority voting method (taking the mask state with the highest frequency) or a nearest neighbor method (taking the pixel with the depth value closest to the voxel center) can be used to fill the voxel's state information. Finally, a 3D voxel array is obtained, where each voxel contains attributes such as "occupied," "occluded," and "depth value range."

[0036] Step 140: Extract the state information of the multi-scale voxel mesh as a geometric prior, and combine it with the context completion prompts generated based on the occlusion state to drive the pre-trained generative model to generate a 3D model.

[0037] Geometric prior refers to prior knowledge about shape and position extracted from a voxel mesh. In this application, the geometric prior is specifically represented as a tensor with the same size as the voxel mesh, where each voxel position is represented by a numerical value to indicate its occupancy status (e.g., 0 indicates empty, 1 indicates occupied) or encoded as a richer feature vector.

[0038] Contextual completion cues are structured instructions used to guide generative models in repairing occluded parts. Pre-trained generative models can employ Latent Diffusion Models (LDMs), Variational Autoencoders (VAEs), or 3D-aware Generative Adversarial Networks (3D-GANs). In this embodiment, multi-scale voxel meshes are encoded as conditional inputs to the generative model. Simultaneously, specific prompts are generated for the voxel regions marked as "occluded" in step 130, guiding the generative model to infer and complete the geometry of the occluded portions based on surrounding visible geometric features (such as edge continuity and texture consistency).

[0039] Prior to step 140, the step of obtaining context completion hints may include: locating the voxel region in the multi-scale voxel mesh that is in an occluded state, and extracting the geometric and texture features of the visible voxels around the voxel region; generating a structured guidance instruction containing structural continuity and surface continuity constraints based on the geometric and texture features, and generating context completion hints.

[0040] Specifically, the voxel mesh constructed in step 130 can be scanned first to find all voxel locations marked as "occluded". Then, the voxels marked as "visible" in the neighborhood of these locations can be retrieved, and their surface normals, curvature information (geometric features), RGB colors, and texture gradients (texture features) can be extracted.

[0041] Based on extracted neighborhood features, structured guidance instructions containing constraints on structural continuity and surface continuity are generated as contextual completion suggestions. For example, if a flat surface and continuous wood grain texture are detected around the occluded area, the completion suggestion could be: "The occluded area should be completed as a continuous wood plane, with the texture direction extending along the positive X-axis and the surface normal pointing upwards." If the occluded area is located in the lower right corner of a book, and the visible part of the book is rectangular, the completion suggestion could be: "Complete as a rectangular structure, maintaining right-angled edges, with a thickness consistent with the visible part." These completion suggestions can be organized as natural language strings or encoded as fixed-dimensional embedding vectors.

[0042] After acquiring geometric priors and contextual completion cues, these are fed along with features from the original image into a pre-trained generative model. The generative model typically employs an encoder-decoder structure: the encoder fuses and encodes the input multimodal information (image, voxel occupancy tensor, text cues) to obtain a feature representation in the latent space; the decoder then progressively generates a complete voxelized 3D model based on this feature representation. During generation, the model is constrained to generate voxels only in regions marked as "occupied" or "occluded" in the geometric priors, while regions marked as "empty" are forced to be filled with blanks. This ensures that the generated result maintains strict alignment with existing visible geometric information.

[0043] Following this, the 3D model can be further optimized. Specifically, to ensure that the generated model conforms to physical principles (e.g., objects will not float in the air or penetrate each other), this embodiment introduces a physical constraint correction step. This step can be performed based on the initial 3D model generated in step 140, or it can be performed alternately with step 140.

[0044] In some implementations, a pre-trained visual language model can be invoked to perform scene understanding on the original image and identify spatial association information between target objects; the spatial association information includes at least one of support relationship, contact relationship and stacking relationship; the spatial association information is mapped to a quantization constraint rule based on voxel coordinates; a constraint cost function is constructed based on the quantization constraint rule, and the 3D model is adjusted based on the constraint cost function to obtain the updated 3D model.

[0045] First, a pre-trained visual language model (such as GPT-4v, LLaVA, or CogVLM) is invoked to perform scene understanding on the original image. The original image and the semantic label list obtained in step 110 are taken as input, along with fixed prompts, such as: "Please analyze the spatial relationships between all objects in the image, including support relationships (e.g., A supports B), contact relationships (e.g., A and B are in contact), and stacking relationships (e.g., A is stacked on B). Please only output the list of relationships in the format 'Object A - Relationship - Object B', without outputting anything else." The visual language model returns a series of semantic relationship descriptions.

[0046] Then, the natural language relation descriptions returned by the visual language model are mapped to quantization constraint rules based on voxel coordinates. For example, the support relationship "table-support-book" can be transformed into a mathematical constraint: the coordinate values ​​of all voxels on the bottom surface of the book in the Z-axis direction must be greater than or equal to the coordinate values ​​of the voxels on the top surface of the table in the Z-axis direction minus a very small tolerance, and the gap between the two should be close to zero. More generally, this application defines the following quantization constraint rules: (1) No relational constraints: If there are no specific spatial relational constraints between two targets, their entity voxels should not overlap, and the minimum distance between voxel boundaries should be ≥ 1 voxel side length (to avoid visually unreasonable proximity).

[0047] (2) Contact constraint: The two target voxels are allowed to fit together, but not to penetrate each other, that is, the number of overlapping voxels is 0, and there is at least one pair of adjacent voxels (face contact or edge contact).

[0048] (3) Support constraint: If target A supports target B, in addition to satisfying the contact constraint, the projection of the bottom surface of B onto the Z-axis must also satisfy the projection overlap rate of the top surface of A with ≥85% (i.e., most of the bottom surface of B falls within the range of the top surface of A).

[0049] (4) Stacking constraint: In addition to satisfying the support constraint, it is further required that the semantic categories of the two targets are the same or similar, and the distance between the voxel centers of the two targets in the Z-axis direction is ≤2×voxel side length (to ensure that the stacked objects are not too high or too low).

[0050] After obtaining the quantification constraint rules (1)-(4) above, a constraint cost function (also known as a loss function) can be constructed. This cost function is used to quantify the degree to which the current 3D model violates each constraint. For example, for support constraints, if there is a gap (floating) between the bottom surface of the book and the top surface of the table, the cost value is proportional to the gap distance; if penetration occurs, the cost value is proportional to the penetration depth and the overlap volume. Specifically: Contact cost = number of overlapping voxels × penalty coefficient (e.g., 10) Support cost = Σ(Z-axis distance between voxels B and A in the region violating support constraints) × penalty coefficient (e.g., 100) Stacking cost = max(0, actual Z-axis center distance - 2 × voxel side length) × penalty coefficient (e.g., 200) Total cost is the sum of all individual costs.

[0051] Then, the gradient descent algorithm is used to iteratively optimize the pose parameters (including translations Δx, Δy, Δz and rotation angles Δroll, Δpitch, Δyaw) of each target object in the 3D model. The gradient of the total cost relative to each parameter is calculated, and the parameters are updated along the gradient descent direction to gradually reduce the total cost. The optimization process sets a learning rate (e.g., 0.01), momentum (e.g., 0.9), and a maximum number of iterations (e.g., 100). The process stops when the total cost falls below a preset threshold (e.g., 10) or the maximum number of iterations is reached. The optimized 3D model is the updated 3D model.

[0052] Because voxel models are composed of discrete small cubes, using them directly for rendering or 3D printing will result in severely jagged edges. Therefore, the voxel models need to be converted into smooth triangular meshes before generating the final model.

[0053] The conversion steps may include: using the moving cube algorithm to convert the voxel structure of the updated 3D model into a smooth triangular mesh; mapping the texture information of the original image to the triangular mesh to generate the final 3D model file.

[0054] Specifically, first, the multi-scale voxel array is converted into a continuous Signed Distance Field (SDF). An SDF is a scalar field where the value at each point represents the distance from that point to the nearest object surface; values ​​are negative inside the object, positive outside, and zero at the surface. Specifically, the SDF value inside a voxel (at least one voxel edge length from the surface) is set to -1, the SDF value at the voxel surface (exactly on the object boundary) is set to 0, and the SDF value outside the voxel is set to 1. Then, the Marching Cubes algorithm is applied to the SDF field to extract zero isosurfaces by searching for 15 predefined topological cases, resulting in a mesh model composed of triangular patches.

[0055] To improve mesh quality, post-processing can be performed: normal interpolation of the mesh (to smooth the surface), merging redundant triangles with too small an area (e.g., triangles with an area ≤ 0.1 mm²), repairing tiny holes (e.g., holes formed by fewer than 10 triangles), while preserving sharp features (such as corners and edges).

[0056] Finally, the texture information of the original image is mapped onto a triangular mesh. Specifically, for each vertex on the mesh, its 3D coordinates are back-projected onto the camera imaging model used in step 120 to obtain its pixel coordinates in the original image, thereby extracting the RGB color value of that vertex. For vertices appearing in multiple original images, a weighted average of the corresponding pixel colors in each image can be taken (the weights can be determined based on the angle between the viewpoint direction and the surface normal). For regions with missing textures (e.g., occluded areas without corresponding pixels in the original image), texture synthesis algorithms can be used to complete them, ensuring that the alignment error between the completed part and the voxel model space is ≤0.5mm. Finally, a common 3D file format, such as OBJ, FBX, or GLTF, is exported, and each point is checked to ensure that the geometric deviation from the voxel model is ≤1 voxel side length, and the visual smoothness difference rate from the directly generated mesh model is ≤5% (difference rate = (variance of curvature after optimization - variance of curvature of directly generated mesh) / variance of curvature of directly generated mesh × 100%). If the requirements are not met, the conversion is repeated until the requirements are met to complete the final 3D reconstruction process.

[0057] For ease of understanding, an exemplary embodiment is provided herein to explain the three-dimensional model generation method provided in this application: (1) Scene description: A user took a photo of a cluttered desk. In the photo, a mug obscures the corner of a notebook behind it, making the notebook appear to float above the desk (due to a perspective illusion).

[0058] (2) Processing flow: a. Target Recognition and Depth Estimation: "Mug," "Notebook," "Desktop," and "Background Wall" were identified. The depth map showed the mug was closest, the notebook was in the center, and the background wall was furthest.

[0059] b. Multi-scale voxel construction: Construct a voxel mesh. For the areas containing the "mug" and "notebook," create a fine mesh with a precision of 1cm; for the "background wall," create a coarse mesh with a precision of 5cm. Mark the area where the notebook is obscured by the mug as the "occluded state."

[0060] c. Generation and suggestion completion: The generated model received the following prompt: "The notebook is occluded, but based on the visible portion, it should be a rectangle, and the wood grain texture should extend to the right." The model then generated a complete voxel model of the notebook, including the occluded portion.

[0061] d. Physical constraint correction: Visual language model analysis indicates that "the notebook should be placed on the desktop".

[0062] The initial model detected a 2cm gap between the bottom of the laptop and the desktop (making it appear to be floating). A cost function was then constructed to move the laptop 2cm downwards along the Z-axis until it just touches the desktop, eliminating the floating appearance.

[0063] e. Meshing and Output: Finally, the calibrated voxel model is converted into a smooth triangular mesh, and wood grain and cup textures extracted from the photo are applied to output a physically plausible and visually realistic 3D scene.

[0064] As can be seen from the above, the embodiments of this application, through multi-scale voxel partitioning, can adaptively allocate voxel resolution according to the importance of the target, resulting in rich foreground details and low background computational overhead, thus balancing reconstruction quality and efficiency. By extracting geometric and texture features around the occluded area to generate structured contextual completion prompts, the generation model is guided to perform occlusion completion with visual coherence and physical rationality, avoiding the problem of structural distortion in the completion results in traditional methods. By using a visual language model, abstract spatial relationships (support, contact, stacking) are quantified into voxel-level constraint rules and computable cost functions, and gradient descent is used to optimize the target pose, so that the correction of physical consistency has clear and repeatable judgment criteria. The entire process starts from a single RGB image, without the need for multi-view input or complex sensor equipment, which facilitates cost control.

[0065] Reference Figure 2 To facilitate understanding, an exemplary embodiment is provided herein to further describe the three-dimensional model generation method provided in this application.

[0066] Step 1: Step 1.1: Obtain the raw RGB image input by the user. Load the pre-trained GroundedSAMv2 model, input the raw image, and set the prompt "all objects in the scene" with a confidence threshold of 0.5. The model outputs the bounding box [x_min, y_min, x_max, y_max], a binary segmentation mask (0 for background, 1 for target), and semantic labels (e.g., "mug", "notebook", "table") for each target object. These outputs constitute the list of target bounding boxes and the basic segmentation mask.

[0067] Step 1.2: Use the MoGe monocular depth estimation model to generate a depth map (i.e.) Figure 1 (Pixel-level depth map). For target pairs with overlapping masks (e.g., a mug mask overlapping a notebook mask), the depth values ​​of each pixel in the overlapping region are read and compared: the target with the smaller depth value is identified as the occluder (mug), and the target with the larger depth value is identified as the occluded target (notebook). The mask value of the overlapping region of the occluded target is changed from 1 to 2, thereby generating an "enhanced segmentation mask list", where mask value 0 = background, 1 = target visible, and 2 = target occluded.

[0068] Step Two: Step 2.1: Based on the bounding boxes and depth maps of each target, backproject each pixel into 3D space to obtain the point cloud of the scene. The calculation formulas are detailed in the relevant sections of the preceding embodiments and will not be repeated here. Extract the extreme values ​​of the X, Y, and Z axes from the point cloud and expand them outwards by 10% to obtain the range of the global 3D physical space.

[0069] Step 2.2: Parse semantic labels: Classify targets with an area ratio ≥ 5% in the image (such as mugs, notebooks, and books) as foreground targets; classify walls, floors, and desktops as background environment targets. Assign a first voxel resolution to the physical space region where the foreground targets are located, with a voxel side length of 1 cm; assign a second voxel resolution to the physical space region where the background environment targets are located, with a voxel side length of 5 cm.

[0070] Step 2.3: Call the Open3D library's voxel meshing interface to generate a multi-scale voxel array based on the global spatial extent and the resolution of each region. Calculate the number of voxels along the X, Y, and Z axes. For each voxel, find its corresponding pixel region using coordinate conversion formulas, and assign the mask state (0 / 1 / 2) and depth value of the pixels in that region to that voxel. When multiple pixels are mapped to the same voxel, a majority voting method is used to determine the final mask state. This constructs a multi-scale voxel mesh that includes both spatial occupancy and occlusion states.

[0071] Step 3: Step 3.1: Extract occupancy state information from the multi-scale voxel mesh and encapsulate it into a tensor of shape (W, H, D) as a geometric prior, limiting the generative model to generate content only in voxel regions with occupancy state = 1. For voxel regions with occlusion state = 2, the terminal locates its position and extracts the geometric features (surface normals, curvature) and texture features (RGB color, texture gradient) of the surrounding visible voxels. Based on these features, generate structured context completion prompts, such as: "Semantic type: notebook, occlusion position: bottom right 1 / 4 area, visible part is rectangular with horizontal wood grain texture, completion requirements: maintain rectangular geometry, wood grain extends continuously along the positive X-axis, thickness is consistent with the visible part, after completion the bottom surface fits the top surface of the desktop." Step 3.2: Load the VAE+LDM model and its pre-trained weights. The original RGB image features, voxel geometric prior tensors, and contextual completion cues are concatenated and used as conditional input to the model. The model generates a voxelized 3D model target by target. During generation, the alignment error between the newly generated voxel and the original voxel mesh in spatial coordinates is checked every 10 voxel steps. If the alignment error of a voxel exceeds one voxel side length (i.e., 1 cm), regeneration is triggered, and the local area is refined until the alignment error of all voxels is ≤1 cm. After this step, an initial voxelized 3D model with occluded areas completed is obtained.

[0072] Step Four: Step 4.1: Input the original RGB image and semantic label list into the GPT-4v model, using fixed prompts: analyze the support, contact, and stacking relationships of all objects in the image, outputting only the relationship list in the format of object A-relationship-object B. The model returns a relationship list, for example: table-support-notebook, table-support-mug, notebook-contact-book. Then, based on the relationship list and the current voxel resolution (1 cm foreground, 5 cm background), the abstract relationships are transformed into quantized constraint rules. Step 4.2: Traverse all target pairs and calculate the total constraint cost based on the current voxel mesh.

[0073] Step 4.3: With the goal of minimizing the total constraint cost, the gradient descent algorithm implemented in PyTorch is used to optimize the translation (Δx, Δy, Δz) and rotation (Δroll, Δpitch, Δyaw) parameters for each objective. The learning rate is set to 0.01, momentum to 0.9, and the maximum number of iterations to 100. In each iteration, the gradient of the total cost with respect to each parameter is calculated, and the parameters are updated in the opposite direction of the gradient. As the laptop descends 0.1 cm along the negative Z-axis each time, the support cost gradually decreases. After 20 iterations, the gap decreases to 0, the support cost decreases to 0, and the total cost is less than the threshold of 10, at which point the optimization terminates. At this point, the laptop lands on the table, eliminating the floating phenomenon. Simultaneously, all objective pairs are checked to ensure no penetration (overlapping voxels = 0). The updated 3D model is obtained after optimization.

[0074] Step 5: Step 5.1: Convert the updated multiscale voxel array into a continuous symbolic distance field (SDF) that can be recognized by the MarchingCubes algorithm in Open3D.

[0075] Step 5.2: Post-process the mesh: Perform normal interpolation to eliminate stepped edges; merge redundant triangular faces with an area ≤0.1mm²; use a hole-filling algorithm to repair tiny holes formed by ≤10 triangular faces; at the same time, preserve key geometric features such as edges and texture boundaries to avoid over-smoothing.

[0076] Step 5.3: Reuse the VAE+LDM model from Step 1, backproject the 3D coordinates of each vertex of the optimized mesh onto the original image using camera intrinsics, and extract the corresponding RGB texture values. For regions lacking texture (e.g., the completed surface), the model synthesizes and completes the texture based on the surrounding visible textures, ensuring that the spatial alignment error between the completed part and the voxel model is ≤0.5 mm.

[0077] Step 5.4: Export the final mesh model as an OBJ file. Compare the geometric deviations of each mesh vertex with the corresponding voxel center in the original voxel model, ensuring the maximum deviation is ≤1 voxel side length (1cm). Simultaneously calculate the difference rate between the optimized mesh curvature variance and the directly generated mesh curvature variance. The formula is (optimized mesh curvature variance - directly generated mesh curvature variance) / directly generated mesh curvature variance × 100%, requiring a difference rate ≤5%. If any indicator fails to meet the standard, repeat steps 3.1 to 3.3 until the requirements are met. Finally, output a geometrically accurate, textured, and physically sound 3D model file, completing the entire reconstruction process.

[0078] To better implement the above methods, this application also provides a three-dimensional model generation device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0079] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the three-dimensional model generation device specifically integrated into the terminal as an example.

[0080] For example, such as Figure 3 As shown, the 3D model generation device may include a segmentation unit 210, a decision unit 220, a construction unit 230, and a generation unit 240, as follows: (a) Segmentation unit 210, used to acquire the original image and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; (ii) Determination unit 220, used to determine the occlusion relationship between each of the target objects and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map; (iii) Construction unit 230 is used to perform multi-scale voxel division on the global three-dimensional physical space according to the image feature information, and construct a multi-scale voxel mesh including space occupancy status and occlusion status. (iv) Generation unit 240 is used to extract the state information of the multi-scale voxel mesh as a geometric prior, and combine it with context completion prompts generated based on occlusion state to drive the pre-trained generative model to generate a 3D model.

[0081] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0082] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0083] As can be seen from the above, the 3D model generation device of this embodiment acquires the original image and performs target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; determines the occlusion relationship between each target object and generates a depth map, and determines the range of the global 3D physical space based on the depth map; divides the global 3D physical space into multi-scale voxels according to the image feature information, and constructs a multi-scale voxel mesh that includes the space occupancy state and the occlusion state; extracts the state information of the multi-scale voxel mesh as a geometric prior, and combines it with the context completion prompts generated based on the occlusion state to drive the pre-trained generative model to generate a 3D model.

[0084] Therefore, the embodiments of this application can adaptively allocate voxel resolution according to the importance of the target through multi-scale voxel partitioning, resulting in rich foreground details and low background computational overhead, thus balancing reconstruction quality and efficiency. By extracting geometric and texture features around the occluded area to generate structured contextual completion prompts, the generative model is guided to perform occlusion completion with visual coherence and physical rationality, avoiding the problem of structural distortion in the completion results in traditional methods. By using a visual language model, abstract spatial relationships (support, contact, stacking) are quantified into voxel-level constraint rules and computable cost functions, and gradient descent is used to optimize the target pose, so that the correction of physical consistency has clear and repeatable judgment criteria. The entire process starts from a single RGB image, without the need for multi-view input or complex sensor equipment, which facilitates cost control.

[0085] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0086] In some embodiments, the three-dimensional model generation device can also be integrated into multiple electronic devices. For example, the three-dimensional model generation device can be integrated into multiple servers, and the three-dimensional model generation method of this application can be implemented by multiple servers.

[0087] In this embodiment, the electronic device will be described in detail as a terminal, for example, such as... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will understand that... Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 310 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 320, and by calling data stored in the memory 320, it performs various functions and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 310.

[0088] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0089] The electronic device also includes a power supply 330 that supplies power to the various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 330 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0090] The electronic device may also include an input module 340, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0091] The electronic device may also include a communication module 350. In some embodiments, the communication module 350 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 350 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0092] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 310 in the electronic device loads the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 runs the application programs stored in the memory 320 to realize various functions, as follows: This application's embodiments utilize multi-scale voxel partitioning to adaptively allocate voxel resolution based on target importance, resulting in rich foreground details and low background computational overhead, balancing reconstruction quality and efficiency. By extracting geometric and texture features around the occluded region to generate structured contextual completion prompts, it guides the generative model to perform occlusion completion with visual coherence and physical plausibility, avoiding the structural distortion problem in traditional completion results. Through a visual language model, abstract spatial relationships (support, contact, stacking) are quantified into voxel-level constraint rules and computable cost functions, and gradient descent is used to optimize the target pose, giving the correction of physical consistency a clear and repeatable judgment criterion. The entire process starts from a single RGB image, eliminating the need for multi-view input or complex sensor equipment, thus facilitating cost control.

[0093] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0094] As can be seen from the above, the embodiments of this application can ensure that the simulation data is equivalent to the real data at the level of syntax rules and data structure, so that it can directly replace the real data for model training and system testing, and achieve efficient simulation output of structurally similar data.

[0095] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0096] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the three-dimensional model generation methods provided in embodiments of this application. For example, the instructions can execute the following steps: The system acquires the original image and performs target recognition and segmentation to obtain at least one target object and its corresponding image feature information. It determines the occlusion relationship between each target object and generates a depth map, and determines the range of the global 3D physical space based on the depth map. It divides the global 3D physical space into multi-scale voxels according to the image feature information and constructs a multi-scale voxel mesh that includes the space occupancy state and the occlusion state. It extracts the state information of the multi-scale voxel mesh as a geometric prior and combines it with context completion prompts generated based on the occlusion state to drive the pre-trained generative model to generate a 3D model.

[0097] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0098] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0099] Since the instructions stored in the storage medium can execute the steps in any of the three-dimensional model generation methods provided in the embodiments of this application, the beneficial effects that any of the three-dimensional model generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0100] The foregoing has provided a detailed description of a three-dimensional model generation method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating a three-dimensional model, characterized in that, include: Acquire the original image, and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; Determine the occlusion relationship between each target object and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map; Based on the image feature information, the global three-dimensional physical space is divided into multi-scale voxels to construct a multi-scale voxel mesh that includes space occupancy status and occlusion status. Locate the occluded voxel regions in the multi-scale voxel mesh, and extract the geometric and texture features of the visible voxels surrounding the voxel regions. Based on the geometric and texture features, a structured guidance instruction containing constraints on structural continuity and surface continuity is generated as a context completion prompt; The state information of the multi-scale voxel mesh is extracted as a geometric prior, and combined with the context completion prompts, the pre-trained generative model is driven to generate a 3D model. A pre-trained visual language model is invoked to perform scene understanding on the original image and identify spatial association information between various target objects; the spatial association information includes contact relationships. The spatial association information is mapped to quantization constraint rules based on voxel coordinates; A constraint cost function is constructed based on the quantization constraint rules, and the three-dimensional model is adjusted based on the constraint cost function to obtain an updated three-dimensional model. The step of performing multi-scale voxel partitioning of the global three-dimensional physical space based on the image feature information includes: The semantic labels and spatial location information in the image feature information are analyzed to classify the target object into foreground objects and background environment objects. The physical space region where the foreground target is located is divided into a first voxel resolution, and the physical space region where the background environment target is located is divided into a second voxel resolution; wherein the voxel size corresponding to the first voxel resolution is smaller than the voxel size corresponding to the second voxel resolution.

2. The three-dimensional model generation method as described in claim 1, characterized in that, The step of constructing a constraint cost function based on the quantization constraint rules, and adjusting the 3D model based on the constraint cost function to obtain an updated 3D model, includes: The voxel penetration rate and spatial gap distance between target objects are calculated based on the quantization constraint rules to construct the constraint cost function; The gradient descent algorithm is used to iteratively optimize the pose parameters of the target object in the 3D model until the constraint cost function converges to a preset threshold, thus obtaining the updated 3D model.

3. The three-dimensional model generation method according to any one of claims 1-2, characterized in that, After constructing a constraint cost function based on the quantization constraint rules and adjusting the 3D model based on the constraint cost function to obtain the updated 3D model, the process further includes: The moving cube algorithm is used to convert the voxel structure of the updated 3D model into a smooth triangular mesh. The texture information of the original image is mapped onto the triangular mesh to generate the final 3D model file.

4. A three-dimensional model generation device, characterized in that, include: The segmentation unit is used to acquire the original image and perform target recognition and segmentation on the original image to obtain at least one target object and its corresponding image feature information; The determination unit is used to determine the occlusion relationship between each of the target objects and generate a depth map, and determine the range of the global three-dimensional physical space based on the depth map; The construction unit is used to perform multi-scale voxel division on the global three-dimensional physical space according to the image feature information, and construct a multi-scale voxel mesh that includes space occupancy status and occlusion status. It is also used to parse the semantic labels and spatial location information in the image feature information, divide the target object into foreground targets and background environment targets; divide the physical space region where the foreground target is located into a first voxel resolution, and divide the physical space region where the background environment target is located into a second voxel resolution; wherein, the voxel size corresponding to the first voxel resolution is smaller than the voxel size corresponding to the second voxel resolution; The generation unit is used to extract the state information of the multi-scale voxel mesh as a geometric prior, and combine it with context completion prompts to drive the pre-trained generative model to generate a 3D model. Before extracting the state information of the multi-scale voxel mesh as a geometric prior and combining it with contextual completion prompts to drive the pre-trained generative model to generate a 3D model, the process further includes locating the voxel regions in the multi-scale voxel mesh that are in an occluded state, and extracting the geometric and texture features of the visible voxels around the voxel regions; and generating structured guidance instructions containing constraints on structural continuity and surface continuity based on the geometric and texture features as contextual completion prompts. After extracting the state information of the multi-scale voxel mesh as a geometric prior and combining it with contextual completion prompts to drive the pre-trained generative model to generate a 3D model, the process further includes calling a pre-trained visual language model to perform scene understanding on the original image and identify spatial association information between various target objects; the spatial association information includes contact relationships; the spatial association information is mapped to quantization constraint rules based on voxel coordinates; a constraint cost function is constructed based on the quantization constraint rules, and the 3D model is adjusted based on the constraint cost function to obtain an updated 3D model.

5. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the three-dimensional model generation method as described in any one of claims 1 to 3.

6. A computer program product, characterized in that, It includes computer program instructions, which, when executed by a processor, implement the steps in the three-dimensional model generation method according to any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the three-dimensional model generation method according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Three-dimensional model reconstruction method and device and electronic equipment

    CN115346018A

  • Spatial intelligent object representation method for traceable generation of occlusion area

    CN121902845A