Three-dimensional model generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the application disclose a three-dimensional model generation method and device, electronic equipment and a storage medium, and belong to the technical field of computer vision. The method comprises the following steps: generating a background three-dimensional model of a two-dimensional image to be reconstructed; performing semantic segmentation on the two-dimensional image to be reconstructed to obtain at least part of a semantic segmentation image of each foreground object, and generating a mask of the corresponding foreground object; repairing at least part of the semantic segmentation image to obtain a complete semantic segmentation image, and generating an object three-dimensional model of the corresponding foreground object according to the complete semantic segmentation image; determining a background point cloud and an object point cloud of each foreground object from a three-dimensional point cloud according to the mask of each foreground object; and determining a target three-dimensional model in a scene corresponding to the two-dimensional image to be reconstructed according to the background point cloud, the background three-dimensional model, the object point cloud of each foreground object and the object three-dimensional model. The application can improve the quality of a three-dimensional model generated by three-dimensional reconstruction of a single image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method, apparatus, electronic device, and storage medium for generating three-dimensional models. Background Technology

[0002] Driven by the global digital wave, the demand for highly immersive and realistic 3D digital content is growing, especially in fields such as virtual reality, digital twins, games and film production, and embodied intelligence.

[0003] In related technologies, a 3D model is generated by reconstructing a single 2D image. However, when only a single image of the target scene can be obtained, the objects in the image (such as furniture, walls, etc.) are usually occluded. When generating a 3D model using a single image, the related technologies tend to treat mutually occluded objects as a whole, which leads to a misunderstanding of the spatial layout between the objects and results in poor quality of the generated 3D model. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for generating three-dimensional models, which can improve the quality of three-dimensional models generated by three-dimensional reconstruction using a single image.

[0005] To achieve the above objectives, one embodiment of this application provides a method for generating a three-dimensional model, including: Obtain the two-dimensional image to be reconstructed, and perform three-dimensional reconstruction on the background image in the two-dimensional image to be reconstructed to generate a three-dimensional background model; Semantic segmentation is performed on the 2D image to be reconstructed to obtain at least a partial semantic segmentation image for each foreground object, and a mask for the corresponding foreground object is generated based on the at least partial semantic segmentation image; Repair at least a portion of the semantic segmentation image to obtain the complete semantic segmentation image of the corresponding foreground object, and generate the corresponding three-dimensional object model of the foreground object based on the complete semantic segmentation image; Determine the 3D point cloud of the 2D image to be reconstructed, and determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud based on the mask of each foreground object; The background layout parameters are determined based on the background point cloud and background 3D model, and the corresponding object layout parameters are determined based on the object point cloud and object 3D model of each foreground object. Based on the background layout parameters and the object layout parameters corresponding to each foreground object, the background 3D model and the 3D model of each object are combined to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0006] In some embodiments, semantic segmentation of the two-dimensional image to be reconstructed yields at least a partially semantically segmented image for each foreground object, including: Retrieve multiple preset semantic tags; Semantic segmentation is performed on the 2D image to be reconstructed based on multiple preset semantic labels to obtain the object bounding box of each foreground object in the 2D image to be reconstructed under different preset semantic labels, and the confidence score of the object bounding box; For each foreground object, the highest confidence score is selected from multiple confidence scores corresponding to the foreground object as the target confidence score. Based on the target confidence score and the bounding box of the object corresponding to the target confidence score, at least a portion of the semantic segmentation image of the foreground object is determined.

[0007] In some embodiments, determining the three-dimensional point cloud of the two-dimensional image to be reconstructed includes: Perform geometric transformations on the two-dimensional image to be reconstructed to obtain a pseudo-visual two-dimensional image corresponding to the two-dimensional image to be reconstructed; Based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image, determine the depth image corresponding to the two-dimensional image to be reconstructed; Obtain the camera parameters corresponding to the captured 2D image to be reconstructed, and determine the 3D point cloud of the 2D image to be reconstructed based on the camera parameters and the depth image.

[0008] In some embodiments, determining the depth image corresponding to the two-dimensional image to be reconstructed based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image includes: Features of the two-dimensional image to be reconstructed are extracted from the features of the two-dimensional image to be reconstructed, and features of the pseudo-visual two-dimensional image are extracted from the features of the pseudo-visual two-dimensional image. The features of the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image features are input into a pre-trained cross-attention model to obtain enhanced features of the two-dimensional image to be reconstructed and enhanced pseudo-visual two-dimensional image features. The pre-trained cross-attention model is trained from the sample two-dimensional image to be reconstructed and the sample two-dimensional image features. Based on the enhanced features of the two-dimensional image to be reconstructed and the enhanced pseudo-visual features of the two-dimensional image, the depth image corresponding to the two-dimensional image to be reconstructed is determined.

[0009] In some embodiments, determining background layout parameters based on background point clouds and background 3D models includes: Obtain the initial background layout parameters and sample the background 3D model to obtain the sampled background point cloud corresponding to the background 3D model; Calculate the background difference between the background point cloud and the sampled background point cloud, update the initial background layout parameters based on the background difference, and obtain the updated background layout parameters; Adjust the background point cloud and sampled background point cloud according to the updated background layout parameters to obtain the updated background point cloud and the updated sampled background point cloud; The updated background point cloud is used as the new background point cloud, the updated sampled background point cloud is used as the new sampled background point cloud, and the updated background layout parameters are used as the new initial background layout parameters. Return to the step of calculating the background difference between the background point cloud and the sampled background point cloud, until the first preset condition is reached, and use the updated background layout parameters obtained from the last update as the background layout parameters.

[0010] In some embodiments, calculating the background difference between the background point cloud and the sampled background point cloud includes: When the number of updates to the initial background layout parameters is less than the preset threshold for background updates, the background three-dimensional spatial difference between the background point cloud and the sampled background point cloud is calculated, and the background difference between the background point cloud and the sampled background point cloud is determined based on the background three-dimensional spatial difference. When the number of updates to the initial background layout parameters is equal to or greater than the preset threshold for background updates, the background point cloud is projected to obtain the first projected background image, and the sampled background point cloud is projected to obtain the second projected background image. Calculate the background two-dimensional image difference between the first projected background image and the second projected background image, and determine the background difference between the background point cloud and the sampled background point cloud based on the background three-dimensional spatial difference and the background two-dimensional image difference.

[0011] In some embodiments, the corresponding object layout parameters are determined based on the object point cloud and the object 3D model of each foreground object, including: Obtain the initial object layout parameters for each foreground object, and sample the 3D model of each foreground object to obtain the sampled object point cloud for each foreground object; For each foreground object, calculate the object difference between the object point cloud and the sampled object point cloud, and update the initial object layout parameters based on the object difference to obtain the updated object layout parameters; Adjust the object point cloud and sampled object point cloud according to the updated object layout parameters to obtain the updated object point cloud and the updated sampled object point cloud; The updated object point cloud is used as the new object point cloud, the updated sampled object point cloud is used as the new sampled object point cloud, and the updated object layout parameters are used as the new initial object layout parameters. Return to the step of calculating the object difference between the object point cloud and the sampled object point cloud until the second preset condition is reached, and use the updated object layout parameters obtained from the last update as the object layout parameters of the corresponding foreground object.

[0012] In some embodiments, calculating the object differences between the object point cloud and the sampled object point cloud includes: When the number of updates to the initial object layout parameters is less than the preset threshold for object updates, the object three-dimensional space difference between the object point cloud and the sampled object point cloud is calculated, and the object difference between the object point cloud and the sampled object point cloud is determined based on the object three-dimensional space difference. When the number of updates to the initial object layout parameters is equal to or greater than the preset threshold for object updates, the object point cloud is projected to obtain the first projected object image, and the sampled object point cloud is projected to obtain the second projected object image. Calculate the object 2D image difference between the first projected object image and the second projected object image, and determine the object difference between the object point cloud and the sampled object point cloud based on the object 3D spatial difference and the object 2D image difference.

[0013] To achieve the above objectives, one embodiment of this application provides a three-dimensional model generation apparatus, comprising: The acquisition module is used to acquire the two-dimensional image to be reconstructed and to perform three-dimensional reconstruction on the background image in the two-dimensional image to be reconstructed, generating a three-dimensional background model; The semantic segmentation module is used to perform semantic segmentation on the two-dimensional image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and to generate a mask of the corresponding foreground object based on the at least partial semantic segmentation image; The repair module is used to repair at least a portion of the semantic segmentation image to obtain a complete semantic segmentation image of the corresponding foreground object, and to generate a 3D model of the corresponding foreground object based on the complete semantic segmentation image. The point cloud determination module is used to determine the 3D point cloud of the 2D image to be reconstructed, and to determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud based on the mask of each foreground object. The layout parameter determination module is used to determine the background layout parameters based on the background point cloud and the background 3D model, and to determine the corresponding object layout parameters based on the object point cloud and the object 3D model of each foreground object. The target 3D model generation module is used to combine the background 3D model and the 3D model of each object according to the background layout parameters and the object layout parameters corresponding to each foreground object to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0014] To achieve the above objectives, one aspect of this application provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the steps in the three-dimensional model generation method provided in this application.

[0015] To achieve the above objectives, one aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps in the three-dimensional model generation method provided in this application.

[0016] To achieve the above objectives, one aspect of this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the three-dimensional model generation method provided in this application.

[0017] The 3D model generation method, apparatus, electronic device, and storage medium proposed in this application acquire a 2D image to be reconstructed, and perform 3D reconstruction on the background image in the 2D image to be reconstructed to generate a background 3D model; perform semantic segmentation on the 2D image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and generate a mask for the corresponding foreground object based on the at least partial semantic segmentation image; repair the at least partial semantic segmentation image to obtain a complete semantic segmentation image of the corresponding foreground object, and generate a 3D object model of the corresponding foreground object based on the complete semantic segmentation image; determine the 3D point cloud of the 2D image to be reconstructed, and determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud based on the mask of each foreground object; determine background layout parameters based on the background point cloud and the background 3D model, and determine corresponding object layout parameters based on the object point cloud and the object 3D model of each foreground object; combine the background 3D model and the 3D model of each object based on the background layout parameters and the object layout parameters corresponding to each foreground object to obtain a target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0018] This embodiment first separates the background and each foreground object in the 2D image to be reconstructed. Then, it restores the complete semantic information of each foreground object by repairing the occluded areas. Subsequently, it independently constructs 3D models of the background image and each repaired foreground object without occlusion, resulting in high-quality 3D models of the background and objects corresponding to each foreground object. In this way, this embodiment can effectively overcome the problem of difficulty in accurately understanding the complete structure and spatial layout of the background and each foreground object due to mutual occlusion between foreground objects when only a single 2D image is available. Next, by determining the point clouds corresponding to the background and each foreground object, the layout parameters corresponding to the background and each foreground object are calculated separately, thereby ensuring that when the target 3D model is finally generated, the spatial position, scale, and occlusion relationship between the background and each foreground object are consistent with the scene represented by the 2D image to be reconstructed. Finally, this embodiment improves the quality of the 3D model generated by 3D reconstruction using a single image.

[0019] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the system framework corresponding to the three-dimensional model generation method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the three-dimensional model generation method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the two-dimensional image to be reconstructed provided in the embodiments of this application; Figure 4 This is a schematic diagram of the data processing flow of the three-dimensional model generation method provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the determination of object layout parameters provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the effect of determining the three-dimensional model of the object provided in an embodiment of this application; Figure 7 This is a schematic diagram comparing the three-dimensional model generation method provided in this application embodiment with the three-dimensional model generated by the traditional method; Figure 8 This is a schematic diagram of the module structure of the three-dimensional model generation device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It should be noted that in all specific embodiments of this application, when it is necessary to obtain the two-dimensional image to be reconstructed, permission or consent from the relevant personnel managing the two-dimensional image to be reconstructed will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of relevant personnel, separate permission or consent from the relevant personnel will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the separate permission or consent of the relevant personnel will the necessary two-dimensional image to be reconstructed be obtained to enable the normal operation of this application embodiment. Other data obtained in this application embodiment are all authorized and legal data, and will not be described in detail here.

[0024] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer computer devices, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0025] First, describe the technical problems existing in the relevant technologies: Driven by the global digital wave, the demand for highly immersive and realistic 3D digital content is growing, especially in fields such as virtual reality, digital twins, games and film production, and embodied intelligence.

[0026] In related technologies, a 3D model is generated by reconstructing a single 2D image. However, when only a single image of the target scene can be obtained, the objects in the image (such as furniture, walls, etc.) are usually occluded. When generating a 3D model using a single image, the related technologies tend to treat mutually occluded objects as a whole, which leads to a misunderstanding of the spatial layout between the objects and results in poor quality of the generated 3D model.

[0027] For example, in a real-life photo of a family living room, a coffee table is placed in front of the sofa, with books and a vase on it, while part of the wall is obscured by the sofa and cabinets. Because only this single two-dimensional image can be captured, traditional single-viewing... Figure 33D reconstruction methods often model sofas, coffee tables, and the items on them as a coherent whole, thus failing to effectively distinguish the boundaries and depth levels of individual objects. The resulting 3D models may exhibit problems such as coffee tables embedded inside sofas, vases floating in mid-air, or holes appearing in the wall geometry due to occlusion. More seriously, the physical support relationships (e.g., vases should be placed on the coffee table surface) and spatial order (e.g., cabinets should be in front of walls) between these objects cannot be correctly reconstructed, leading to viewpoint conflicts, structural inconsistencies, and even geometric penetration when viewing the scene from multiple perspectives.

[0028] For example, in film and television production, an art director may only provide a concept design as the sole reference for a virtual scene. If the character in the image is standing behind a table, with only the upper body visible, traditional methods may struggle to determine whether the character's legs are present or whether there are drawers or supporting structures under the table. Due to the lack of decoupling between the foreground and background, the reconstructed 3D assets often lose the reasonable structure of the occluded parts, or even incorrectly merge the character's body with the tabletop into a single mesh. This not only reduces the geometric accuracy and texture realism of the model but also greatly limits the subsequent interactive editing and dynamic simulation capabilities in game engines or virtual reality (VR) environments.

[0029] The 3D model generation method, apparatus, electronic device, and storage medium proposed in this application acquire a 2D image to be reconstructed, and perform 3D reconstruction on the background image in the 2D image to be reconstructed to generate a background 3D model; perform semantic segmentation on the 2D image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and generate a mask for the corresponding foreground object based on the at least partial semantic segmentation image; repair the at least partial semantic segmentation image to obtain a complete semantic segmentation image of the corresponding foreground object, and generate a 3D object model of the corresponding foreground object based on the complete semantic segmentation image; determine the 3D point cloud of the 2D image to be reconstructed, and determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud based on the mask of each foreground object; determine background layout parameters based on the background point cloud and the background 3D model, and determine corresponding object layout parameters based on the object point cloud and the object 3D model of each foreground object; combine the background 3D model and the 3D model of each object based on the background layout parameters and the object layout parameters corresponding to each foreground object to obtain a target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0030] This embodiment first separates the background and each foreground object in the 2D image to be reconstructed. Then, it restores the complete semantic information of each foreground object by repairing the occluded areas. Subsequently, it independently constructs 3D models of the background image and each repaired foreground object without occlusion, resulting in high-quality 3D models of the background and objects corresponding to each foreground object. In this way, this embodiment can effectively overcome the problem of difficulty in accurately understanding the complete structure and spatial layout of the background and each foreground object due to mutual occlusion between foreground objects when only a single 2D image is available. Next, by determining the point clouds corresponding to the background and each foreground object, the layout parameters corresponding to the background and each foreground object are calculated separately, thereby ensuring that when the target 3D model is finally generated, the spatial position, scale, and occlusion relationship between the background and each foreground object are consistent with the scene represented by the 2D image to be reconstructed. Finally, this embodiment improves the quality of the 3D model generated by 3D reconstruction using a single image.

[0031] The specific details of the three-dimensional model generation method, apparatus, electronic device, and storage medium provided in the embodiments of this application will be described in detail below.

[0032] Please see Figure 1 , Figure 1 This is a schematic diagram of the system framework corresponding to the three-dimensional model generation method provided in this application embodiment. The three-dimensional model generation method provided in this application embodiment can be applied to this system framework.

[0033] It includes terminal 140, Internet 130, gateway 120, server 110, etc.

[0034] Terminal 140 or server 110 can be a device that performs a three-dimensional model generation method.

[0035] Terminal 140 includes, but is not limited to, mobile phones, tablets, computers, and intelligent computing centers. Terminal 140 can be a single device or a collection of multiple devices. For example, multiple computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, thus forming a terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0036] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0037] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0038] The embodiments of this application can be applied to various scenarios, such as geometric editing, game development, and the construction of embodied intelligent simulation environments. These embodiments can also be used in other 3D model generation scenarios; however, they are only illustrative examples and do not imply any limitation on the application scenarios of the 3D model generation method.

[0039] Next, we will describe the 3D model generation device from the perspective of dimensions, such as... Figure 2 As shown, Figure 2 This is a flowchart illustrating the three-dimensional model generation method provided in this application embodiment. The three-dimensional model generation method is applied to a three-dimensional model generation device. Figure 2 The method may include, but is not limited to, the following steps 210 to 260. When the 3D model generation device executes the 3D model generation method, the specific process is as follows. It should be noted that this embodiment... Figure 2 The order of steps 210 to 260 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0040] Step 210: Obtain the two-dimensional image to be reconstructed, and perform three-dimensional reconstruction on the background image in the two-dimensional image to be reconstructed to generate a three-dimensional background model; Step 220: Perform semantic segmentation on the two-dimensional image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and generate a mask for the corresponding foreground object based on the at least partial semantic segmentation image; Step 230: Repair at least part of the semantic segmentation image to obtain the complete semantic segmentation image of the corresponding foreground object, and generate the corresponding three-dimensional object model of the foreground object based on the complete semantic segmentation image; Step 240: Determine the 3D point cloud of the 2D image to be reconstructed, and determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud according to the mask of each foreground object; Step 250: Determine the background layout parameters based on the background point cloud and background 3D model, and determine the corresponding object layout parameters based on the object point cloud and object 3D model of each foreground object. Step 260: Based on the background layout parameters and the object layout parameters corresponding to each foreground object, combine the background 3D model and the 3D model of each object to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0041] Steps 210 to 260 are described in detail below.

[0042] In step 210, the two-dimensional image to be reconstructed is obtained, and the background image in the two-dimensional image to be reconstructed is reconstructed in three dimensions to generate a three-dimensional background model.

[0043] The 2D image to be reconstructed refers to the 2D image used as the input source for 3D modeling. It typically includes the target scene (such as a shopping mall, office area, or outdoor space) containing at least one foreground object (such as furniture, people, or vehicles) and a background area (such as walls, ground, or sky). The background area is the background image in the 2D image to be reconstructed. Only one 2D image is used for reconstruction within the target scene, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the two-dimensional image to be reconstructed provided in an embodiment of this application. Figure 3 The image showcases a color 2D image of a minimalist interior office scene, containing a desk, a chair, a side table, and a desk lamp against a light blue wall background. A pendant light hangs above. The image exhibits partial occlusion between foreground objects, such as the chair obscuring the desk, and the lamp and side table surfaces obscuring each other. When reconstructing the scene from this image, only this one can be used as a reference; no other images from different angles will be provided as supplementary or auxiliary images.

[0044] Furthermore, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the data processing flow of the three-dimensional model generation method provided in the embodiments of this application. In order to obtain the target three-dimensional model corresponding to the two-dimensional image to be reconstructed, the embodiments of this application need to perform three branch processing on the two-dimensional image to be reconstructed in parallel: (1) background reconstruction; (2) foreground reconstruction; (3) point cloud estimation.

[0045] First, let's explain (1) foreground removal: In order to obtain the background image in the 2D image to be reconstructed, all foreground objects in the 2D image to be reconstructed need to be removed. In some embodiments, the 2D image to be reconstructed can be input into a Vision Language Model (VLM) for iterative object removal to obtain a complete background image with all foreground objects and their shadows removed. Iterative object removal refers to identifying and erasing one or more foreground objects (such as people, furniture, etc.) from the 2D image to be reconstructed through multiple loops and step-by-step processes. Iterative object removal removes only one target object at a time, and fills in the missing areas after removal, for example, by intelligently filling with the surrounding background texture, so as to finally obtain a pure background image without any specified foreground objects. After each removal and repair, the VLM performs the next round of detection and removal based on the updated image until all foreground objects are cleared, and then outputs the background image in the 2D image to be reconstructed.

[0046] Furthermore, a background point cloud with color information is estimated based on the background image in the 2D image to be reconstructed. Then, post-processing steps such as denoising, vertex normal estimation, Poisson reconstruction and plane smoothing are performed. A physically based rendering (PBR) material map is generated for the post-processed model to obtain a 3D background model with color attributes. The PBR material map includes, but is not limited to, metallicity, roughness and bump mapping.

[0047] In step 220, semantic segmentation is performed on the two-dimensional image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and a mask of the corresponding foreground object is generated based on the at least partial semantic segmentation image.

[0048] In some embodiments, such as Figure 4 As shown, in branch (2), semantic segmentation is performed on the 2D image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object. Semantic segmentation aims to classify each pixel in the 2D image to be reconstructed, assigning it to a predefined semantic category. For example, people, sofas, walls, and floors belong to different semantic categories, thereby achieving pixel-level scene understanding. Foreground objects refer to object instances in the 2D image to be reconstructed that are located in front of the scene and are usually the main focus of the user's attention. They are closer to the viewing angle than the background and often have clear semantic categories and independent physical structures, such as... Figure 4 The table, lamp, and chair in the indoor scene shown are all foreground objects.

[0049] Here, at least part of the semantic segmentation image refers to the semantic segmentation result image of the visible region corresponding to each foreground object obtained after semantic segmentation of the 2D image to be reconstructed. Specifically, since the foreground object may be occluded by other objects in the 2D image to be reconstructed, for example... Figure 4 The table is partially obscured by the chair, so its segmentation only covers the unobscured area, not the complete outline; while Figure 4 In this example, the chair is not occluded by any other foreground objects in the two-dimensional plane; therefore, the chair, as a foreground object, can be fully semantically segmented. That is, the phrase "at least partially" in this embodiment emphasizes that the segmentation result for each foreground object may be complete or incomplete, but regardless, the semantically segmented image contains at least a portion of the complete segmentation result. For example... Figure 4 In the image, the semantic segmentation of the table is incomplete due to the obstruction caused by the chair.

[0050] Furthermore, a mask for the corresponding foreground object needs to be generated based on at least a portion of the semantic segmentation image. The mask is a binary image of the same size as the corresponding foreground object, typically composed of 0s and 1s, used to precisely mark the position and shape of the corresponding foreground object in the 2D image to be reconstructed. In this embodiment, the mask is generated from the 2D image to be reconstructed and at least a portion of the semantic segmentation image; pixels with a value of 1 represent regions belonging to the foreground object, and pixels with a value of 0 represent background or other objects. The mask corresponding to each foreground object not only separates the foreground object from the 2D image to be reconstructed for independent 3D reconstruction, but also plays a crucial role in the subsequent separation of background point clouds and object point clouds.

[0051] In some embodiments, semantic segmentation of the two-dimensional image to be reconstructed yields at least a partially semantically segmented image for each foreground object, including: (1.1) Obtain multiple preset semantic tags; (1.2) Perform semantic segmentation on the two-dimensional image to be reconstructed based on multiple preset semantic labels to obtain the object bounding box of each foreground object in the two-dimensional image to be reconstructed under different preset semantic labels, and the confidence score corresponding to the object bounding box; (1.3) For each foreground object, select the highest value from multiple confidence scores corresponding to the foreground object as the target confidence score, and determine at least part of the semantic segmentation image of the corresponding foreground object based on the target confidence score and the bounding box of the object corresponding to the target confidence score.

[0052] In some embodiments, multiple preset semantic tags are first obtained. Preset semantic tags refer to a set of predefined text descriptors. Preset semantic tags specify the specific types of objects to be searched and identified for the semantic segmentation task. They can be customized according to the actual situation. For example, preset semantic tags include, but are not limited to, chair, lamp, book, etc.

[0053] Furthermore, the two-dimensional image to be reconstructed is based on multiple preset semantic labels. Perform foreground object detection, which involves establishing features of the two-dimensional image to be reconstructed and predefined semantic labels. The relationships between these elements identify candidate foreground objects. Specifically, the final output generates the scene... Each foreground object corresponds to a target confidence score. Target confidence score corresponding bounding box Target confidence score Corresponding category tags , represented as: .in, Indicates the current foreground object index. Indicates the current foreground object and its category. The degree of matching. Foreground object detection can be achieved by inputting the two-dimensional image to be reconstructed into a language-guided open-set object detection model (Marrying DINO with Grounded Pre-training for Open-Set Object Detection, Grounding DINO). Of course, this is only an example for illustration. In actual applications, the model used to detect foreground objects can be selected according to specific circumstances. This application embodiment does not impose any restrictions on this.

[0054] Furthermore, for each foreground object, a corresponding bounding box is generated. The defined high-confidence candidate regions are then finely optimized at the pixel level to generate at least a partial semantic segmentation image for each foreground object. and mask , represented as: .

[0055] It is understood that this embodiment of the application effectively resolves potential ambiguities in object classification by generating multiple initial semantic segmentation images with confidence scores for the same object and selecting the one with the highest confidence score as the final result, thus significantly improving the accuracy and reliability of segmentation. Ultimately, this process can output clear and explicit foreground object segmentation information for subsequent 3D reconstruction tasks, laying a solid foundation for generating high-quality, high-fidelity 3D scenes.

[0056] In step 230, at least a portion of the semantic segmentation image is repaired to obtain a complete semantic segmentation image of the corresponding foreground object, and a 3D model of the corresponding foreground object is generated based on the complete semantic segmentation image.

[0057] In some embodiments, since foreground objects in the two-dimensional image to be reconstructed may occlude each other, the semantic segmentation image generated for each foreground object may be incomplete. If the three-dimensional model of the corresponding foreground object is directly generated using at least a portion of the semantic segmentation image generated in step 220, its quality will be degraded. Based on this, the embodiments of this application introduce a segmentation image inpainting operation before generating the three-dimensional model of each foreground object.

[0058] In some embodiments, the two-dimensional image to be reconstructed is input into a VLM (Virtual Model). Leveraging the VLM's superior semantic understanding capabilities, text prompts are used to visually and accurately locate the defective regions of each foreground object, generating a repaired image that effectively maintains the structural integrity of the foreground object. The model used to perform the repair operation can also be GPT-4o or other models, which can be set according to actual conditions; this application embodiment does not impose any limitations on this.

[0059] Furthermore, the set of at least some semantically segmented images corresponding to all repaired foreground objects is... The input is fed into a pre-trained generative model, which maps each 2D image to an independent 3D model. Each foreground object's 3D model possesses accurate geometry and textures highly consistent with the 2D image to be reconstructed, along with PBR materials, thus exhibiting high visual realism and rendering quality. The generative model is trained based on sample 2D images to be reconstructed. The specific type of generative model can be selected according to actual circumstances, and this application embodiment does not impose any limitations on it.

[0060] In step 240, the three-dimensional point cloud of the two-dimensional image to be reconstructed is determined, and the background point cloud and the object point cloud of each foreground object are determined in the three-dimensional point cloud according to the mask of each foreground object.

[0061] In some embodiments, such as Figure 4As shown, in branch (3), in order to obtain accurate three-dimensional spatial layout information of the two-dimensional image to be reconstructed, the device needs to determine the overall three-dimensional point cloud of the two-dimensional image to be reconstructed. The three-dimensional point cloud refers to a set of a large number of three-dimensional spatial coordinate points, which can collectively describe the geometric shape of the entire scene (including each foreground object and background). This can usually be achieved by combining a deep learning-based monocular depth estimation algorithm with camera intrinsic parameter estimation.

[0062] Furthermore, after obtaining the three-dimensional point cloud of the scene represented by the two-dimensional image to be reconstructed, the mask generated for each foreground object in step 220 is mapped to the three-dimensional space, thereby accurately separating the background point cloud belonging to the background region and the object point cloud of each foreground object in the three-dimensional point cloud. The separated point clouds retain their relative position and pose information in the original scene.

[0063] In some embodiments, determining the three-dimensional point cloud of the two-dimensional image to be reconstructed includes: (2.1) Perform geometric transformation on the two-dimensional image to be reconstructed to obtain the pseudo-visual two-dimensional image corresponding to the two-dimensional image to be reconstructed; (2.2) Based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image, determine the depth image corresponding to the two-dimensional image to be reconstructed; (2.3) Obtain the camera parameters corresponding to the captured two-dimensional image to be reconstructed, and determine the three-dimensional point cloud of the two-dimensional image to be reconstructed based on the camera parameters and the depth image.

[0064] In some embodiments, to infer 3D information from a single image, a geometric transformation is first performed on the 2D image to be reconstructed to obtain a pseudo-visual 2D image corresponding to the 2D image to be reconstructed. The pseudo-visual 2D image does not rely on real camera parameters but is generated through preset, simplified geometric transformations such as horizontal translation. Thus, by simulating binocular visual input pairs, an input pair consisting of the original 2D image to be reconstructed and its copy is constructed to adapt to depth estimation models processing binocular or multi-view inputs, cleverly solving the limitation that the input source has only a single viewpoint.

[0065] Furthermore, based on the 2D image to be reconstructed and its corresponding pseudo-visual 2D image, a depth image corresponding to the 2D image to be reconstructed is jointly determined. Specifically, the image set can be input into a pre-trained deep learning model. This model analyzes the feature correspondence between the two images and uses its learned 3D geometric prior knowledge to infer the distance of each pixel in the scene relative to the camera. The inference results ultimately constitute the depth image.

[0066] Furthermore, after obtaining the depth image corresponding to the 2D image to be reconstructed, the camera parameters corresponding to the captured 2D image are acquired, and a 3D point cloud corresponding to the 2D image to be reconstructed is generated based on the camera parameters and the depth image. Camera parameters refer to intrinsic camera parameters such as focal length and principal point coordinates, which can be output by the depth estimation model or obtained through other calibration methods. The device combines the camera parameters and the depth image to determine the 3D point cloud of the 2D image to be reconstructed. Specifically, for each pixel in the depth image, its depth value and 2D coordinates on the image are used to calculate a coordinate point in 3D space through inverse projection of the camera model. All these 3D coordinate points are then aggregated to obtain a 3D point cloud describing the geometry of the entire scene.

[0067] It is understood that the embodiments of this application overcome the inherent problem of insufficient information in a single view by constructing a pseudo-visual two-dimensional image, and finally generate a dense three-dimensional point cloud that can accurately reflect the spatial structure of the original scene. The three-dimensional point cloud will serve as the geometric benchmark for object segmentation and layout optimization in subsequent steps, providing crucial basic data support to ensure that the final generated three-dimensional scene maintains a high degree of consistency with the original image in terms of spatial layout.

[0068] The depth image in step (2.2) can be obtained through the following specific implementation method: In some embodiments, determining the depth image corresponding to the two-dimensional image to be reconstructed based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image includes: (2.2.1) Extract the features of the two-dimensional image to be reconstructed from the features of the two-dimensional image to be reconstructed, and extract the features of the pseudo-visual two-dimensional image from the features of the pseudo-visual two-dimensional image; (2.2.2) Input the features of the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image features into the pre-trained cross-attention model to obtain the enhanced features of the two-dimensional image to be reconstructed and the enhanced pseudo-visual two-dimensional image features. The pre-trained cross-attention model is trained from the sample two-dimensional image to be reconstructed and the sample two-dimensional image features. (2.2.3) Determine the depth image corresponding to the two-dimensional image to be reconstructed based on the enhanced features of the two-dimensional image to be reconstructed and the enhanced pseudo-visual features of the two-dimensional image to be reconstructed.

[0069] In some embodiments, the input image data is converted into a format more suitable for machine learning model processing. Features of the 2D image to be reconstructed are extracted from the original 2D image to be reconstructed. Simultaneously, the same feature extraction operation is performed on the pseudo-visual 2D image used as auxiliary input to obtain pseudo-visual 2D image features. Feature extraction is typically performed by the encoder part of a deep neural network, which transforms the raw pixel information of the image into a set of high-dimensional feature vectors. The features of the 2D image to be reconstructed and the pseudo-visual 2D image features are abstract representations of the corresponding image content, capable of capturing deep visual information such as edges, textures, and shapes that are crucial for understanding scene geometry.

[0070] Furthermore, the features of the 2D image to be reconstructed and the pseudo-visual 2D image features are fed into the core processing unit, namely the pre-trained cross-attention model. After processing by this model, enhanced features of the 2D image to be reconstructed and enhanced pseudo-visual 2D image features are obtained. The pre-trained cross-attention model is a neural network module pre-trained with a large amount of data. The cross-attention mechanism of the cross-attention model enables information interaction and mutual reference between the features of the 2D image to be reconstructed and the pseudo-visual 2D image features to align their geometric representations. Finally, it outputs enhanced features of the 2D image to be reconstructed and enhanced pseudo-visual 2D image features. The enhanced features are obtained by fusing geometric consistency information brought about by cross-view reasoning on the basis of the original features. The pre-trained cross-attention model is trained from sample 2D images to be reconstructed and sample features of the 2D images to be reconstructed.

[0071] Furthermore, by utilizing the enhanced features of the 2D image to be reconstructed and the enhanced pseudo-visual 2D image features, the depth image corresponding to the 2D image to be reconstructed is determined. This process is usually completed by a decoder network. The decoder network receives the enhanced features, which are rich in geometric information and have undergone cross-attention enhancement, as input and maps them back to the 2D image space to obtain the depth image. Unlike ordinary images, each pixel value in the depth image does not represent color, but directly encodes the distance or depth from each pixel in the original 2D image to be reconstructed to the camera.

[0072] It is understood that the embodiments of this application utilize a pre-trained cross-attention model to establish an information interaction channel between feature streams originating from the same 2D image to be reconstructed, forcing the model to perform self-comparison and geometric reasoning. The cross-attention mechanism greatly enhances the model's ability to perceive 3D structures from a single view, overcomes the problem of inaccurate or unstable depth estimation caused by insufficient information in traditional methods, and ultimately generates corresponding high-quality depth images, which provide a data foundation for subsequent 3D point cloud generation.

[0073] In step 250, background layout parameters are determined based on the background point cloud and background 3D model, and corresponding object layout parameters are determined based on the object point cloud and object 3D model of each foreground object.

[0074] In some embodiments, after completing the 3D reconstruction of the background and each foreground object, the next step is to achieve precise spatial alignment of the 3D assets. This process consists of two parts: First, the background 3D model generated in step 210 is aligned with the background point cloud extracted in step 240 to solve for a set of background layout parameters that describe the optimal position, pose, and scale of the background 3D model in 3D space; on the other hand, a similar method is used for each foreground object, matching its object 3D model generated in step 230 with the corresponding object point cloud extracted in step 240 to calculate the corresponding object layout parameters for each foreground object.

[0075] The background layout parameters and the layout parameters of each object include transformation parameters such as translation, rotation, and scaling. Together, the background layout parameters and the layout parameters of each object define the final state of the target 3D model in the 3D scene.

[0076] In some embodiments, determining background layout parameters based on background point clouds and background 3D models includes: (3.1) Obtain the initial background layout parameters and sample the background 3D model to obtain the sampled background point cloud corresponding to the background 3D model; (3.2) Calculate the background difference between the background point cloud and the sampled background point cloud, and update the initial background layout parameters according to the background difference to obtain the updated background layout parameters; (3.3) Adjust the background point cloud and sampled background point cloud according to the updated background layout parameters to obtain the updated background point cloud and the updated sampled background point cloud; (3.4) The updated background point cloud is used as the new background point cloud, the updated sampled background point cloud is used as the new sampled background point cloud, and the updated background layout parameters are used as the new initial background layout parameters. (3.5) Return to the step of calculating the background difference between the background point cloud and the sampled background point cloud until the first preset condition is reached, and use the updated background layout parameters obtained from the last update as the background layout parameters.

[0077] In some embodiments, initial background layout parameters corresponding to the background are first obtained, which describe an initial guess or default value for the position, pose, and scaling of the background 3D model in 3D space. Simultaneously, in order to compare with point cloud data extracted from the original 2D image to be reconstructed, the device also samples the background 3D model to obtain a sampled background point cloud. Figure 4(Not shown in the image), sampling refers to selecting a large number of discrete points on the surface of the background 3D model to generate a point set that represents its geometry, i.e., the sampled background point cloud corresponding to the background 3D model. The initial background layout parameters include a set of learnable parameters in 3D space. ={T, R, S}, where T represents the background translation parameter, R represents the background rotation parameter, and S represents the background scaling parameter.

[0078] Furthermore, the background difference is calculated between the background point cloud, which serves as the geometric truth, and the sampled background point cloud, representing the current model state. This background difference is used to assess the degree of mismatch between the two point cloud sets in 3D space, and algorithms such as Chamfer Distance are commonly used to determine this difference. Chamfer Distance is a commonly used metric in computer vision, graphics, and machine learning. It is used to calculate the degree of mismatch between two point sets. The basic idea of ​​Chamfer Distance is to compare the distances from each point in one set to the nearest point in the other set, and to use the sum of these distances as a measure of the difference between the two point sets.

[0079] Furthermore, the initial background layout parameters are updated based on the background differences. This update process is usually driven by optimization algorithms such as gradient descent, with the goal of adjusting the layout parameters to reduce the background differences, thereby obtaining a better set of updated background layout parameters.

[0080] Furthermore, the updated background layout parameters are applied to geometric alignment. Specifically, the device adjusts the sampled background point cloud according to the updated background layout parameters, that is, it performs translation, rotation, and scaling transformations to make it spatially closer to the background point cloud, thereby obtaining an updated sampled background point cloud with a better position after transformation. At the same time, the background point cloud, which is the alignment target, remains unchanged, but conceptually, it and the updated sampled point cloud together form the basis for a new round of comparison. The background point cloud in this state can also be called the updated background point cloud.

[0081] Furthermore, to achieve iterative optimization, the updated background point cloud obtained in the previous step is used as the new background point cloud at the start of the next iteration, the updated sampled background point cloud is used as the new sampled background point cloud, and the calculated updated background layout parameters are used as the new initial background layout parameters at the start of the next iteration. In this way, the optimization results are continuously accumulated and propagated, enabling the background layout parameters to generate a better solution in each iteration.

[0082] Furthermore, in the new iterative data processing, the device returns to the step of calculating the background difference between the background point cloud and the sampled background point cloud, starting a new round of calculation, updating, and adjustment. This iterative process continues until a first preset condition is met, at which point the updated background layout parameters obtained from the last update are used as the background layout parameters. The first preset condition can be reaching a preset maximum number of iterations, reaching a preset total iteration time, or the calculated background difference being less than a sufficiently small threshold, etc. Once the first preset condition is met, the iteration stops, and the updated background layout parameters obtained from the last update are output as the final background layout parameters.

[0083] It is understood that the embodiments of this application gradually and meticulously align the generated 3D model with the real spatial position extracted from the original image by repeatedly calculating geometric differences and adjusting parameters accordingly. This effectively overcomes the problem of inaccurate initial guesses or local optima that may be encountered during the optimization process, ensuring that the finally determined background layout parameters can achieve the most accurate matching between the background 3D model and the background geometry in the original scene. This lays the foundation for the subsequent construction of a 3D scene with accurate and highly consistent overall spatial relationships.

[0084] In some embodiments, the background difference between the background point cloud and the sampled background point cloud can be measured by the three-dimensional spatial difference. Specifically, only the background three-dimensional spatial difference between the background point cloud and the sampled background point cloud is calculated. The background three-dimensional spatial difference is used to measure the geometric deviation in the three-dimensional coordinate system between the background point cloud representing the geometry of the real scene and the sampled background point cloud representing the current model pose. Subsequently, the background three-dimensional spatial difference is directly taken as the overall background difference, and the layout parameters are updated based on this.

[0085] Projection refers to the process of mapping points in three-dimensional space onto a two-dimensional image plane using known camera parameters. The first and second projected background images can be understood as the outlines or two-dimensional representations of two point clouds observed from the original camera viewpoint.

[0086] In some embodiments, the background difference between the background point cloud and the sampled background point cloud can be measured by a two-dimensional spatial difference. Specifically, the background point cloud is first projected to obtain a first projected background image, and the sampled background point cloud is projected to obtain a second projected background image. Then, the background three-dimensional spatial difference and the background two-dimensional image difference are calculated, and the background two-dimensional image difference is used as the background difference between the background point cloud and the sampled background point cloud. The background two-dimensional image difference is used to measure the degree of mismatch between the two two-dimensional projections on the image plane, such as calculating their pixel overlap or contour distance.

[0087] In some embodiments, in each round of iterative data processing, the background three-dimensional spatial difference and the background two-dimensional image difference between the background point cloud and the sampled background point cloud are calculated, and in each round of iterative data processing, the background difference between the background point cloud and the sampled background point cloud is determined based on the background three-dimensional spatial difference and the background two-dimensional image difference.

[0088] In some embodiments, the background difference between the background point cloud and the sampled background point cloud can be calculated in stages according to the different number of iterations of data processing, including: (3.2.1) When the number of updates to the initial background layout parameters is less than the preset threshold for background updates, calculate the background three-dimensional spatial difference between the background point cloud and the sampled background point cloud, and determine the background difference between the background point cloud and the sampled background point cloud based on the background three-dimensional spatial difference. (3.2.2) When the number of updates to the initial background layout parameters is equal to or greater than the preset threshold for background updates, the background point cloud is projected to obtain the first projected background image, and the sampled background point cloud is projected to obtain the second projected background image. (3.2.3) Calculate the background two-dimensional image difference between the first projected background image and the second projected background image, and determine the background difference between the background point cloud and the sampled background point cloud based on the background three-dimensional spatial difference and the background two-dimensional image difference.

[0089] In some embodiments, when the number of updates to the initial background layout parameters is less than a preset background update threshold, the device only calculates the background 3D spatial difference between the background point cloud and the sampled background point cloud, and determines the background difference between the background point cloud and the sampled background point cloud at this stage. The preset background update threshold is a preset number of iterations or optimization steps.

[0090] Furthermore, when the optimization process enters the later stage, i.e., when the number of updates to the initial background layout parameters is equal to or greater than the preset threshold for background updates, the device introduces two-dimensional spatial constraints for more refined adjustments. Specifically, the difference between the two-dimensional background image is determined based on the first projected background image obtained from the projection of the background point cloud and the second projected background image obtained from the projection of the sampled background point cloud. In the later stage, the difference between the three-dimensional background space and the difference between the two-dimensional background image are combined, and the final background difference used to guide the update of the background layout parameters is determined through weighted summation and other methods.

[0091] It is understood that the embodiments of this application adopt a phased and cross-dimensional hybrid optimization strategy. In the early stage of optimization, relying solely on the differences in the three-dimensional space of the background can achieve fast and efficient coarse alignment, avoiding unnecessary computational overhead. In the later stage of optimization, by introducing the differences in the two-dimensional background image as an additional constraint, the ambiguity problem that may arise from relying solely on three-dimensional alignment is effectively solved. Ultimately, it is ensured that the generated three-dimensional background model not only matches the geometric height of the target in three-dimensional space, but also maintains strict visual consistency with the input image in its projection under the original camera view, thereby achieving more robust and accurate scene layout restoration.

[0092] In step 260, the background 3D model and the 3D model of each object are combined according to the background layout parameters and the object layout parameters corresponding to each foreground object to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0093] In some embodiments, such as Figure 4 As shown, after the background layout parameters and the object layout parameters corresponding to each foreground object are determined, the corresponding 3D models are spatially transformed using these parameters to obtain the target 3D model of the scene corresponding to the combined 2D image to be reconstructed. Specifically, the device applies the background layout parameters to the background 3D model to place it in the correct position in the scene; simultaneously, it applies each object layout parameter to its corresponding object 3D model to restore their precise position, orientation, and size in the original scene. By combining the transformed background 3D model and all object 3D models in the same 3D coordinate system, a target 3D model that is highly consistent with the original scene depicted by the 2D image to be reconstructed in terms of spatial layout, geometry, and appearance is finally constructed.

[0094] It is understood that this application embodiment ensures high-quality generation of the 3D model of each independent foreground object by employing a segmentation-then-repair strategy; and by extracting and separating the overall point cloud from the original 2D image to be reconstructed, it preserves the precise relative spatial relationships between each foreground object; then, by aligning and optimizing the background and the independently generated 3D models of each foreground object using the point cloud, it achieves accurate restoration of the original scene layout, ultimately generating a complete 3D scene that includes foreground and background, is consistent across multiple perspectives, has reasonable spatial relationships, and whose assets can be independently edited. Thus, this application embodiment, by decoupling the 3D model generation process of the background and each foreground object in a single image, effectively overcomes the problems of geometric distortion, loss of detail, and layout disorder caused by traditional methods when dealing with scenes occluded by multiple objects, significantly improving the quality of 3D content generated from a single image.

[0095] In some embodiments, the corresponding object layout parameters are determined based on the object point cloud and the object 3D model of each foreground object, including: (4.1) Obtain the initial object layout parameters for each foreground object, and sample the 3D model of each foreground object to obtain the sampled object point cloud for each foreground object; (4.2) For each foreground object, calculate the object difference between the object point cloud and the sampled object point cloud, update the initial object layout parameters according to the object difference, and obtain the updated object layout parameters; (4.3) Adjust the object point cloud and sampled object point cloud according to the updated object layout parameters to obtain the updated object point cloud and the updated sampled object point cloud; (4.4) The updated object point cloud is used as the new object point cloud, the updated sampled object point cloud is used as the new sampled object point cloud, and the updated object layout parameters are used as the new initial object layout parameters; (4.5) Return to the step of performing the calculation of object differences between object point cloud and sampled object point cloud until the second preset condition is reached, and use the updated object layout parameters obtained from the last update as the object layout parameters of the corresponding foreground object.

[0096] Similar to steps (3.1) to (3.5), in some embodiments, for each foreground object, its corresponding initial object layout parameters are continuously adjusted until its corresponding object layout parameters are obtained. The initial object layout parameters represent a preliminary estimate of the ideal position, pose, and size of the corresponding foreground object in three-dimensional space. To enable geometric comparison with the point cloud extracted from the original image, the three-dimensional model of each foreground object is sampled to obtain a sampled object point cloud. The initial object layout parameters also include a learnable set of parameters in three-dimensional space. ={T, R, S}, where T represents the object translation parameter, R represents the object rotation parameter, and S represents the object scaling parameter.

[0097] Furthermore, for each foreground object, an independent optimization loop is executed. Specifically, the object difference is calculated between the object point cloud, which serves as the geometric truth, and the sampled object point cloud, representing the current model pose, to assess the degree of geometric mismatch between the two point cloud sets in 3D space. Algorithms such as chamfer distance are commonly used for this calculation. Subsequently, the initial object layout parameters of the foreground object are updated based on the object difference. This update process is driven by optimization algorithms such as gradient descent, aiming to adjust the layout parameters to reduce the object difference, thereby obtaining a better set of updated object layout parameters.

[0098] Furthermore, the calculated updated object layout parameters are applied to the alignment of geometric entities. Specifically, the sampled object point cloud is adjusted according to these updated object layout parameters, i.e., it is subjected to corresponding translation, rotation, and scaling transformations to make its pose in three-dimensional space closer to the object point cloud, thus obtaining an updated sampled object point cloud with better positioning after transformation. At the same time, the original object point cloud, which serves as the alignment target, remains unchanged, but conceptually, it and the updated sampled point cloud together form the basis for a new round of comparison. The object point cloud in this state can also be called the updated object point cloud.

[0099] Furthermore, to achieve continuous iterative optimization, the device needs to prepare the updated state for the next iteration. This step is a state update operation where the system designates the updated object point cloud from the previous step as the new object point cloud for the start of the next iteration, and designates the updated sampled object point cloud as the new sampled object point cloud. Crucially, the system uses the just-calculated updated object layout parameters as the new initial object layout parameters for the start of the next iteration. In this way, the results of each optimization step are accumulated and propagated, ensuring that the layout parameters steadily move towards the optimal solution in each iteration.

[0100] Furthermore, to achieve iterative optimization, the updated object point cloud obtained in the previous step is used as the new object point cloud at the start of the next iteration, the updated sampled object point cloud is used as the new sampled object point cloud, and the calculated updated object layout parameters are used as the new initial object layout parameters at the start of the next iteration. In this way, the optimization results are continuously accumulated and passed on, enabling the object layout parameters to generate a better solution in each loop.

[0101] Furthermore, in the new iterative data processing, the device returns to the step of calculating the object difference between the object point cloud and the sampled object point cloud, starting a new round of calculation, updating, and adjustment. This iterative process continues until a second preset condition is reached, at which point the updated object layout parameters obtained from the last update are used as the object layout parameters. The second preset condition can be reaching a preset maximum number of iterations, reaching a preset total iteration time, or the calculated object difference being less than a sufficiently small threshold, etc. Once the second preset condition is met, the iteration stops, and the updated object layout parameters obtained from the last update are output as the final object layout parameters.

[0102] It is understood that, by iteratively aligning the geometric ground truth point cloud with the model sampled point cloud for each object, the independent position, pose, and scaling of each foreground object in the original scene are recovered, effectively avoiding positioning errors caused by mutual influence in multi-object scenes. This strategy of optimizing one by one ensures that in the final combined 3D scene, all foreground objects can be accurately placed in their proper spatial positions, thereby achieving a highly faithful reproduction of the original complex scene layout.

[0103] In some embodiments, calculating the object differences between the object point cloud and the sampled object point cloud includes: (4.2.1) When the number of updates to the initial object layout parameters is less than the preset threshold for object updates, calculate the object three-dimensional space difference between the object point cloud and the sampled object point cloud, and determine the object difference between the object point cloud and the sampled object point cloud based on the object three-dimensional space difference. (4.2.2) When the number of updates to the initial object layout parameters is equal to or greater than the preset threshold for object updates, the object point cloud is projected to obtain the first projected object image, and the sampled object point cloud is projected to obtain the second projected object image. (4.2.3) Calculate the object two-dimensional image difference between the first projected object image and the second projected object image, and determine the object difference between the object point cloud and the sampled object point cloud based on the object three-dimensional spatial difference and the object two-dimensional image difference.

[0104] In some embodiments, the definition of the object update preset threshold is similar to that of the background update preset threshold; the definition of the object's three-dimensional spatial difference is similar to that of the background's three-dimensional spatial difference; the definition of the object's difference is similar to that of the background difference; the definition of the first projected object image is similar to that of the first projected background image; the definition of the second projected object image is similar to that of the second projected background image; and the definition of the object's two-dimensional spatial difference is similar to that of the background's two-dimensional spatial difference. That is, the specific implementation of steps (4.2.1) to (4.2.3) is similar to that of steps (3.2.1) to (3.2.3); and the object difference can also be determined solely by the object's three-dimensional spatial difference, or solely by the object's two-dimensional spatial difference, or simultaneously by both the object's three-dimensional spatial difference and the object's two-dimensional spatial difference, or determined in stages by both the object's three-dimensional spatial difference and the object's two-dimensional spatial difference, which will not be elaborated here.

[0105] For example, for each foreground object, the bidirectional chamfer distance is used as the core optimization objective, i.e., the sampling object point cloud. and object 3D point cloud 3D spatial differences between objects in 3D space (3D chamfer distance loss) However, while depth priors help capture the relative spatial relationships between objects, relying solely on 3D constraints often fails to achieve stable parameter convergence, potentially leading to errors in object translation and rotation estimation. To address this issue, embodiments of this application introduce a 2D projection constraint mechanism, utilizing camera parameters... Will and Project onto a 2D plane to obtain a set of projected points, and minimize the 3D spatial differences of the object (2D chamfer distance loss). This method forces two point sets to match through 2D projection constraints to learn 3D structure, establishing cross-dimensional geometric consistency. The final loss function formula for determining object layout parameters is defined as follows: .in, and These are the weighting coefficients that balance different monitoring signals, for example, Set to 1, Set it to 0.05; This represents a projection operation based on the camera projection matrix.

[0106] like Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the determination of object layout parameters provided in an embodiment of this application. Specifically, in the layout parameter determination stage, 20 optimization cycles are performed for each foreground object, with each cycle containing 2,000 iterations. In the first 1,200 iterations, only the 3D chamfer distance loss is optimized to focus on the spatial alignment of the point cloud. In the subsequent 800 iterations, a 2D projection chamfer distance loss is introduced to enforce image plane projection alignment constraints. The iterative optimization process is as follows: Figure 5 As shown, the iterative optimization process used the Adam optimizer, with a learning rate of 0.01 for the translation, rotation, and scaling parameters.

[0107] To help readers better understand the beneficial effects of the embodiments of this application, the following is combined with... Figure 6 To explain, Figure 6 This is a schematic diagram illustrating the effect of determining the 3D model of an object provided in an embodiment of this application. Since the focus of this embodiment is on reconstructing multiple foreground objects with occlusion relationships, therefore... Figure 6 In the effect illustration, background reconstruction is ignored, and only the 3D reconstruction effect of the foreground object is shown. Specifically: (1) Obtain the two-dimensional image to be reconstructed; (2) Generate a binary mask of the foreground object after performing target detection on the two-dimensional image to be reconstructed; (3) Next, extract at least a partial semantic segmentation image of each foreground object from the mask. It can be seen that due to the existence of occlusion, the semantic segmentation images of some foreground objects are not complete, such as Figure 6 The table in the middle where the lamp is placed; (4) Next, at least part of the semantic segmentation image is repaired to obtain the complete semantic segmentation image of the corresponding foreground object; (5) Generate a 3D model of the corresponding foreground object based on the complete semantic segmentation image; (6) Since the background portion is not considered in this example, the point cloud of each foreground object is used to calculate the object point cloud. Figure 6 (Not shown in the image) and the corresponding object layout parameters are determined based on the object's 3D model. Then, each object's 3D model is combined according to the object layout parameters corresponding to each foreground object, ultimately resulting in a scene 3D model whose spatial position and size proportions match the original scene. Figure 6 The diagram illustrates three different perspectives of the scene's 3D model. It can be observed that there are no modeling errors such as missing models in the 3D models of each object in the scene's 3D model, and the positional relationships between multiple object 3D models are correct. Furthermore, the combination of this scene 3D model and the corresponding background 3D model will yield the target 3D model of the scene corresponding to the 2D image to be reconstructed.

[0108] Furthermore, the embodiments of this application are compared with the target 3D model generated by conventional methods, such as... Figure 7 As shown, Figure 7 This diagram illustrates a comparison between the 3D model generation method provided in this application and 3D models generated by traditional methods. The same 2D image to be reconstructed is reconstructed using the Hunyuan3D 2.5 model, the Multi-Instance Diffusion for 3D Reconstruction (MIDI) model, and the Scene Generation Model (SceneGen) model, respectively. The diagram emphasizes the key reconstruction aspects of this application's embodiments. Figure 7 The diagram only illustrates the 3D reconstruction results from different perspectives for multiple foreground objects. It can be observed that the table model reconstructed by Hunyuan3D 2.5 has incorrect color and is missing a chair; the chair model reconstructed by MIDI has incorrect structure and color; and the table model reconstructed by SceneGen has incorrect color and exhibits clipping errors between different chairs. In contrast, the embodiment of this application correctly reconstructs the 2D image to be reconstructed and effectively maintains spatial rationality and multi-view functionality when handling complex object interactions and occlusions. Figure 1 This approach achieves consistency and overcomes the limitations of existing methods in areas such as insufficient asset generation quality and weak modeling of spatial relationships between objects.

[0109] like Figure 8 As shown, Figure 8This is a schematic diagram of the module structure of the three-dimensional model generation device provided in this application embodiment. The three-dimensional model generation device 300 may include the following modules 310 to 360: The acquisition module 310 is used to acquire the two-dimensional image to be reconstructed and to perform three-dimensional reconstruction on the background image in the two-dimensional image to be reconstructed to generate a three-dimensional background model. The semantic segmentation module 320 is used to perform semantic segmentation on the two-dimensional image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and to generate a mask of the corresponding foreground object based on the at least partial semantic segmentation image; Repair module 330 is used to repair at least part of the semantic segmentation image to obtain the complete semantic segmentation image of the corresponding foreground object, and generate the corresponding three-dimensional object model of the foreground object based on the complete semantic segmentation image; The point cloud determination module 340 is used to determine the three-dimensional point cloud of the two-dimensional image to be reconstructed, and to determine the background point cloud and the object point cloud of each foreground object in the three-dimensional point cloud according to the mask of each foreground object. The layout parameter determination module 350 is used to determine the background layout parameters based on the background point cloud and the background 3D model, and to determine the corresponding object layout parameters based on the object point cloud and the object 3D model of each foreground object. The target 3D model generation module 360 ​​is used to combine the background 3D model and the 3D model of each object according to the background layout parameters and the object layout parameters corresponding to each foreground object to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0110] In some embodiments, the semantic segmentation module 320 is used for: Retrieve multiple preset semantic tags; Semantic segmentation is performed on the 2D image to be reconstructed based on multiple preset semantic labels to obtain the object bounding box of each foreground object in the 2D image to be reconstructed under different preset semantic labels, and the confidence score of the object bounding box; For each foreground object, the highest confidence score is selected from multiple confidence scores corresponding to the foreground object as the target confidence score. Based on the target confidence score and the bounding box of the object corresponding to the target confidence score, at least a portion of the semantic segmentation image of the foreground object is determined.

[0111] In some embodiments, the point cloud determination module 340 is used for: Perform geometric transformations on the two-dimensional image to be reconstructed to obtain a pseudo-visual two-dimensional image corresponding to the two-dimensional image to be reconstructed; Based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image, determine the depth image corresponding to the two-dimensional image to be reconstructed; Obtain the camera parameters corresponding to the captured 2D image to be reconstructed, and determine the 3D point cloud of the 2D image to be reconstructed based on the camera parameters and the depth image.

[0112] In some embodiments, the point cloud determination module 340 is further configured to: Features of the two-dimensional image to be reconstructed are extracted from the features of the two-dimensional image to be reconstructed, and features of the pseudo-visual two-dimensional image are extracted from the features of the pseudo-visual two-dimensional image. The features of the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image features are input into a pre-trained cross-attention model to obtain enhanced features of the two-dimensional image to be reconstructed and enhanced pseudo-visual two-dimensional image features. The pre-trained cross-attention model is trained from the sample two-dimensional image to be reconstructed and the sample two-dimensional image features. Based on the enhanced features of the two-dimensional image to be reconstructed and the enhanced pseudo-visual features of the two-dimensional image, the depth image corresponding to the two-dimensional image to be reconstructed is determined.

[0113] In some embodiments, the layout parameter determination module 350 is used for: Obtain the initial background layout parameters and sample the background 3D model to obtain the sampled background point cloud corresponding to the background 3D model; Calculate the background difference between the background point cloud and the sampled background point cloud, update the initial background layout parameters based on the background difference, and obtain the updated background layout parameters; Adjust the background point cloud and sampled background point cloud according to the updated background layout parameters to obtain the updated background point cloud and the updated sampled background point cloud; The updated background point cloud is used as the new background point cloud, the updated sampled background point cloud is used as the new sampled background point cloud, and the updated background layout parameters are used as the new initial background layout parameters. Return to the step of calculating the background difference between the background point cloud and the sampled background point cloud, until the first preset condition is reached, and use the updated background layout parameters obtained from the last update as the background layout parameters.

[0114] In some embodiments, the layout parameter determination module 350 is further configured to: When the number of updates to the initial background layout parameters is less than the preset threshold for background updates, the background three-dimensional spatial difference between the background point cloud and the sampled background point cloud is calculated, and the background difference between the background point cloud and the sampled background point cloud is determined based on the background three-dimensional spatial difference. When the number of updates to the initial background layout parameters is equal to or greater than the preset threshold for background updates, the background point cloud is projected to obtain the first projected background image, and the sampled background point cloud is projected to obtain the second projected background image. Calculate the background two-dimensional image difference between the first projected background image and the second projected background image, and determine the background difference between the background point cloud and the sampled background point cloud based on the background three-dimensional spatial difference and the background two-dimensional image difference.

[0115] In some embodiments, the layout parameter determination module 350 is further configured to: Obtain the initial object layout parameters for each foreground object, and sample the 3D model of each foreground object to obtain the sampled object point cloud for each foreground object; For each foreground object, calculate the object difference between the object point cloud and the sampled object point cloud, and update the initial object layout parameters based on the object difference to obtain the updated object layout parameters; Adjust the object point cloud and sampled object point cloud according to the updated object layout parameters to obtain the updated object point cloud and the updated sampled object point cloud; The updated object point cloud is used as the new object point cloud, the updated sampled object point cloud is used as the new sampled object point cloud, and the updated object layout parameters are used as the new initial object layout parameters. Return to the step of calculating the object difference between the object point cloud and the sampled object point cloud until the second preset condition is reached, and use the updated object layout parameters obtained from the last update as the object layout parameters of the corresponding foreground object.

[0116] In some embodiments, the layout parameter determination module 350 is further configured to: When the number of updates to the initial object layout parameters is less than the preset threshold for object updates, the object three-dimensional space difference between the object point cloud and the sampled object point cloud is calculated, and the object difference between the object point cloud and the sampled object point cloud is determined based on the object three-dimensional space difference. When the number of updates to the initial object layout parameters is equal to or greater than the preset threshold for object updates, the object point cloud is projected to obtain the first projected object image, and the sampled object point cloud is projected to obtain the second projected object image. Calculate the object 2D image difference between the first projected object image and the second projected object image, and determine the object difference between the object point cloud and the sampled object point cloud based on the object 3D spatial difference and the object 2D image difference.

[0117] The 3D model generation method, apparatus, electronic device, and storage medium proposed in this application acquire a 2D image to be reconstructed, and perform 3D reconstruction on the background image in the 2D image to be reconstructed to generate a background 3D model; perform semantic segmentation on the 2D image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and generate a mask for the corresponding foreground object based on the at least partial semantic segmentation image; repair the at least partial semantic segmentation image to obtain a complete semantic segmentation image of the corresponding foreground object, and generate a 3D object model of the corresponding foreground object based on the complete semantic segmentation image; determine the 3D point cloud of the 2D image to be reconstructed, and determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud based on the mask of each foreground object; determine background layout parameters based on the background point cloud and the background 3D model, and determine corresponding object layout parameters based on the object point cloud and the object 3D model of each foreground object; combine the background 3D model and the 3D model of each object based on the background layout parameters and the object layout parameters corresponding to each foreground object to obtain a target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0118] This embodiment first separates the background and each foreground object in the 2D image to be reconstructed. Then, it restores the complete semantic information of each foreground object by repairing the occluded areas. Subsequently, it independently constructs 3D models of the background image and each repaired foreground object without occlusion, resulting in high-quality 3D models of the background and objects corresponding to each foreground object. In this way, this embodiment can effectively overcome the problem of difficulty in accurately understanding the complete structure and spatial layout of the background and each foreground object due to mutual occlusion between foreground objects when only a single 2D image is available. Next, by determining the point clouds corresponding to the background and each foreground object, the layout parameters corresponding to the background and each foreground object are calculated separately, thereby ensuring that when the target 3D model is finally generated, the spatial position, scale, and occlusion relationship between the background and each foreground object are consistent with the scene represented by the 2D image to be reconstructed. Finally, this embodiment improves the quality of the 3D model generated by 3D reconstruction using a single image.

[0119] like Figure 9 As shown, Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes: The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and called by the processor 401 to execute the image analysis method of the embodiments of this application. Input / output interface 403 is used to implement information input and output; The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0120] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image analysis method.

[0121] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The 3D model generation method, apparatus, electronic device, and storage medium proposed in this application acquire a 2D image to be reconstructed, and perform 3D reconstruction on the background image in the 2D image to be reconstructed to generate a background 3D model; perform semantic segmentation on the 2D image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and generate a mask for the corresponding foreground object based on the at least partial semantic segmentation image; repair the at least partial semantic segmentation image to obtain a complete semantic segmentation image of the corresponding foreground object, and generate a 3D object model of the corresponding foreground object based on the complete semantic segmentation image; determine the 3D point cloud of the 2D image to be reconstructed, and determine the background point cloud and the object point cloud of each foreground object in the 3D point cloud based on the mask of each foreground object; determine background layout parameters based on the background point cloud and the background 3D model, and determine corresponding object layout parameters based on the object point cloud and the object 3D model of each foreground object; combine the background 3D model and the 3D model of each object based on the background layout parameters and the object layout parameters corresponding to each foreground object to obtain a target 3D model in the scene corresponding to the 2D image to be reconstructed.

[0123] This embodiment first separates the background and each foreground object in the 2D image to be reconstructed. Then, it restores the complete semantic information of each foreground object by repairing the occluded areas. Subsequently, it independently constructs 3D models of the background image and each repaired foreground object without occlusion, resulting in high-quality 3D models of the background and objects corresponding to each foreground object. In this way, this embodiment can effectively overcome the problem of difficulty in accurately understanding the complete structure and spatial layout of the background and each foreground object due to mutual occlusion between foreground objects when only a single 2D image is available. Next, by determining the point clouds corresponding to the background and each foreground object, the layout parameters corresponding to the background and each foreground object are calculated separately, thereby ensuring that when the target 3D model is finally generated, the spatial position, scale, and occlusion relationship between the background and each foreground object are consistent with the scene represented by the 2D image to be reconstructed. Finally, this embodiment improves the quality of the 3D model generated by 3D reconstruction using a single image.

[0124] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0125] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0128] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0129] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0131] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating a three-dimensional model, characterized in that, include: A two-dimensional image to be reconstructed is acquired, and a three-dimensional reconstruction is performed on the background image in the two-dimensional image to be reconstructed to generate a three-dimensional background model; The two-dimensional image to be reconstructed is semantically segmented to obtain at least a partial semantic segmentation image of each foreground object, and a mask of the corresponding foreground object is generated based on the at least a partial semantic segmentation image; The at least part of the semantic segmentation image is repaired to obtain the complete semantic segmentation image of the corresponding foreground object, and a 3D model of the corresponding foreground object is generated based on the complete semantic segmentation image; The three-dimensional point cloud of the two-dimensional image to be reconstructed is determined, and the background point cloud and the object point cloud of each foreground object are determined in the three-dimensional point cloud according to the mask of each foreground object; Background layout parameters are determined based on the background point cloud and the background 3D model, and corresponding object layout parameters are determined based on the object point cloud and object 3D model of each foreground object. Based on the background layout parameters and the object layout parameters corresponding to each foreground object, the background 3D model and each object 3D model are combined to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

2. The three-dimensional model generation method according to claim 1, characterized in that, The step of semantically segmenting the two-dimensional image to be reconstructed to obtain at least a partial semantic segmentation image for each foreground object includes: Retrieve multiple preset semantic tags; The two-dimensional image to be reconstructed is semantically segmented according to multiple preset semantic labels to obtain the object bounding box of each foreground object in the two-dimensional image to be reconstructed under different preset semantic labels, and the confidence score of the object bounding box; For each foreground object, the highest value is selected from the multiple confidence scores corresponding to the foreground object as the target confidence score, and at least a portion of the semantic segmentation image of the corresponding foreground object is determined based on the target confidence score and the object bounding box corresponding to the target confidence score.

3. The three-dimensional model generation method according to claim 1, characterized in that, Determining the three-dimensional point cloud of the two-dimensional image to be reconstructed includes: Perform a geometric transformation on the two-dimensional image to be reconstructed to obtain a pseudo-visual two-dimensional image corresponding to the two-dimensional image to be reconstructed. Based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image, determine the depth image corresponding to the two-dimensional image to be reconstructed; The camera parameters corresponding to the captured 2D image to be reconstructed are obtained, and the 3D point cloud of the 2D image to be reconstructed is determined based on the camera parameters and the depth image.

4. The three-dimensional model generation method according to claim 3, characterized in that, The step of determining the depth image corresponding to the two-dimensional image to be reconstructed based on the two-dimensional image to be reconstructed and the pseudo-visual two-dimensional image includes: The features of the two-dimensional image to be reconstructed are extracted from the features of the two-dimensional image to be reconstructed, and the features of the pseudo-visual two-dimensional image are extracted from the features of the pseudo-visual two-dimensional image; The features of the two-dimensional image to be reconstructed and the features of the pseudo-visual two-dimensional image are input into a pre-trained cross-attention model to obtain enhanced features of the two-dimensional image to be reconstructed and enhanced features of the pseudo-visual two-dimensional image. The pre-trained cross-attention model is trained from the sample two-dimensional image to be reconstructed and the sample features of the two-dimensional image to be reconstructed. Based on the enhanced features of the two-dimensional image to be reconstructed and the enhanced pseudo-visual two-dimensional image features, the depth image corresponding to the two-dimensional image to be reconstructed is determined.

5. The three-dimensional model generation method according to claim 1, characterized in that, The step of determining the background layout parameters based on the background point cloud and the background 3D model includes: Obtain initial background layout parameters and sample the background 3D model to obtain the sampled background point cloud corresponding to the background 3D model; Calculate the background difference between the background point cloud and the sampled background point cloud, and update the initial background layout parameters based on the background difference to obtain the updated background layout parameters; Adjust the background point cloud and the sampled background point cloud according to the updated background layout parameters to obtain the updated background point cloud and the updated sampled background point cloud; The updated background point cloud is used as the new background point cloud, the updated sampled background point cloud is used as the new sampled background point cloud, and the updated background layout parameters are used as the new initial background layout parameters. Return to the step of calculating the background difference between the background point cloud and the sampled background point cloud, until the first preset condition is reached, and use the updated background layout parameter obtained from the last update as the background layout parameter.

6. The three-dimensional model generation method according to claim 5, characterized in that, The calculation of the background difference between the background point cloud and the sampled background point cloud includes: When the number of updates to the initial background layout parameters is less than the preset threshold for background updates, the background three-dimensional spatial difference between the background point cloud and the sampled background point cloud is calculated, and the background difference between the background point cloud and the sampled background point cloud is determined based on the background three-dimensional spatial difference. When the number of updates to the initial background layout parameters is equal to or greater than the preset threshold for background updates, the background point cloud is projected to obtain a first projected background image, and the sampled background point cloud is projected to obtain a second projected background image. Calculate the background two-dimensional image difference between the first projected background image and the second projected background image, and determine the background difference between the background point cloud and the sampled background point cloud based on the background three-dimensional spatial difference and the background two-dimensional image difference.

7. The three-dimensional model generation method according to claim 1, characterized in that, The step of determining the corresponding object layout parameters based on the object point cloud and object 3D model of each foreground object includes: Obtain the initial object layout parameters for each foreground object, and sample the 3D object model of each foreground object to obtain the sampled object point cloud of each foreground object; For each foreground object, calculate the object difference between the object point cloud and the sampled object point cloud, and update the initial object layout parameters based on the object difference to obtain the updated object layout parameters; Adjust the object point cloud and the sampled object point cloud according to the updated object layout parameters to obtain the updated object point cloud and the updated sampled object point cloud; The updated object point cloud is used as the new object point cloud, the updated sampled object point cloud is used as the new sampled object point cloud, and the updated object layout parameters are used as the new initial object layout parameters. Return to the step of calculating the object difference between the object point cloud and the sampled object point cloud, until the second preset condition is reached, and use the updated object layout parameters obtained from the last update as the object layout parameters of the corresponding foreground object.

8. The three-dimensional model generation method according to claim 7, characterized in that, The calculation of object differences between the object point cloud and the sampled object point cloud includes: When the number of updates to the initial object layout parameters is less than the preset threshold for object updates, the object three-dimensional space difference between the object point cloud and the sampled object point cloud is calculated, and the object difference between the object point cloud and the sampled object point cloud is determined based on the object three-dimensional space difference. When the number of updates to the initial object layout parameters is equal to or greater than the preset threshold for object updates, the object point cloud is projected to obtain a first projected object image, and the sampled object point cloud is projected to obtain a second projected object image. Calculate the object 2D image difference between the first projected object image and the second projected object image, and determine the object difference between the object point cloud and the sampled object point cloud based on the object 3D spatial difference and the object 2D image difference.

9. A three-dimensional model generation device, characterized in that, include: The acquisition module is used to acquire the two-dimensional image to be reconstructed and to perform three-dimensional reconstruction on the background image in the two-dimensional image to be reconstructed to generate a three-dimensional background model. The semantic segmentation module is used to perform semantic segmentation on the two-dimensional image to be reconstructed to obtain at least a partial semantic segmentation image of each foreground object, and generate a mask of the corresponding foreground object based on the at least partial semantic segmentation image; The repair module is used to repair at least a portion of the semantic segmentation image to obtain a complete semantic segmentation image of the corresponding foreground object, and to generate a 3D model of the corresponding foreground object based on the complete semantic segmentation image. The point cloud determination module is used to determine the three-dimensional point cloud of the two-dimensional image to be reconstructed, and to determine the background point cloud and the object point cloud of each foreground object in the three-dimensional point cloud according to the mask of each foreground object; The layout parameter determination module is used to determine background layout parameters based on the background point cloud and the background 3D model, and to determine corresponding object layout parameters based on the object point cloud and object 3D model of each foreground object. The target 3D model generation module is used to combine the background 3D model and each object 3D model according to the background layout parameters and the object layout parameters corresponding to each foreground object to obtain the target 3D model in the scene corresponding to the 2D image to be reconstructed.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the three-dimensional model generation method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional model generation method according to any one of claims 1 to 8.

Images