Three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation

Abstract

This invention discloses a three-dimensional structure modeling method based on zero-shot semantic segmentation and physical attribute estimation, belonging to the field of image data processing technology. The method includes the following steps: constructing an image set D1; generating an initial three-dimensional point cloud model M0 based on D1; dividing M0 into S sub-point clouds; and generating the s-th sub-point cloud M. s Construct a multi-view physical property consistency error function L from the physical property maps at each visible viewpoint. s To minimize L s Adjust M s The optimized sub-model Ms′ and optimized attribute map C are obtained. s ; using all sub-point clouds and C s An optimized point cloud G is constructed, and a 3D conditional diffusion model is trained for use in 3D reconstruction. This invention enables fine modeling of target structures and details, improves reconstruction efficiency and realism, and supports controllable material and style reconstruction. It has the advantages of weakening supervision dependence and enhancing model interpretability and adjustability. It is applicable to fields such as virtual reality, industrial digital modeling, and metaverse scene generation.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation. Background Technology

[0002] Current 3D reconstruction methods primarily rely on multi-view geometric computation (such as Structure from Motion (SfM) and Dense Reconstruction (MVS)) or active perception technologies (such as structured light and LiDAR scanning). These methods are relatively mature in acquiring 3D structures. However, when faced with complex scenes, these methods still have significant shortcomings: First, in terms of texture detail representation, due to lighting variations, reflection interference, and image resolution limitations, the surface texture of the reconstructed model is often blurred or lacks detail. Second, geometric methods mainly focus on structural restoration and lack the recognition of semantic and physical information such as material and optical properties, resulting in limited expressiveness of the final model in realistic rendering.

[0003] In recent years, with the development of deep learning, large semantic segmentation models (such as SAM, CLIP-SAM, and SEEM) have made significant progress in the field of 2D images, possessing zero-shot generalization capabilities and accurately completing cross-category and cross-scene segmentation tasks without targeted training. The semantic understanding and region discrimination capabilities of these models provide strong support for high-level perception tasks. However, the integrated application of these models in 3D reconstruction systems remains relatively limited, especially in unsupervised or weakly supervised reconstruction scenarios, where semantic segmentation and 3D modeling have not yet formed an effective linkage.

[0004] Furthermore, in the transition from 3D modeling to realistic rendering, the surface optical properties of objects (such as normals, reflectivity, BRDF parameters, roughness, etc.) are key factors affecting the realism of the rendering. Accurately extracting these properties from images not only helps improve the realism and detail of the reconstructed model, but also provides a foundation for subsequent physically based rendering (PBR) or neural rendering. However, current systems generally struggle to simultaneously complete object segmentation, surface attribute extraction, reconstruction modeling, and rendering feedback within a unified workflow, leading to workflow fragmentation, inefficiency, and even inconsistencies between semantic information and geometric structure.

[0005] In summary, existing 3D reconstruction systems generally lack a collaborative modeling mechanism for semantic and physical attributes, making it difficult to meet the demands of scenarios requiring higher levels of detail realism and modeling efficiency. Therefore, how to introduce large semantic models to assist in 3D structure partitioning and combine them with optical attribute constraints to achieve a high-fidelity, renderable modeling workflow has become one of the key challenges in the current interdisciplinary field of 3D vision and graphics.

[0006] Definitions:

[0007] Structure from Motion (SfM) is a computer vision technique that uses only multiple 2D images from different viewpoints to simultaneously estimate the camera's intrinsic and extrinsic parameters (i.e., "motion") and the 3D point coordinates of the scene (i.e., "structure") for each image. It is often considered the first step in a multi-view 3D reconstruction process, outputting a sparse point cloud and camera pose.

[0008] MVS (Multi-View Stereo) is typically located in the middle to later stages of the 3D reconstruction process. It is used to expand sparse points into a dense 3D structure based on camera pose and sparse point cloud provided by SfM.

[0009] Chamfer distance is a commonly used distance metric for measuring the similarity between two sets of point clouds, and it is particularly suitable for tasks such as 3D reconstruction, registration and shape matching. Summary of the Invention

[0010] The purpose of this invention is to provide a three-dimensional structure modeling method based on zero-sample semantic segmentation and physical property estimation that solves the above-mentioned problems, achieves zero-sample semantic segmentation, semantic and geometric co-optimization, and point cloud three-dimensional reconstruction by fusing optical properties.

[0011] To achieve the above objectives, the technical solution adopted by this invention is as follows: a three-dimensional structure modeling method based on zero-sample semantic segmentation and physical attribute estimation, comprising the following steps:

[0012] S1, acquires multi-view images of a 3D scene to form an image set D1;

[0013] S2, perform 3D reconstruction on image set D1 to generate initial 3D point cloud model M0, and map image texture onto point cloud surface;

[0014] S3 uses a zero-shot semantic segmentation model to semantically partition M0 into S sub-point clouds with semantic category labels, where the s-th sub-point cloud is labeled as M. s , 1≤s≤S;

[0015] S4, for M s Select images from set D1 that show M. s There are N images as its visible view, where the i-th image is I. i perspective d i For each I i Based on its depth information and point cloud projection range, the viewpoint d is estimated on M0. i Physical property diagram A below i =(R iN i ,θ i ), where R i N i θ i respectively, perspective d i The reflectance map, normal map, and BRDF parameter map are shown below, where 1≤i≤N;

[0016] S5, construct M s Multi-view physical property consistency error function L s Including S51~S52;

[0017] S51, by adjusting M s From the point cloud structure and physical property map, a prediction sub-model is obtained. Extracting from various perspectives The physical property map is used as the predicted property map, where the viewpoint d i Predicted attribute graph ,in , , respectively, perspective d i The predicted reflectance map, predicted normal map, and predicted BRDF parameter map are shown below.

[0018] S52, based on M s and Geometric reprojection error and properties Figure 1 Construction of consistency error L s ;

[0019] S6, to minimize L s To achieve the goal, adjust M. s The point cloud structure and physical property map are used to obtain the optimized sub-model. and optimized attribute atlas ,in , , , respectively, perspective d i Optimized property map, optimized reflectivity map, optimized normal map, optimized BRDF parameter map;

[0020] S7, process each sub-point cloud according to steps S4~S6 to obtain the optimized point cloud set. And all optimized attribute graph sets are combined to form the attribute graph database D2;

[0021] S8, obtain the three-dimensional conditional diffusion model, for each of G... Adding 3D Gaussian noise to create a noise point cloud ,Will As initial input As a condition, training the 3D training diffusion model is... Denoising and generating reconstructed point clouds with surface texture. ;

[0022] S9 specifies the semantic category label of the target to be reconstructed and a set of optimized attribute maps in D2. The 3D Gaussian point cloud is used as the initial input, and after denoising by the 3D trained diffusion model, the reconstructed point cloud of the target to be reconstructed is generated.

[0023] As a preferred embodiment, S1 specifically involves: using an RGB camera, a structured light camera, or an RGB-D device to acquire images, using methods such as wrapping, translation, and spiraling to change the image viewing angle during acquisition, ensuring that adjacent images have at least 60% overlap, and that all viewing angle images completely cover the three-dimensional scene.

[0024] As a preferred option, S2 is specifically as follows:

[0025] The SfM algorithm is used to estimate the camera pose of images from different viewpoints in D1 and generate sparse point clouds.

[0026] The sparse point cloud is densified using the MVS algorithm or depth estimation algorithm to obtain a dense point cloud, which is then filtered.

[0027] The image texture in D1 is mapped onto the filtered dense point cloud surface to obtain the initial 3D point cloud model M0.

[0028] Preferably, in S3, the zero-shot semantic segmentation model includes the SAMPro3D model, the Point-SAM model, and the Diff2Scene model, and the segmentation method in S3 is as follows:

[0029] The images in D1 are sequentially input into the zero-sample semantic segmentation model, which outputs a two-dimensional semantic mask for each independent region in each viewpoint image.

[0030] The pixels of the two-dimensional semantic mask region are mapped to M0 through camera pose and depth. M0 is then divided into semantic instances to obtain S sub-point clouds with semantic category labels.

[0031] As a preferred option, in S4, for R i N i θ i All are projected from M0 onto image I i Then, the properties are obtained through a property estimation algorithm based on illumination modeling.

[0032] As a preferred embodiment, in S5, L is specifically constructed according to the following formula. s ;

[0033] ,

[0034] ,

[0035] ,

[0036] In the formula, L g For geometric reprojection error, L c For attributes Figure 1 Consistency error, λ g , λ c L respectively g L c The weighting factor, Ω i For I i M corresponds to s The set of pixels, P ij Ω i The j-th pixel p j Real coordinates p is obtained from the three-dimensional geometric projection. j The predicted coordinates, R(j), N(j), and θ(j) are respectively p j Corresponding to R i N i θ i The value in , , p j Corresponding to , , The value in It is the square of the L2 norm.

[0037] Preferably, in S8, the three-dimensional conditional diffusion model is a DiffPoint-based denoising generation network, and the training includes forward denoising and conditional guided denoising.

[0038] The forward noise addition is: for Add time-step 3D Gaussian noise to generate noisy point clouds at different time steps, with the noisy point cloud obtained at the last time step. ;

[0039] The condition-guided denoising method is: to... As a conditional input, it guides the three-dimensional conditional diffusion model from Noise is estimated and denoised time-by-time, ultimately generating a reconstructed point cloud with surface texture. ;

[0040] During training, and Symmetrical chamfering distance loss is used. and minimize Adjust the network parameters of the three-dimensional conditional diffusion model;

[0041] ,

[0042] In the formula, for The total number of midpoints, p is The point in the middle, for The total number of midpoints, g is The point in the middle, min is the minimum value. It is an L2 norm.

[0043] The concept of this invention is as follows:

[0044] (1) Following steps S1~S3, construct the initial three-dimensional point cloud model M0 of the three-dimensional scene based on the multi-view images, and then use the zero-shot semantic segmentation model to segment M0 into multiple sub-point clouds based on the multi-view images.

[0045] (2) Following step S4, process each sub-point cloud M s Estimate the d values ​​from various perspectives i Physical property diagram A i =(R i N i ,θ i This serves as a preliminary estimate of the physical properties from this perspective, for use in subsequent optimization. i =(R i N i ,θ i Based on I i Its depth information and point cloud projection range were obtained through an attribute estimation algorithm based on illumination modeling. A i Including reflectance map R i Normal diagram N i and BRDF parameter diagram θ i .

[0046] (2.1) The Reflectance Map: This refers to the intrinsic reflectance distribution of an object's surface, typically equivalent to a diffuse reflectance map or albedo map. This is the object's intrinsic color texture, unaffected by illumination. During extraction, existing multi-view illumination correction and reflectance estimation methods can be used: utilizing the brightness variations of the same surface under different illuminations in multiple images, the illumination-independent intrinsic color can be deduced. For example, M can be obtained by removing illumination effects from each visible view through multi-view intrinsic image decomposition or physically based inverse rendering techniques. s The diffuse color. The input to this technique is the image and geometry of the object from that viewpoint, and the output is the intrinsic color value of each pixel on the object's surface from that viewpoint.

[0047] (2.2) Normal Map: This refers to the mapping of surface normal vectors onto a two-dimensional image, typically represented by RGB colors to indicate XYZ direction deviations. In this invention, normal map extraction is based on existing geometric information: since M0, including the surface normals of the object, has already been obtained through S2, the normal distribution map of each instance at each viewpoint can be directly rendered or read. Furthermore, if more refined normals are required, existing techniques such as Photometric Stereo can be used to optimize the normals. The input is the initial normal data of the object and viewpoint parameters; the output is the image of the object's normal directions at that viewpoint.

[0048] (2.3) BRDF Parameter Map: BRDF is short for Bidirectional Reflectance Distribution Function, used to describe the reflection characteristics of a material surface to different incident / outgoing directions. BRDF parameter map refers to a spatially varying material parameter map (also known as Spatially Variable BRDF or SVBRDF parameters). Common BRDF parameters include specular reflectivity, roughness, metallicity, etc. In this invention, we choose a BRDF model (such as a simplified form of the Phong model or Cook-Torrance micro-surface model) to fit the illumination reflection characteristics of the object surface. Extracting the BRDF parameter map of each view is an existing inverse rendering technique. Based on a known lighting model, using the brightness of the view image and the object geometry, the reflection components of pixels are associated with material parameters. The material parameter values ​​(such as roughness values) corresponding to each pixel under that view are estimated by optimization or lookup table, thus obtaining the distribution map of BRDF parameters on the image. For example, a multi-view material estimation method can be used: the input is an image with lighting effects corrected, the normal of the corresponding pixel, and the existing reflectivity. The material parameters that make the simulated brightness match the measured brightness are calculated using the physical lighting model formula. The output is the BRDF parameter value for each pixel, which is then used to form an image based on pixel position.

[0049] (3) Construct the multi-view physical property consistency error function L according to steps S5~S7. s and minimize L s Adjust M s ,get and Specifically, it can be divided into (3.1) to (3.4):

[0050] (3.1) First, preset the prediction sub-model. Generation method: By adjusting M s The point cloud structure and physical property map were obtained, and then... Mapped to various perspectives d i Estimate and predict attribute graph Estimation methods and estimation of A i =(R i Ni ,θ i )same.

[0051] (3.2) Based on M s and Differences in constructing multi-perspective physical property consistency error function L s and minimize L s Optimize for the goal The point cloud structure and physical property map are used to achieve high-precision local optimization of sub-point clouds.

[0052] (3.3)L s Including geometric reprojection error L g and attributes Figure 1 Consistency error L c The aim is to comprehensively characterize the sub-point cloud M. s The consistency of three-dimensional structure and physical properties across multiple perspectives ensures physical rationality and optical consistency in subsequent modeling and reconstruction.

[0053] In L g In the calculation formula, For I i Ω i pixel p within the region j The true coordinates are obtained by first mapping them to three-dimensional space using three-dimensional geometric projection, and then projecting them onto I. i The corresponding image plane obtains the predicted coordinates. L g This is achieved through calculation. and The difference between them is obtained. Therefore, L g Used to measure from The deviation between the 3D point coordinates reconstructed by projection and the actual 3D position in the original image reflects whether the geometry of the sub-point cloud accurately reflects the actual outline of the object. It is an important indicator of the accuracy of the point cloud structure and belongs to the physical rationality optimization.

[0054] L c Then it is used to measure M s and The differences in the obtained physical property maps from multiple perspectives are analyzed by calculating R(j) and... , N(j) and , θ(j) and The difference between the two sides is obtained; therefore, this error term encourages the optimized model to present consistent optical performance under different viewing angles, thereby enhancing the visual consistency of objects in the real world, which belongs to optical consistency optimization.

[0055] By jointly minimizing L sIt can collaboratively optimize structural and physical properties, realize the transformation from single-view estimation to globally consistent physical modeling, and improve the overall performance of 3D models in shape reconstruction and optical consistency.

[0056] (3.4) To minimize L s Adjust all M s For each M s The optimized sub-model was obtained. and optimized attribute atlas It is used to train a three-dimensional conditional diffusion model.

[0057] (4) Train the three-dimensional conditional diffusion model so that it has the following characteristics: The ability to generate reconstructed point clouds with surface textures by progressively denoising from noisy point clouds, given the condition of [condition].

[0058] (5) Application and reasoning: The trained 3D conditional diffusion model only requires the user to specify the semantic category of the target to be reconstructed and a set of optimized attribute maps. It can then use the 3D Gaussian point cloud as a starting point to gradually denoise and obtain the reconstructed point cloud of the target to be reconstructed.

[0059] Compared with the prior art, the advantages of the present invention are as follows:

[0060] (1) This invention generates an initial 3D point cloud model M0 based on a dense point cloud reconstruction strategy that combines multi-view geometric reconstruction and depth prediction. This significantly improves the detail fidelity of the 3D model. Based on D1, a zero-shot semantic segmentation model is used to perform semantic instance segmentation on M0, realizing joint semantic segmentation of multi-view images and 3D models. This breaks through the limitations of traditional methods that rely on a large amount of manual annotation training and has better generalization ability and accuracy. Secondly, this method supports natural language prompts and multi-class semantic understanding, and can perform automatic instance-level segmentation of 3D models, providing a precise semantic foundation for subsequent physical modeling.

[0061] (2) Construct M s Multi-view physical property consistency error function L s The geometric reprojection error L was used to determine the result. g With attributes Figure 1 Consistency error L c Structural optimization and physical rationality and optical consistency optimization were performed, and L... g L c Through joint optimization, the optimized sub-model is finally obtained. and optimized attribute graphs from various perspectives . Its point cloud structure provides a more accurate description of the shape of the target object from multiple perspectives, with more complete geometric details; Presenting physical property estimates with higher consistency across different viewpoints is beneficial for downstream material control and rendering tasks.

[0062] (3) The optimized point cloud G obtained based on this invention brings the following advantages to the subsequent training of the three-dimensional conditional diffusion model:

[0063] Cross-view attribute consistency guarantee: The physical attribute map obtained through optimization has a high degree of consistency across multiple views, making the conditional control learned by the 3D conditional diffusion model more stable and avoiding problems such as style drift or lighting mismatch during the inference stage.

[0064] Achieving collaborative modeling of structure and properties: This invention not only reconstructs the geometric structure of point clouds, but also couples physical properties that reflect optical characteristics, making the reconstruction results closer to the surface properties of real objects and expanding the application boundaries of traditional geometry-driven generation methods.

[0065] Supports controllable material and style reconstruction: By introducing physical features such as reflectivity, normals and BRDF parameters into the conditions, the model has the ability to generate 3D models with specified material styles, providing a highly flexible control interface for simulation, rendering and other tasks.

[0066] Reduced reliance on supervision: Compared to traditional supervised training methods that require a large number of labeled point clouds, this invention only requires image-level input without semantic labels to achieve a closed loop from data acquisition to structural modeling, significantly reducing the cost of data construction and labeling.

[0067] Enhanced cross-category / zero-sample generalization ability: By combining conditional control mechanisms with a 3D diffusion framework, the model can generate physically consistent new structures under conditions of no known categories or new material combinations, exhibiting strong generalization adaptability.

[0068] Enhanced model interpretability and adjustability: The optimized attribute graph provides clear and traceable input control signals for the generation process, making it easier for users or downstream systems to understand the model behavior and make edits, interventions or personalized adjustments.

[0069] In summary, this invention combines pose estimation, point cloud structure, and physical properties to achieve refined modeling of target structure and details, improving the efficiency and realism of model reconstruction. The extracted physical properties can be directly used for realistic rendering and 3D model generation, and can also achieve realistic material replacement within the model. This method is widely applicable to fields such as virtual reality, industrial digital modeling, and metaverse scene generation, and has significant application value. Attached Figure Description

[0070] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0071] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0072] Example 1: See Figure 1 A three-dimensional structure modeling method based on zero-sample semantic segmentation and physical property estimation includes the following steps:

[0073] S1, acquires multi-view images of a 3D scene to form an image set D1;

[0074] S2, perform 3D reconstruction on image set D1 to generate initial 3D point cloud model M0, and map image texture onto point cloud surface;

[0075] S3 uses a zero-shot semantic segmentation model to semantically partition M0 into S sub-point clouds with semantic category labels, where the s-th sub-point cloud is labeled as M. s , 1≤s≤S;

[0076] S4, for M s Select images from set D1 that show M. s There are N images as its visible view, where the i-th image is I. i perspective d i For each I i Based on its depth information and point cloud projection range, the viewpoint d is estimated on M0. i Physical property diagram A below i =(R i N i ,θ i ), where R i N i θ i respectively, perspective d i The reflectance map, normal map, and BRDF parameter map are shown below, where 1≤i≤N;

[0077] S5, construct M s Multi-view physical property consistency error function L s Including S51~S52;

[0078] S51, by adjusting M s From the point cloud structure and physical property map, a prediction sub-model is obtained. Extracting from various perspectives The physical property map is used as the predicted property map, where the viewpoint d i Predicted attribute graph ,in , , respectively, perspective d i The predicted reflectance map, predicted normal map, and predicted BRDF parameter map are shown below.

[0079] S52, based on M s and Geometric reprojection error and properties Figure 1 Construction of consistency error L s ;

[0080] S6, to minimize L s To achieve the goal, adjust M. s The point cloud structure and physical property map are used to obtain the optimized sub-model. and optimized attribute atlas ,in , , , respectively, perspective d i Optimized property map, optimized reflectivity map, optimized normal map, optimized BRDF parameter map;

[0081] S7, process each sub-point cloud according to steps S4~S6 to obtain the optimized point cloud set. And all optimized attribute graph sets are combined to form the attribute graph database D2;

[0082] S8, obtain the three-dimensional conditional diffusion model, for each of G... Adding 3D Gaussian noise to create a noise point cloud ,Will As initial input As a condition, training the 3D training diffusion model is... Denoising and generating reconstructed point clouds with surface texture. ;

[0083] S9 specifies the semantic category label of the target to be reconstructed and a set of optimized attribute maps in D2. The 3D Gaussian point cloud is used as the initial input, and after denoising by the 3D trained diffusion model, the reconstructed point cloud of the target to be reconstructed is generated.

[0084] Example 2: See Figure 1 More specifically, based on Example 1:

[0085] S1 specifically refers to: using an RGB camera, structured light camera, or RGB-D device to acquire images, using methods such as wrapping, translation, and spiraling to change the image perspective during acquisition, ensuring that adjacent images have at least 60% overlap, and that all perspective images completely cover the 3D scene.

[0086] S2 specifically refers to:

[0087] The SfM algorithm is used to estimate the camera pose of images from different viewpoints in D1 and generate sparse point clouds.

[0088] The sparse point cloud is densified using the MVS algorithm or depth estimation algorithm to obtain a dense point cloud, which is then filtered.

[0089] The image texture in D1 is mapped onto the filtered dense point cloud surface to obtain the initial 3D point cloud model M0.

[0090] The zero-shot semantic segmentation models described in S3 include the SAMPro3D model, the Point-SAM model, and the Diff2Scene model. The segmentation method in S3 is as follows:

[0091] The images in D1 are sequentially input into the zero-sample semantic segmentation model, which outputs a two-dimensional semantic mask for each independent region in each viewpoint image.

[0092] The pixels of the two-dimensional semantic mask region are mapped to M0 through camera pose and depth. M0 is then divided into semantic instances to obtain S sub-point clouds with semantic category labels.

[0093] In S4, for R i N i θ i All are projected from M0 onto image I i Then, the properties are obtained through a property estimation algorithm based on illumination modeling.

[0094] In S5, L is specifically constructed according to the following formula. s ;

[0095] ,

[0096] ,

[0097] ,

[0098] In the formula, L g For geometric reprojection error, L c For attributes Figure 1 Consistency error, λ g , λ c L respectively g L c The weighting factor, Ω i For I i M corresponds to s The set of pixels, P ij Ω i The j-th pixel p j Real coordinates p is obtained from the three-dimensional geometric projection. j The predicted coordinates, R(j), N(j), and θ(j) are respectively p j Corresponding to R i N i θ i The value in , , pj Corresponding to , , The value in It is the square of the L2 norm.

[0099] In S8, the three-dimensional conditional diffusion model is a denoising generation network based on DiffPoint, and the training includes forward denoising and conditional guided denoising.

[0100] The forward noise addition is: for Add time-step 3D Gaussian noise to generate noisy point clouds at different time steps, with the noisy point cloud obtained at the last time step. ;

[0101] The condition-guided denoising method is: to... As a conditional input, it guides the three-dimensional conditional diffusion model from Noise is estimated and denoised time-by-time, ultimately generating a reconstructed point cloud with surface texture. ;

[0102] During training, and Symmetrical chamfering distance loss is used. and minimize Adjust the network parameters of the three-dimensional conditional diffusion model;

[0103] ,

[0104] In the formula, for The total number of midpoints, p is The point in the middle, for The total number of midpoints, g is The point in the middle, min is the minimum value. It is an L2 norm.

[0105] Example 3: To illustrate the effectiveness of the present invention, the following comparative experiment was designed:

[0106] Dataset: Real-world urban scene data, covering types such as buildings, trees, ground, and water, with a total of 120 perspective images collected for 3D reconstruction and generation tasks;

[0107] Experimental environment: NVIDIA A100 GPU, PyTorch 2.0, CUDA 11.8;

[0108] Experimental group method: The method of this invention includes zero-sample segmentation, multi-view physical property consistency optimization, and conditional diffusion modeling;

[0109] Control group method: The original model was obtained by using the PointGroup segmentation model. There was no physical property consistency optimization. The original model was directly projected with the property map and modeled using the standard Diffusion Point network without conditional guidance.

[0110] Experimental Results: The relevant performance indicators of the experimental group method and the control group method were compared and obtained in Table 1 below:

[0111] Table 1. Comparison of Experimental Results

[0112] method Segmentation accuracy (%) Attribute consistency score Point cloud reconstruction quality Material control capability Reconstruction time (min) Method of the present invention 87.6 0.925 0.944 powerful 42 Control group method 79.4 0.812 0.867 Uncontrollable 41

[0113] In Table 1, the attribute consistency score is a consistency index of the attribute map prediction error under multiple perspectives (the higher the score, the more consistent it is across perspectives).

[0114] Point cloud reconstruction quality: a quality score based on a comprehensive assessment of human perception and Chamfer Distance.

[0115] Data Analysis: While preserving the accuracy of semantic structure, the method of this invention introduces a cross-view optimization mechanism for reflectivity, normals, and BRDF parameters, which makes the samples in the training set have higher physical realism, thereby improving the training effect and generation quality of the diffusion model.

[0116] Compared to the control group, this invention can significantly improve the accuracy and geometric restoration capability of 3D reconstruction after segmentation; it can perform material style transfer and rendering control on different physical property maps; it supports training diffusion models under low-supervision conditions and has stronger generalization and reconstruction adaptability.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation, characterized in that, Includes the following steps: S1, acquires multi-view images of a 3D scene to form an image set D1; S2, perform 3D reconstruction on image set D1 to generate initial 3D point cloud model M0, and map image texture onto point cloud surface; S3 uses a zero-shot semantic segmentation model to semantically partition M0 into S sub-point clouds with semantic category labels, where the s-th sub-point cloud is labeled as M. s , 1≤s≤S; S4, for M s Select images from set D1 that show M. s There are N images as its visible view, where the i-th image is I. i perspective d i For each I i Based on its depth information and point cloud projection range, the viewpoint d is estimated on M0. i Physical property diagram A below i =(R i N i ,θ i ), where R i N i θ i respectively, perspective d i The reflectance map, normal map, and BRDF parameter map are shown below, where 1 ≤ i ≤ N; S5, construct M s Multi-view physical property consistency error function L s Including S51~S52; S51, by adjusting M s From the point cloud structure and physical property map, a prediction sub-model is obtained. Extracting from various perspectives The physical property map is used as the predicted property map, where the viewpoint d i Predicted attribute graph ,in , , respectively, perspective d i The predicted reflectance map, predicted normal map, and predicted BRDF parameter map are shown below. S52, based on M s and Construction of geometric reprojection error and property map consistency error L s ; S6, to minimize L s To achieve the goal, adjust M. s The point cloud structure and physical property map are used to obtain the optimized sub-model. and optimized attribute atlas ,in , , , respectively, perspective d i Optimized property map, optimized reflectivity map, optimized normal map, optimized BRDF parameter map; S7, process each sub-point cloud according to steps S4~S6 to obtain the optimized point cloud set. And all optimized attribute graph sets are combined to form the attribute graph database D2; S8, obtain the three-dimensional conditional diffusion model, for each of G... Adding 3D Gaussian noise to create a noise point cloud ,Will As initial input As a condition, training the 3D training diffusion model is... Denoising and generating reconstructed point clouds with surface texture. ; S9 specifies the semantic category label of the target to be reconstructed and a set of optimized attribute maps in D2. The 3D Gaussian point cloud is used as the initial input, and after denoising by the 3D trained diffusion model, the reconstructed point cloud of the target to be reconstructed is generated.

2. The three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation according to claim 1, characterized in that, S1 specifically refers to: using an RGB camera, structured light camera, or RGB-D device to acquire images, using methods such as wrapping, translation, and spiraling to change the image perspective during acquisition, ensuring that adjacent images have at least 60% overlap, and that all perspective images completely cover the 3D scene.

3. The three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation according to claim 1, characterized in that, S2 specifically refers to: The SfM algorithm is used to estimate the camera pose of images from different viewpoints in D1 and generate sparse point clouds. The sparse point cloud is densified using the MVS algorithm or depth estimation algorithm to obtain a dense point cloud, which is then filtered. The image texture in D1 is mapped onto the filtered dense point cloud surface to obtain the initial 3D point cloud model M0.

4. The three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation according to claim 1, characterized in that, In S3, the zero-shot semantic segmentation model includes the SAMPro3D model, the Point-SAM model, and the Diff2Scene model. The segmentation method in S3 is as follows: The images in D1 are sequentially input into the zero-sample semantic segmentation model, which outputs a two-dimensional semantic mask for each independent region in each viewpoint image. The pixels of the two-dimensional semantic mask region are mapped to M0 through camera pose and depth. M0 is then divided into semantic instances to obtain S sub-point clouds with semantic category labels.

5. The three-dimensional structure modeling method based on zero-sample semantic segmentation and physical property estimation according to claim 1, characterized in that, In S4, for R i N i θ i All are projected from M0 onto image I i Then, the properties are obtained through a property estimation algorithm based on illumination modeling.

6. The three-dimensional structure modeling method based on zero-shot semantic segmentation and physical property estimation according to claim 1, characterized in that, In S5, L is specifically constructed according to the following formula. s ; ， ， ， In the formula, L g For geometric reprojection error, L c λ represents the property graph consistency error. g , λ c L respectively g L c The weighting factor, Ω i For I i M corresponds to s The set of pixels, P ij Ω i The j-th pixel p j Real coordinates p is obtained from the three-dimensional geometric projection. j The predicted coordinates, R(j), N(j), and θ(j) are respectively p j Corresponding to R i N i θ i The value in , , p j Corresponding to , , The value in It is the square of the L2 norm.

7. The three-dimensional structure modeling method based on zero-sample semantic segmentation and physical property estimation according to claim 1, characterized in that, In S8, the three-dimensional conditional diffusion model is a denoising generation network based on DiffPoint, and the training includes forward denoising and conditional guided denoising. The forward noise addition is: for Add time-step 3D Gaussian noise to generate noisy point clouds at different time steps, with the final time step yielding the noisy point cloud. ; The condition-guided denoising method is: to... As a conditional input, it guides the three-dimensional conditional diffusion model from Noise is estimated and denoised time-by-time, ultimately generating a reconstructed point cloud with surface texture. ; During training, and Symmetrical chamfering distance loss is used. and minimize Adjust the network parameters of the three-dimensional conditional diffusion model; ， In the formula, for The total number of midpoints, p is The point in the middle, for The total number of midpoints, g is The point in the middle, min is the minimum value. It is an L2 norm.

Images