A structural prior visual three-dimensional reconstruction method, system, device and medium

Abstract

The present invention relates to the field of three-dimensional reconstruction technology, and in particular, to a structural prior visual three-dimensional reconstruction method and system, device, and medium. The goal of the present invention is to develop an innovative three-dimensional reconstruction method, the core of which is to use a set of predefined, parameterized structured primitives and their numerous instances in the scene to represent and reconstruct the three-dimensional world. The difference from the prior art is that this solution does not use traditional point clouds, grids, or a large number of independent ellipsoid primitives, but models the scene as a collection of structured components. Through a differentiable rendering engine, the complete parameter set of each primitive instance is optimized end-to-end based on the input image, including spatial position, orientation, size, shape details, and appearance attributes, so as to accurately reconstruct the geometry and appearance of the scene and naturally obtain an efficient, semantically inclusive, and easy-to-edit scene representation.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a structure-prior visual three-dimensional reconstruction method, system, device, and medium. Background Art

[0002] The core design goal of current mainstream 3D vision technologies, whether they are laser scanning technologies that rely on active measurement, 3D reconstruction methods based on passive image acquisition (such as structure from motion (SfM) and multi-view stereo matching (MVS), and emerging scene representation technologies based on neural radiance fields or Gaussian primitives (such as 3DGS), is to achieve universal geometric and appearance expression for arbitrarily complex scenes.

[0003] To achieve this goal, these techniques typically employ a "bottom-up" reconstruction strategy that relies little or no on prior knowledge of specific structures. They use a massive, fine-grained set of basic representational units—such as discrete points (point clouds), parameterized ray samples, independent Gaussian ellipsoids, or basic triangular facets—to approximate and fit the observed scene one by one. While ensuring universality, this strategy also suffers from representational redundancy and inefficiency. When a scene contains a large number of regular structures, repetitive patterns (such as building facades, industrial pipelines, and vegetation forms), or specific object categories, the use of atomic, structure-free primitives for representation inevitably leads to a high degree of redundancy in data representation. Summary of the Invention

[0004] The purpose of the present invention is to provide a structure-prior visual three-dimensional reconstruction method and system, equipment, and medium to solve the above-mentioned problems in the prior art.

[0005] The present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a structure prior visual 3D reconstruction method, comprising:

[0007] Obtain several standard images of the modeled object, obtain basic parameters of primitive instances in the standard images, generate parameter sets of the primitive instances, build a differentiable rendering model, train the differentiable rendering model using the parameter sets and the basic images, and output the trained differentiable rendering model;

[0008] Obtain the current base image, and obtain the basic parameters of the target primitive instance in the base image to generate a parameter set, optimize the current parameter set through the differentiable rendering model, and output the target parameter set;

[0009] A mapping relationship between the current target parameter set and the corresponding target primitive instance is established to generate a primitive instance target group, and a primitive instance list is generated after saving several primitive instance target groups and category labels of primitive instances. The primitive instance list constitutes the modeled object.

[0010] Preferably, the training of the differentiable rendering model using the parameter set and the base image includes:

[0011] S201: Obtain a parameter set of a primitive instance, and generate a synthetic image through a differentiable rendering model based on the parameter set of the primitive instance;

[0012] S202: Compare the synthesized image with the standard image and calculate one or more loss functions;

[0013] S203: using the automatic differentiation framework to back-propagate the loss through the rendering pipeline to obtain the gradient of the loss function with respect to each optimizable parameter of each primitive instance;

[0014] S204: using a gradient descent optimizer to update parameter sets of all primitive instances according to the gradient;

[0015] S205: looping the updated parameter set from step S201, recording the number of iterations and setting an upper limit for the number of iterations, and outputting the current differentiable rendering model when the upper limit for the number of iterations is reached.

[0016] Preferably, the optimizing the current parameter set by using the differentiable rendering model includes:

[0017] Get the current parameter set, which includes the location parameters, orientation parameters, size parameters, color parameters, border parameters, and texture parameters of the primitive instance;

[0018] Get the world coordinates of the primitive instance in the three-dimensional world according to the position parameter, orientation parameter and size parameter;

[0019] Using the camera's internal and external parameters, several three-dimensional vertices are projected onto the two-dimensional image plane to obtain the 2D polygonal outline of the primitive instance on the image;

[0020] Determine the pixels occluded by the 2D polygon outline, determine the occluded pixels in the base image through the pixels occluded by the 2D polygon outline, and calculate the final pixel color through the surface normal vector, light source direction and surface color;

[0021] Calculate the gradient of the loss function for each element in the parameter set of the primitive instance and adjust the parameters within the element according to the gradient.

[0022] Preferably, obtaining the world coordinates of the primitive instance in the three-dimensional world according to the position parameter, the orientation parameter and the size parameter comprises:

[0023] Calculate local world coordinates based on the shape of the primitive instance;

[0024] Combine position parameters, orientation parameters and size parameters into a transformation matrix;

[0025] Multiply the transformation matrix and the local world coordinates to get the actual three-dimensional world coordinates.

[0026] Preferably, it also includes:

[0027] Set an error threshold. If the rendering error between the current synthetic image and the standard image is greater than the error threshold, or if there is a missing structure area in the synthetic image, add new primitive instances and remove redundant instances that are too small, too low in transparency, contribute little to the final rendering, or are completely occluded.

[0028] Preferably, it also includes:

[0029] Initiate candidate primitive selection process;

[0030] The candidate primitive selection process includes running a pre-trained or scene-optimized instance segmentation model on the original input image corresponding to the error region, identifying each primitive instance in the image through the instance segmentation model, and outputting their category labels and masks;

[0031] If the category label of a recognized primitive instance matches the category label of a primitive instance in the primitive instance list library, the matched primitive instance is replaced by a primitive instance in the primitive instance list library.

[0032] Preferably, when a primitive instance is replaced and the pixel loss value is still greater than the error threshold, or there are still missing structure areas in the synthesized image, a proliferation 3DGS ellipsoid is added to the primitive instance to fit the area where the pixel loss value of the primitive instance is still greater than the error threshold or the missing structure area.

[0033] In a second aspect, the present invention further provides a structure prior visual 3D reconstruction system, comprising:

[0034] a model training module configured to obtain a plurality of standard images of the modeled object, obtain basic parameters of primitive instances in the plurality of standard images, generate a parameter set of the primitive instances, construct a differentiable rendering model, train the differentiable rendering model using the parameter set and the basic images, and output the trained differentiable rendering model;

[0035] an optimization module configured to obtain a current base image, obtain basic parameters of a target primitive instance in the base image, generate a parameter set, optimize the current parameter set using a differentiable rendering model, and output a target parameter set;

[0036] The list establishment module is configured to establish a mapping relationship between the current target parameter set and the corresponding target primitive instance to generate a primitive instance target group, save several primitive instance target groups and category labels of the primitive instances, and then generate a primitive instance list. The primitive instance list constitutes the modeled object.

[0037] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned structural prior visual three-dimensional reconstruction method when executing the computer program.

[0038] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned structure-prior visual three-dimensional reconstruction method when executed by a processor.

[0039] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0040] The goal of the present invention is to develop an innovative 3D reconstruction method, the core of which is to use a set of predefined, parameterized structured primitives and their numerous instances in the scene to represent and reconstruct the 3D world. First, a standard multi-view image is used as input. The difference from the existing technology is that this solution does not use traditional point clouds, meshes, or a large number of independent ellipsoid primitives, but models the scene as a collection of structured components. Through a differentiable rendering engine, the complete parameter set of each primitive instance is optimized end-to-end based on the input image, including spatial position, orientation, size, shape details, and appearance attributes, so as to accurately reconstruct the geometry and appearance of the scene and naturally obtain an efficient, semantic and easy-to-edit scene representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a control flow diagram of the present invention;

[0043] Figure 2 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0045] The terms "first," "second," and so on, used in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The naming or numbering of steps in this application does not necessarily imply that the steps in the method flow must be executed in the chronological or logical order indicated by the naming or numbering. Named or numbered process steps may be executed in a different order based on the desired technical objectives, as long as the same or similar technical effects are achieved. The module divisions described in this application are logical divisions; in actual implementation, other divisions may be employed. For example, multiple modules may be combined or integrated into another system, or some features may be omitted or not implemented. Furthermore, the connection, coupling, or communication referred to in this application may be a direct connection, coupling, or communication between related objects, or an indirect connection, coupling, or communication via other devices. Furthermore, the connection, coupling, or communication between objects may be electrical or other similar forms, which are not limited in this application. Independently described modules or submodules may or may not be physically separate; they may be implemented in software or hardware. Some modules or submodules may be implemented in software, with the processor invoking the software to implement the functionality of these modules or submodules, while other modules or submodules may be implemented in hardware, such as hardware circuits. Furthermore, some or all of the modules may be selected based on actual needs to achieve the objectives of the present application.

[0046] Please refer to Figure 1-Figure 2 The present invention provides a structure prior visual 3D reconstruction method, comprising:

[0047] S101: Acquire several standard images of the modeled object, obtain basic parameters of primitive instances in the several standard images, generate parameter sets of the primitive instances, construct a differentiable rendering model, train the differentiable rendering model using the parameter sets and the basic images, and output the trained differentiable rendering model;

[0048] The design of this approach focuses on creating parameterized primitive models that effectively capture recurring structures or typical elements in specific scene types. These primitives are more than just simple points, lines, and surfaces. For example, when reconstructing a photovoltaic plant, a "photovoltaic panel" primitive can be defined (with parameterized position and orientation). The appearance of each primitive is also controlled through parameterization (such as spherical harmonics to represent illumination response, or small neural networks or texture atlases to represent surface details). This structural prior-based primitive design is key to distinguishing it from general-purpose representation methods.

[0049] S102: Obtain a current base image, obtain basic parameters of a target primitive instance in the base image to generate a parameter set, optimize the current parameter set using a differentiable rendering model, and output a target parameter set;

[0050] S103: Establish a mapping relationship between the current target parameter set and the corresponding target primitive instance to generate a primitive instance target group, save several primitive instance target groups and category labels of primitive instances to generate a primitive instance list, and the primitive instance list constitutes the modeled object.

[0051] In this approach, the final 3D scene is represented as a list of primitive instances. Each entry in the list represents a specific object or component in the scene and consists of an identifier for the primitive type used and a set of instance-specific parameters. These parameters primarily include the rigid body pose (3D position and rotation), scale (possibly anisotropic scaling), and specific shape parameters (such as panel deformation). By optimizing these instance parameters, we can accurately reproduce observed scene details (the optimization process is referenced by the 3DGS method). This instanced representation not only significantly reduces representation redundancy (for example, a photovoltaic plant with 100,000 panels requires only 100,000 primitives, while 3DGS may require hundreds of millions of primitives, and point clouds may require tens of billions of primitives), but also naturally supports instance-based editing, analysis, and efficient rendering.

[0052] In order to achieve end-to-end optimization from input image to structured primitive parameters, this scheme adopts (or integrates) a differentiable rendering mechanism, the core idea of ​​which is to transform the rendering process into a differentiable mathematical operation.

[0053] The core of differentiable rendering lies in constructing each stage of the traditional computer graphics rendering pipeline—such as geometric transformation (converting three-dimensional object parameters such as position, rotation, and scale into world coordinates, and then projecting them onto the two-dimensional image plane using camera parameters), visibility determination (determining which object or surface is visible at a specific pixel, often involving rasterization or ray tracing), and shading calculation (calculating pixel color based on material properties, light source information, and surface normal vectors)—into mathematically differentiable operations (or using differentiable approximations). Once the entire forward rendering path from parameters to pixel values ​​is established, the automatic differentiation framework can effectively calculate reverse gradients to optimize the input parameters.

[0054] In an exemplary embodiment of the present invention, the training of the differentiable rendering model using the parameter set and the base image includes:

[0055] S201: Obtain a parameter set of a primitive instance, and generate a synthetic image through a differentiable rendering model based on the parameter set of the primitive instance;

[0056] Among them, the parameters of the camera (image), including its location, are first estimated, and then the three-dimensional space is reconstructed through primitive instances. The synthetic image seen by a certain camera in the reconstructed three-dimensional space is obtained through projection (rendering).

[0057] S202: Compare the synthesized image with the standard image and calculate one or more loss functions;

[0058] S203: using the automatic differentiation framework to back-propagate the loss through the rendering pipeline to obtain the gradient of the loss function with respect to each optimizable parameter of each primitive instance;

[0059] S204: using a gradient descent optimizer to update parameter sets of all primitive instances according to the gradient;

[0060] S205: looping the updated parameter set from step S201, recording the number of iterations and setting an upper limit for the number of iterations, and outputting the current differentiable rendering model when the upper limit for the number of iterations is reached.

[0061] Specifically, when instances are differentiable, a deep framework can be used for gradient learning. First, a differentiable renderer is used to generate a synthetic image using the current set of primitive instance parameters. The synthetic image is then compared to the real input image from the corresponding viewpoint, and one or more loss functions are calculated (for example, pixel-wise L1 or L2 loss, combined with perceptual losses such as the structural similarity index (D-SSIM) to ensure visual fidelity). Next, an automatic differentiation framework (such as PyTorch or TensorFlow) is used to backpropagate the loss through the rendering pipeline, obtaining the gradient of the loss function with respect to each optimizable parameter of each primitive instance. Finally, a gradient descent optimizer (such as Adam) is used to update the parameters of all instances based on these gradients. This process is repeated, gradually adjusting the parameters of the primitive instances until the rendered result closely resembles the real image, achieving accurate reconstruction of the scene structure and appearance. The optimization targets are the relatively small number of well-defined primitive instance parameters, rather than millions of unstructured, low-level elements.

[0062] In an exemplary embodiment of the present invention, optimizing the current parameter set using a differentiable rendering model includes:

[0063] Get the current parameter set, which includes the location parameters, orientation parameters, size parameters, color parameters, border parameters, and texture parameters of the primitive instance;

[0064] Get the position coordinates, rotation angle and scaling parameters of the primitive instance in the three-dimensional world according to the position parameters, orientation parameters and size parameters;

[0065] Using the camera's internal and external parameters, several three-dimensional vertices are projected onto the two-dimensional image plane to obtain the 2D polygonal outline of the primitive instance on the image;

[0066] Determine the pixels occluded by the 2D polygon outline, determine the occluded pixels in the base image through the pixels occluded by the 2D polygon outline, and calculate the final pixel color through the surface normal vector, light source direction and surface color;

[0067] Specifically, by performing pixel loss (SSIM algorithm) on the projection onto the two-dimensional image plane and the actual image, a quantized loss value can be obtained. The occluded pixels are restored using the quantized loss value to obtain the final pixel.

[0068] Calculate the gradient of the loss function for each element in the parameter set of the primitive instance and adjust the parameters within the element according to the gradient.

[0069] Since each primitive is differentiable, the gradient direction of each primitive parameter can be obtained by using the gradient descent algorithm, and the parameters of the primitive can be adjusted.

[0070] Specifically, we use photovoltaic panels as an example to illustrate how to implement differentiable rendering:

[0071] A photovoltaic panel instance is defined by a set of parameters: P = {position (x, y, z), orientation (rx, ry, rz), size (length L, width W), surface base color (R, G, B), border color (Rr, Gr, Br), and possibly texture mapping parameters, etc.}.

[0072] Geometric Construction and Transformation: First, based on the position, orientation, and dimensions in the parameter P, the world coordinates of the photovoltaic panel in the 3D world (e.g., the four or eight corner points of a rectangular panel) are calculated. This process is a standard affine transformation and is completely differentiable. Then, using the camera's intrinsic and extrinsic parameters, these 3D vertices are projected onto the 2D image plane to obtain the panel's 2D polygonal outline on the image. This projection process is also differentiable.

[0073] Rasterization / Visibility: It is necessary to determine which pixels are covered by this 2D polygon. Traditional rasterization has hard boundaries, where pixels are either completely inside or completely outside, and are not differentiable at the boundaries. Differentiable renderers use "soft rasterization" techniques, where, for example, the coverage of a pixel can be a continuous value between 0 and 1, representing the degree of coverage (similar to the contribution of the Gaussian kernel to the pixel in 3D GaussianSplatting), or a smooth boundary can be obtained by interpolation. In this way, small changes in the panel parameters will smoothly change the pixel's coverage value and its color contribution, allowing gradients to propagate smoothly.

[0074] Colorization: For pixels covered (or partially covered) by the panel:

[0075] Normal vector: The surface normal vector of the panel is determined by its orientation parameter and can be differentiated.

[0076] Color / Texture: The base color is determined based on the panel's surface base color parameters or a texture map (texture sampling typically uses bilinear interpolation, which is also differentiable). The border can be handled in a similar way.

[0077] Lighting: Apply a simplified, differentiable lighting model (such as the Lambertian model) to calculate the final pixel color based on the surface normal, the light source direction (which can be assumed or optimized), and the surface color. All of these calculations are differentiable.

[0078] Loss calculation and optimization: The rendered image is compared to the corresponding true input image (e.g., calculating L1 or L2 loss). Since the entire process from parameters P to the final rendered image is differentiable, the gradient of the loss function with respect to each element of the photovoltaic panel parameters P can be calculated. For example, if the rendered panel position is off, the gradient will indicate how to adjust (x, y, z); if the color is incorrect, the gradient will indicate how to adjust (R, G, B), and so on. The optimizer (e.g., SGD, Adam) then uses these gradients to iteratively update the parameters P, making the rendered result increasingly close to the true image, thereby achieving accurate reconstruction of the photovoltaic panel instance.

[0079] In an exemplary embodiment of the present invention, obtaining the world coordinates of the primitive instance in the three-dimensional world according to the position parameter, the orientation parameter, and the size parameter includes:

[0080] Calculate local world coordinates based on the shape of the primitive instance;

[0081] Combine position parameters, orientation parameters and size parameters into a transformation matrix;

[0082] Multiply the transformation matrix and the local world coordinates to get the actual three-dimensional world coordinates.

[0083] An exemplary embodiment of the present invention further includes:

[0084] An error threshold is set. If the pixel loss value between the current synthetic image and the standard image is greater than the error threshold, or if there is a missing structure area in the synthetic image, new primitive instances are added and redundant instances that are too small, too low in transparency, contribute little to the final rendering, or are completely occluded are removed. SSIM is used to calculate the pixel loss value.

[0085] For example, when fitting a scene using photovoltaic panels, if the SSIM values ​​in certain areas where the panels are not yet installed (only the brackets or grass) are present, no matter how you fit, the fitting will be very large, indicating a fitting failure. You can then remove the photovoltaic panels in these areas and use the photovoltaic bracket primitives for fitting.

[0086] Specifically, based on 3D Gaussian sputtering, an adaptive control mechanism for primitive instances is introduced to dynamically adjust the complexity of scene representation and resource allocation during the reconstruction process. This includes: instance proliferation (Densification), which automatically adds new primitive instances in areas with large rendering errors or missing structures based on geometric analysis (for example, automatically adding more "leaf cluster" instances in sparse tree canopy areas, or adding "photovoltaic panel" instances when unrepresented photovoltaic panels are detected); instance pruning (Pruning), which removes redundant instances that are too small, too low in transparency, contribute little to the final rendering, or are completely occluded; and possible instance splitting / merging (Splitting / Merging), such as splitting a long "branch segment" instance with an inaccurate representation into two shorter instances to allow for more flexible deformation, or merging instances that are too close and similar to simplify the representation.

[0087] This adaptive control at the level of structured primitive instances is another key innovation of this solution. Compared to adding, deleting, and adjusting millions of low-level elements (such as Gaussian points), the management of structured instances can more directly utilize the macroscopic structural information of the scene. This allows the optimization process to allocate computing resources more intelligently, using fewer "high-level" primitive instances to efficiently represent simple, repetitive structures, while also increasing instance density where details are required. The resulting scene representation is not only accurate in geometry and appearance, but also concise and meaningful in structure, and the number of instances is far less than that of point-based or Gaussian primitive-based methods, which is conducive to storage, transmission, and subsequent applications (such as editing and simulation).

[0088] An exemplary embodiment of the present invention further includes an alternative primitive selection process:

[0089] The candidate primitive selection process includes running a pre-trained or scenario-optimized instance segmentation model on the original input image corresponding to the error region, identifying each primitive instance in the image through the instance segmentation model, and outputting their category labels and masks;

[0090] If the category label of a recognized primitive instance matches the category label of a primitive instance in the primitive instance list library, the matched primitive instance is replaced by a primitive instance in the primitive instance list library.

[0091] Specifically, in primitive instance replacement, when the system determines that the current structured primitive type in a certain area does not fit well (for example, based on persistently high reconstruction error or geometric mismatch analysis), the alternative primitive selection process is initiated. This process can include a variety of mechanisms, one of which is an advanced and efficient way to use image instance segmentation technology for semantic guidance. Specifically, the system will run a pre-trained or scene-specific optimized instance segmentation model (such as Mask R-CNN or its variants) on the original input image (of one or more perspectives) corresponding to the high-error area. The model can identify each primitive instance in the image and output their category labels and accurate pixel-level masks.

[0092] The system then analyzes the segmentation results that fall within the problem area. If the category label of one or more identified instances (e.g., "tree," "small outbuilding," "ventilation duct," "specific model equipment housing," etc.) matches the semantic description of a primitive type in the predefined structured primitive library, then the matching primitive type will be selected as a replacement candidate. For example, if instance segmentation identifies a clear "cylindrical tank" instance in an area that was originally attempted to be represented by a "general flat plate" primitive but did not work well, the system will automatically call the predefined "cylinder" primitive from the primitive library for replacement. Furthermore, the pixel mask and bounding box information provided by instance segmentation can also be used to initialize the spatial pose (position and orientation) and approximate size of the newly replaced primitive instance with high quality, thereby significantly accelerating the convergence speed of the subsequent optimization process and improving the success rate.

[0093] This alternative primitive selection method based on instance segmentation makes the primitive replacement decision no longer limited to simple error pattern analysis or fixed trial sequence, but introduces powerful image content understanding and semantic priors. It can more accurately determine what type of object the local area "should" be, thereby selecting the most appropriate structured primitive for modeling. This not only improves the success rate and efficiency of replacement and reduces blind trial and error, but also enables the entire 3D reconstruction framework to better handle unexpected and diverse object categories in the scene, greatly enhancing the system's robustness, degree of automation and the semantic accuracy of the final reconstructed model. Of course, the effectiveness of this method depends in part on the performance of the instance segmentation model and the completeness of the correspondence between its recognition categories and the primitive types defined in the primitive library.

[0094] Secondly, while the parameterized structured primitives designed in this solution can efficiently and semantically represent most major components in a scene (such as the panel arrays in a photovoltaic plant), real-world surfaces often contain subtle, unstructured, or random details that cannot be fully captured by simple parameterized models. Examples include irregular stains, snow, and fallen leaves on photovoltaic panels, small objects with unique and complex textures in the scene, and edge transition regions that are difficult for structured primitives to accurately capture. To improve the reconstruction fidelity of these high-frequency details without sacrificing the macro-advantages of structured representation (such as semantics, editability, and model compactness), this solution further integrates a complementary detail modeling layer based on general implicit primitives (such as the ellipsoid in 3DGS).

[0095] Secondly, when a primitive instance is replaced and the pixel loss value is still greater than the error threshold, or there are still missing structure areas in the synthesized image, a proliferation 3DGS ellipsoid is added to the primitive instance to fit the area where the pixel loss value of the primitive instance is still greater than the error threshold or the missing structure area.

[0096] Specifically, the 3DGS supplementary layer is not intended to independently reconstruct the entire scene, but rather to serve as a "residual corrector" or "detail enhancer" for the structured primitive model. Its activation and application are controlled and targeted: for example, only when the image reconstruction loss in certain local areas is still significantly higher than a preset threshold after the optimization of the structured primitives, or when the geometric analysis indicates the presence of visual features that are not fully explained by the structured primitives, will 3DGS ellipsoids be introduced or proliferated in these specific areas. The number, position, shape, color, and opacity of these supplementary Gaussian ellipsoids are also optimized through differentiable rendering, with the goal of fitting the image residuals that are not explained by the structured primitives, where the image residuals are areas where the pixel loss value of the primitive instance is greater than the error threshold or areas with missing structure.

[0097] During rendering and optimization, the structured primitive layer and the supplementary 3DGS layer work together. During rendering, their contributions are typically alpha-blended based on depth and opacity to form the final output image. During optimization, a joint optimization strategy can be employed to simultaneously adjust the parameters of the structured primitives and the parameters of the 3DGS ellipsoid. To ensure the dominance of the structured representation and avoid overfitting of the 3DGS, a prioritization mechanism or regularization term can be introduced, for example, guiding the structured primitives to prioritize interpreting the overall structure and primary appearance of the scene, while the 3DGS focuses on capturing the remaining subtle variations. Through this intelligent fusion strategy, the proposed scheme can fully leverage the macro-modeling advantages of structured primitives and the micro-detail expression capabilities of 3DGS, ultimately generating a 3D scene model that combines rich semantic structure with high visual fidelity.

[0098] The beneficial effects brought about by this program are specifically as follows:

[0099] 1. Highly efficient and highly compressed scene representation

[0100] This method significantly reduces redundancy by using parameterized structured primitives and their numerous instances to represent 3D scenes. Compared to traditional point cloud or dense Gaussian primitive approaches, this method significantly reduces data storage and computational overhead. This compact representation improves the overall efficiency of large-scale scene reconstruction, transmission, and rendering.

[0101] 2. Semantically rich and easily editable 3D models

[0102] The scene model generated by this method naturally incorporates object-level instance information and structural priors, imbuing the reconstruction with clear semantic meaning, far surpassing unstructured point clouds or purely neural representations. This instance-based structured representation greatly facilitates subsequent scene analysis, single object editing, physical simulation, and integration with downstream applications such as BIM.

[0103] 3. High-fidelity reconstruction quality with attention to detail

[0104] By optimizing the complete parameter set of structured primitives end-to-end, combined with adaptive instance management and intelligent primitive replacement strategies, the proposed method accurately reconstructs the scene's macrostructure and major components. Furthermore, through an optional 3DGS supplementation mechanism, it effectively captures complex surface details and unstructured elements that are difficult to express using structured primitives, achieving high-fidelity reconstruction that balances macrostructure and microscopic details.

[0105] 4. Powerful scene adaptability and automation

[0106] The adaptive instance management (including proliferation, pruning, splitting / merging) and dynamic primitive replacement strategy (guided by instance segmentation) proposed in this paper enable the reconstruction system to dynamically adjust the type and distribution of primitives based on scene complexity and reconstruction quality. This high degree of adaptability and automation reduces manual intervention and improves robustness and applicability to different scene types.

[0107] In a second aspect, the present invention further provides a structure prior visual 3D reconstruction system, comprising:

[0108] a model training module configured to obtain a plurality of standard images of the modeled object, obtain basic parameters of primitive instances in the plurality of standard images, generate a parameter set of the primitive instances, construct a differentiable rendering model, train the differentiable rendering model using the parameter set and the basic images, and output the trained differentiable rendering model;

[0109] an optimization module configured to obtain a current base image, obtain basic parameters of a target primitive instance in the base image, generate a parameter set, optimize the current parameter set using a differentiable rendering model, and output a target parameter set;

[0110] The list establishment module is configured to establish a mapping relationship between the current target parameter set and the corresponding target primitive instance to generate a primitive instance target group, save several primitive instance target groups and category labels of the primitive instances, and then generate a primitive instance list. The primitive instance list constitutes the modeled object.

[0111] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored on a computer-readable storage medium. This computer software product, stored on a storage medium, includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A structure prior visual 3D reconstruction method, characterized in that: include: Obtain several standard images of the modeled object, obtain basic parameters of primitive instances in the standard images, generate parameter sets of the primitive instances, build a differentiable rendering model, train the differentiable rendering model using the parameter sets and the basic images, and output the trained differentiable rendering model; Obtain the current base image, and obtain the basic parameters of the target primitive instance in the base image to generate a parameter set, optimize the current parameter set through the differentiable rendering model, and output the target parameter set; Establishing a mapping relationship between a current target parameter set and a corresponding target primitive instance to generate a primitive instance target group, saving a plurality of primitive instance target groups and category labels of primitive instances to generate a primitive instance list, wherein the primitive instance list constitutes the modeled object; The training of the differentiable rendering model using the parameter set and the base image includes: S201: Obtain a parameter set of a primitive instance, and generate a synthetic image through a differentiable rendering model based on the parameter set of the primitive instance; S202: Compare the synthesized image with the standard image and calculate one or more loss functions; S203: using the automatic differentiation framework to back-propagate the loss through the rendering pipeline to obtain the gradient of the loss function with respect to each optimizable parameter of each primitive instance; S204: using a gradient descent optimizer to update parameter sets of all primitive instances according to the gradient; S205: looping the updated parameter set from step S201, recording the number of iterations and setting an upper limit for the number of iterations, and outputting the current differentiable rendering model when the upper limit for the number of iterations is reached; Set an error threshold. If the pixel loss between the current synthesized image and the standard image is greater than the error threshold, or if there is a missing structure area in the synthesized image, add a new primitive instance and remove the replaced primitive instance. Adding a new primitive instance and removing the replaced primitive instance includes: Initiate candidate primitive selection process; The candidate primitive selection process includes running a pre-trained instance segmentation model on the original input image corresponding to the error region, identifying each primitive instance in the image through the instance segmentation model, and outputting a category label and a mask of the primitive instance; If the category label of a recognized primitive instance matches the category label of a primitive instance in the primitive instance list library, the matched primitive instance is replaced by a primitive instance in the primitive instance list library; When a primitive instance is replaced and the pixel loss value is still greater than the error threshold, or there is still a missing structure area in the synthesized image, a proliferation 3DGS ellipsoid is added to the primitive instance to fit the area where the pixel loss value of the primitive instance is still greater than the error threshold or the missing structure area.

2. The structure prior visual 3D reconstruction method according to claim 1, characterized in that: Optimizing the current parameter set through the differentiable rendering model includes: Get the current parameter set, which includes the location parameters, orientation parameters, size parameters, color parameters, border parameters, and texture parameters of the primitive instance; Get the world coordinates of the primitive instance in the three-dimensional world according to the position parameter, orientation parameter and size parameter; Based on the camera parameters and world coordinates, several primitive instances are projected onto the two-dimensional image plane to obtain the 2D polygonal outline of the primitive instance on the image; Determine the pixels occluded by the 2D polygon outline, determine the occluded pixels in the base image through the pixels occluded by the 2D polygon outline, and calculate the final pixel color through the surface normal vector, light source direction and surface color; Calculate the gradient of the loss function for each element in the parameter set of the primitive instance and adjust the parameters within the element according to the gradient.

3. The structure prior visual 3D reconstruction method according to claim 2, characterized in that: The obtaining of the world coordinates of the primitive instance in the three-dimensional world according to the position parameter, the orientation parameter and the size parameter comprises: Calculate local world coordinates based on the shape of the primitive instance; Combine position parameters, orientation parameters and size parameters into a transformation matrix; Multiply the transformation matrix and the local world coordinates to get the world coordinates of the 3D world.

4. A structure prior visual 3D reconstruction system, characterized by: include: The model training module is configured to obtain a number of standard images of the modeled object, obtain basic parameters of primitive instances in the standard images, generate parameter sets of the primitive instances, construct a differentiable rendering model, and train the differentiable rendering model through the parameter sets and the basic images, including S201: obtaining the parameter set of the primitive instance, and generating a synthetic image through the differentiable rendering model based on the parameter set of the primitive instance; S202: comparing the synthetic image with the standard image and calculating one or more loss functions; S203: using the automatic differentiation framework to backpropagate the loss through the rendering pipeline to obtain the gradient of the loss function with respect to each optimizable parameter of each primitive instance; S204: using the gradient descent optimizer to update the parameter sets of all primitive instances according to the gradient; S205: looping the updated parameter set starting from step S201, recording the number of iterations and setting an upper limit on the number of iterations, and when the upper limit on the number of iterations is reached, outputting the current differentiable rendering model and the trained differentiable rendering model; an optimization module configured to obtain a current base image, obtain basic parameters of a target primitive instance in the base image, generate a parameter set, optimize the current parameter set using a differentiable rendering model, and output a target parameter set; a list establishment module configured to establish a mapping relationship between a current target parameter set and a corresponding target primitive instance to generate a primitive instance target group, save a plurality of primitive instance target groups and category labels of the primitive instances, and generate a primitive instance list, wherein the primitive instance list constitutes the modeled object, including setting an error threshold. If the pixel loss value between the current synthetic image and the standard image is greater than the error threshold, or if there is a missing structure area in the synthetic image, a new primitive instance is added and the replaced primitive instance is removed; Adding a new primitive instance and removing the replaced primitive instance includes: Initiate candidate primitive selection process; The candidate primitive selection process includes running a pre-trained instance segmentation model on the original input image corresponding to the error region, identifying each primitive instance in the image through the instance segmentation model, and outputting a category label and a mask of the primitive instance; If the category label of a recognized primitive instance matches the category label of a primitive instance in the primitive instance list library, the matched primitive instance is replaced by a primitive instance in the primitive instance list library; When a primitive instance is replaced and the pixel loss value is still greater than the error threshold, or there is still a missing structure area in the synthesized image, a proliferation 3DGS ellipsoid is added to the primitive instance to fit the area where the pixel loss value of the primitive instance is still greater than the error threshold or the missing structure area.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for visual three-dimensional reconstruction based on structural prior is implemented as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements a structure-prior visual three-dimensional reconstruction method according to any one of claims 1 to 3.

Images