A high-fidelity real scene three-dimensional model lightweight method, device and equipment

Abstract

This invention discloses a method, apparatus, and device for lightweighting high-fidelity real-scene 3D models. The method includes: stitching together one or more tiles of a real-scene 3D model to form a complete 3D model G and resolving it into a vertex set V and a face set T; generating a low-resolution real-scene 3D model G' from the original model G as an initial state; and constructing a joint evaluation function for appearance and geometry. Loss This method is used to evaluate the topological quality of G' and its geometrical distance and appearance similarity to G; appearance images of G and G' are simultaneously captured from different viewpoints using a graphics rendering pipeline; and a differentiable rendering pipeline is used to... Loss The gradient is passed to the geometry and position of G' for iterative optimization until... Loss Value convergence generates a high-fidelity, lightweight, realistic 3D model. This invention achieves high-fidelity lightweight reconstruction of non-manifold realistic 3D models, enabling rapid lightweight reconstruction even on ultra-large-scale models.

Description

Technical Field

[0001] This invention belongs to the field of surveying and mapping data processing, and in particular relates to a method, apparatus and equipment for lightweighting high-fidelity real-scene 3D models. Background Technology

[0002] With the rapid development of drone technology and optical sensor technology, airborne LiDAR and oblique photogrammetry technologies can now accurately acquire large-scale three-dimensional surface information. As a high-precision representation integrating geometry, structure, and appearance, real-world 3D models possess accurate 3D geometric data and highly realistic visual effects. They can precisely recreate urban buildings, topography, and landscape environments, providing a foundation for in-depth analysis in multiple fields such as urban planning, environmental monitoring, and disaster simulation, and becoming an important data support for virtual geographic environments and digital twin cities.

[0003] Compared to 3D models in computer graphics, real-world 3D models boast centimeter-level precision and contain a vast number of vertices and polygons, resulting in massive data volumes and high storage and transmission costs. Furthermore, the sheer volume of data significantly increases the computational resource demands during geographic analysis, often leading to inefficient rendering and analysis, especially in resource-constrained scenarios such as mobile devices and web services. Therefore, many 3D geographic analysis applications prefer lightweight 3D models, prioritizing high-quality textures over high-density geometric polygons during rendering. In scientific simulations, lightweight 3D models require high geometric accuracy to ensure the accuracy of analysis results while maintaining visual consistency for visualization needs. Against this backdrop, achieving high-fidelity lightweight real-world 3D models that maintain a high degree of consistency in geometric details and appearance with the original model has become an increasingly important and pressing issue.

[0004] For lightweighting of real-world 3D models, existing methods mostly rely on geometric error metrics to simplify the model. Although this can effectively reduce the number of geometric elements in the 3D model, it is difficult to ensure high fidelity with the original model. The main problems are as follows: the simplification method driven by geometric error metrics ignores the consistency of the model's appearance, resulting in a significant difference between the simplified model and the original model in visual perception. This fails to meet the dual requirements of geometric accuracy and appearance consistency in scientific simulation applications. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, this invention proposes a method, apparatus and equipment for lightweighting high-fidelity real-scene 3D models. It adopts a joint "appearance-geometry" driving approach, and constructs a model with a high lightweight ratio while ensuring high appearance and high geometric consistency with the original model. This is of great value for reducing the volume redundancy of real-scene 3D models and optimizing their storage, transmission and analysis efficiency.

[0006] To address the problems of the prior art, the present invention is achieved through the following technical solution:

[0007] A method for lightweighting high-fidelity real-world 3D models includes the following steps:

[0008] Step 1, Preprocessing of the Realistic 3D Model:

[0009] Piecing together one or more tiles of a real-world 3D model to form a complete original real-world 3D model G, and resolving it into a single... A tuple, wherein the set of vertices is The set of face elements is ,and The number of vertices. The number of geometric elements;

[0010] Step 2, Low-resolution realistic 3D model Construction:

[0011] Randomly and uniformly sample from the vertex set of the original real-world 3D model G Each vertex serves as a low-resolution realistic 3D model. initial vertex position set In a given set of points The above uses a triangulation method to generate a triangular mesh, which is then minimized. Deriving the Poisson equation Discretized into a sparse linear system on a finite basis function space After solving, the isosurface is extracted to obtain 3D surface element set This results in a rough, low-resolution 3D model of the real scene. This serves as the initial state for the final lightweight real-world 3D model.

[0012] Step 3: Construct a joint evaluation function Loss for appearance and geometry to evaluate low-resolution realistic 3D models. Topological quality, and low-resolution realistic 3D models Geometric distance and appearance similarity with the original real-world 3D model G;

[0013] Step 4, Appearance Similarity Perception Between Models: Using the graphics rendering pipeline, the original realistic 3D model G and the low-resolution model are simultaneously compared from different camera viewpoints and directions. Take a picture and obtain an image pair (I, I');

[0014] Step 5: Optimize the low-resolution 3D model Appearance and Geometry: Computing Low-Resolution 3D Models The joint evaluation function Loss between the model and the original realistic 3D model G is used to transfer the gradient of the Loss value to the low-resolution 3D model via a differentiable rendering pipeline. In terms of geometry and location, based on the gradient of the joint loss function Loss, the model is... The position, geometry, and possible normal information are updated to gradually approximate the appearance and geometric features of the original model G;

[0015] Step 6: Iterate through steps 4 and 5 until a low-resolution 3D model is achieved. The value of the joint evaluation function Loss between the model and the original realistic 3D model G is less than a specified threshold or the number of iterations reaches a specified threshold, thereby generating the final high-fidelity lightweight realistic 3D model.

[0016] Preferably, the joint evaluation function constructed in step 3 is:

[0017] ,

[0018] in, There are two images. To evaluate low-resolution real-world 3D models The similarity between the appearance of the original real-world 3D model G and the actual scene model G from different angles. Evaluation of low-resolution real-world 3D models Geometric similarity between the model and the original real-world 3D model G Used to evaluate low-resolution realistic 3D models Mesh topology quality, , and These are three weighted terms.

[0019] Preferably, geometrical position similarity

[0020] ,

[0021] in, and These are the corresponding points on the two models, i.e. For the model Above and The point with the shortest distance, where N is the number of sampling points.

[0022] Preferably, appearance similarity

[0023] ,

[0024] in, It is calculated by weighting the structural similarity index and the L1 pixel loss, with the weights being: and The L1 pixel loss can be calculated using the mask norm:

[0025] ,

[0026] ,

[0027] in, For brightness, For contrast, It is a structural item.

[0028] Preferably, the mesh topology quality

[0029] ,

[0030] in, Each is regularized by Laplace. and side length regularization term Weighted composition, The weights of the Laplace regularization term. The weight of the side length regularization term.

[0031] More preferably, the formula for calculating the Laplace regularization term is as follows:

[0032] ,

[0033] yes a vertex Laplace coordinates Let v be the set of vertices adjacent to vertex v; the edge length regularization term is used to prevent excessively long edges, and its calculation formula is: .

[0034] Preferably, the differentiable rendering pipeline in step 5 performs a series of differentiable rendering operations.

[0035] ,

[0036] Among them, Rasterize is the rasterization operation, Interpolate is the interpolation operation, Texturing is the texture sampling operation, and AntiAlias ​​is the anti-aliasing operation.

[0037] A device for lightweighting a real-scene 3D model according to any one of the preceding claims, comprising the following modules:

[0038] Real-world 3D model preprocessing module: Used to input a real-world 3D model, stitch together one or more tiles of the real-world 3D model to form a complete original real-world 3D model G, and parse it into a... tuple; Low-resolution model initial state generation module: Used to extract, sample, or reconstruct a coarse, inaccurate low-resolution realistic 3D model from the original realistic 3D model G. This serves as the initial state for the final lightweight real-world 3D model.

[0039] Joint evaluation module for appearance and geometry: used to evaluate low-resolution realistic 3D models. The topological quality of G, and its geometrical distance and appearance similarity to the original real-world 3D model;

[0040] Model Appearance Similarity Perception Module: This module collects appearance information from multiple viewpoints on two 3D models. Specifically, it uses the spatial position and orientation of a preset set of camera viewpoints to compare the appearance of the original real-world 3D model G and the low-resolution real-world 3D model G. Simultaneous shooting is performed to generate two corresponding sets of appearance images; these sets are used for appearance loss calculation, thereby ensuring the low-resolution real-scene 3D model... The original model can be reproduced well from multiple perspectives. Texture, lighting, and structural details;

[0041] Appearance and geometry iteration optimization module: used for iterative calculation of low-resolution 3D models. The joint evaluation function Loss of G between the model and the original realistic 3D model is calculated, and the gradient of the Loss value of the evaluation function is transferred to the low-resolution 3D model using a differentiable rendering pipeline. In terms of geometry and position, thereby optimizing The appearance and geometry are used to ultimately generate a high-fidelity, lightweight, realistic 3D model. Output module: Used to input the path where the model is saved, and output a high-fidelity, lightweight, realistic 3D model.

[0042] An electronic device includes a distributed memory, a processor, and a computer program in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the high-fidelity real-scene 3D model lightweighting method described above.

[0043] A computer-readable storage medium storing computer software instructions that implement the steps of a high-fidelity lightweight method for real-world 3D models as described in any of the preceding claims.

[0044] Beneficial effects:

[0045] Compared with existing technologies, the present invention provides a method, apparatus, and device for lightweighting high-fidelity real-scene 3D models, which has the following advantages:

[0046] (1) To address the problems of appearance distortion and oversimplification of detailed features in the simplification of real-world 3D models at high lightweight ratios, this invention proposes a "appearance-geometry" driven high-fidelity lightweight method system to achieve high-fidelity lightweighting of non-manifold real-world 3D models.

[0047] (2) Existing 3D model simplification methods have exponentially increased computation time when the number of geometric elements in the original real scene 3D model is too large. The computation time of the present invention is independent of the number of geometric elements in the original real scene 3D model and the number of geometric primitives to be removed. It can achieve rapid lightweight model reconstruction on ultra-large-scale original real scene 3D models.

[0048] (3) The lightweight model generated by the present invention can still generate a lightweight model with an appearance and geometry highly consistent with the original real scene 3D model when the lightweight ratio is only 1%, with visual errors all below 0.015 and geometric errors better than 0.001. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 This is a flowchart of a method for lightweighting a high-fidelity real-scene 3D model according to an embodiment of the present invention.

[0051] Figure 2 These are three real-world 3D models used in the embodiments of this invention.

[0052] Figure 3 This is the result of high-fidelity lightweighting of three real-scene 3D models in this embodiment of the invention.

[0053] Figure 4 This is an example of how a real-world 3D model changes with the number of iterations during a high-fidelity lightweighting process in an embodiment of the present invention.

[0054] Figure 5 This is a high-fidelity lightweighting result of a real-scene 3D model in an embodiment of the present invention at different lightweighting ratios.

[0055] Figure 6 This is a comparison of the geometric structure of a real-world 3D model before and after weight reduction in an embodiment of the present invention.

[0056] Figure 7This is a module diagram of a high-fidelity real-scene 3D model lightweight device in another embodiment of the present invention.

[0057] Figure 8 This is a structural block diagram of a high-fidelity, lightweight real-scene 3D model device according to another embodiment of the present invention.

[0058] Figure 9 This is a data processing schematic diagram of a high-fidelity real-scene 3D model lightweight device in another embodiment of the present invention. Detailed Implementation

[0059] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0060] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this disclosure will be apparent to those skilled in the art.

[0061] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0062] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0063] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0064] However, it should be understood that the disclosed embodiments are merely examples of this disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure this disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use this disclosure in a variety of substantially any suitable detailed structures.

[0065] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0066] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0067] Example 1

[0068] like Figure 1 As shown, this embodiment of the invention provides a method for lightweighting high-fidelity real-scene 3D models, including the following steps:

[0069] S1, Model Preprocessing and Parsing:

[0070] Reading Figure 2 The OBJ format reality 3D model shown (osgb and 3DTiles formats can also be selected in practical applications) is used to stitch together one or more tiles of the reality 3D model to form a complete original reality 3D model G, and then parsed into a single image. A tuple, wherein the set of vertices is The set of face elements is . The number of vertices. This represents the number of geometric facets. Each facet stores its vertex coordinates and texture mapping coordinates.

[0071] S2, Low-resolution realistic 3D model Construction:

[0072] Extracting, sampling, or reconstructing a coarse, inaccurate, low-resolution realistic 3D model from the original realistic 3D model G. This serves as the initial state for the final lightweight, realistic 3D model.

[0073] In this embodiment, for The texture appearance can use random values ​​as its texture; for vertex set The values ​​can be obtained by first randomly and uniformly sampling from the vertex set of the original real-world 3D model G. Each vertex as The initial vertex position; for Surface element set Values, in a given set of points Above, triangulation methods can be used to generate geometrically excellent triangular meshes, by minimizing... Deriving the Poisson equation

[0074] ,

[0075] Discretize into a sparse linear system on a finite basis function space. After solving, the isosurfaces are extracted to obtain a set of three-dimensional surface elements. The process of generating a triangular network using the Poisson equation requires... Normalization and standardization processes are performed, and reconstruction depth and scale are set to ensure the geometric stability and triangle quality of the generated mesh. The reconstruction depth is set to 9 and the reconstruction scale is set to 1.1.

[0076] S3, the joint evaluation function Loss for appearance and geometry: used to evaluate low-resolution realistic 3D models. The loss function is constructed based on the topological quality of the model, as well as its geometric distance and appearance similarity to the original real-world 3D model G.

[0077] ,

[0078] in, There are two images. To evaluate low-resolution real-world 3D models The similarity between the appearance of the original real-world 3D model G and the actual scene model G from different angles. Evaluation of low-resolution real-world 3D models Geometric similarity between the model and the original 3D model G Used to evaluate low-resolution realistic 3D models Mesh topology quality, , and These are three weighted terms. , and The value of is selected through manual experience, and the range is 0-1. In this embodiment, it is set to 0.5, 0.4 and 0.1.

[0079] Specifically, the data for S3 is calculated as follows:

[0080] S3-1, Calculate a low-resolution real-world 3D model The geometric positional differences between G and the original 3D model G will be considered when comparing G and G. Input to geometric location similarity In the function, the formula is:

[0081] ,

[0082] in and These represent the corresponding points on the two models, and N is the number of sampling points, which is set to 8500 in this embodiment. for On the surface of the model, away The nearest point.

[0083] S3-2, Calculate a low-resolution real-world 3D model The topological quality will Input to function

[0084] ,

[0085] in, Each is regularized by Laplace. and side length regularization term Weighted composition, The weights of the Laplace regularization term. The weights for the edge length regularization term are 0.7 and 0.3. The Laplace regularization term penalizes vertices for excessively free movement, ensuring adjacent vertices have similar movement distances, thus preventing model self-intersection and excessive model deformation during each iteration. The formula for calculating the Laplace regularization term is:

[0086] ,

[0087] yes a vertex Laplace coordinates Let v be the set of adjacent vertices of vertex v; the edge length regularization term is used to prevent excessively long edges, and its calculation formula is: .

[0088] S3-3, Calculate a low-resolution 3D model of a real-world scene. Differences in appearance between G and the original realistic 3D model G: G and Input to appearance difference function

[0089] ,

[0090] in, It is calculated by weighting the structural similarity index and the L1 pixel loss, with the weights being: and In this embodiment, the values ​​are 0.5 and 0.5, and the L1 pixel loss can be calculated from the mask norm.

[0091] ,

[0092] ,

[0093] in, For brightness, For contrast, It is a structural item.

[0094] S4, Appearance Perception Module Construction: Employing a graphics rendering pipeline, low-resolution realistic 3D models are rendered under identical viewing conditions. Based on the original real-world 3D model G, generate several sets of required image pairs. In the computer graphics rendering pipeline, to achieve the conversion from a 3D model to a 2D image, the parameter mvp needs to be determined to convert the model's vertex coordinates... Switch to clipping space , as in the formula

[0095] ,

[0096] ,

[0097] in, , and These represent the projection transformation matrix, view transformation matrix, and model transformation matrix, respectively. Simultaneous observation under the same conditions... That is, before rasterization, the same method is used. Will The vertex coordinates are transformed to clip space. Since it's single-model rendering rather than scene rendering, therefore... It can be an identity matrix. A perspective projection matrix is ​​used and remains constant throughout the training process. And for Matrix, determined by the observation position Observe the direction and the given direction on the coordinates , as in the formula

[0098] ,

[0099] in, The function calculates the view transformation matrix based on the input eye, at, and up, specifically involving the calculation of vectors, rotation matrices, and translation matrices. It observes the real-world 3D model from different positions. and Each time an image is observed, a new image of the current state is generated. Sample images, and corresponding The labeled image at that observation angle. For the orientation of the observation, i.e., the camera's gaze point, to prevent overlearning and optimize features at specific locations, images can be randomly selected from... vertex set A point v is selected as the gaze point for each observation, i.e. ,in, It indicates a uniform distribution.

[0100] S Differentiable rendering backpropagation loss: calculating low-resolution 3D models After applying the Loss of the joint evaluation function G between the original 3D model and the model, a differentiable rendering pipeline is used to transfer the gradient of the Loss value of the evaluation function to the low-resolution 3D model. In terms of geometry and position, the differentiable rendering pipeline used in this embodiment is the nvdiffrast tool library. The nvdiffrast differentiable rendering pipeline performs a series of differentiable rendering operations of this invention:

[0101] ,

[0102] It consists of rasterization, interpolation, texture sampling, and anti-aliasing. Rasterization: The rasterization operation is used to map a 3D model located in clip space to a 2D plane.

[0103] The output of rasterization is a four-channel image, where each pixel is represented by a 4-tuple (u, v, z / w, tid), where tid represents the ID of the geometric primitive in which the pixel resides, u and v represent the pixel's position within that primitive, and z / w represents the normalized depth value. Interpolate: The interpolation operation interpolates vertex material properties, such as texture coordinates, into image space. The output of the interpolation operation is a multi-channel image with the same dimensions as the rasterized image; the number of channels depends on the number of material properties. Texturing: The texture sampling operation retrieves the corresponding texture value from the texture image based on the texture coordinates of each pixel. AntiAlias: The anti-aliasing operation optimizes the rendering image for discontinuities at contour edges.

[0104] S6, optimizes low-resolution 3D models Appearance and Geometry: Depending on the value of the joint evaluation function Loss, stochastic gradient optimization or the Adam optimizer can be used. Based on the value of Loss and the loss gradient of differentiable rendering backpropagation, the optimization is automatically performed. The appearance and geometry were adjusted.

[0105] S7, iterate through S4-S6 until a condition is met. This condition can have two thresholds, such as the joint loss value falling below a specified threshold, or the number of iterations reaching a specified threshold. In this embodiment, the condition is reaching the maximum iteration threshold of 10000, thus obtaining the following... Figure 3 The high-fidelity lightweight model shown. Then... The geometric and visual errors relative to G are calculated to evaluate the quality of the lightweight method, where the geometric error can be expressed as... Calculations can be made using visual errors. In this embodiment, calculations show that the visual error is less than 0.015 and the geometric error is better than 0.001.

[0106] Low-resolution 3D model The appearance and geometry change with the number of iterations, as follows Figure 4 As shown.

[0107] Different low-resolution real-world 3D models The impact of the number of vertices (or lightweight ratio) on the final result is as follows: Figure 5 As shown.

[0108] Low-resolution realistic 3D model The geometry and the comparison with the original real-world 3D model, for example Figure 6 As shown.

[0109] Example 2

[0110] like Figure 7 As shown, another embodiment of the present invention also provides a high-fidelity lightweight real-scene 3D model device. This device mainly consists of a real-scene 3D model preprocessing module, a low-resolution model initial state generation module, a joint evaluation module for appearance and geometry, a model appearance similarity perception module, an appearance and geometry iterative optimization module, and an output module. These modules can be interconnected via a data transmission channel to jointly generate a high-fidelity lightweight real-scene 3D model.

[0111] Real-world 3D model preprocessing module: This module receives the input real-world 3D model data. The original real-world 3D model is typically stored as several tiles. The preprocessing module first stitches together one or more tiles to obtain a complete real-world 3D model G. Subsequently, through a parsing step, the complex real-world 3D model data structure can be standardized, facilitating calculations and optimizations in subsequent modules.

[0112] Low-resolution model initial state generation module: This module is used to extract, sample, or reconstruct a coarse, inaccurate low-resolution realistic 3D model from the 3D model G. This serves as the initial state for the final lightweight real-world 3D model.

[0113] Joint evaluation module for appearance and geometry: This module is used for evaluating low-resolution realistic 3D models. The overall quality of the model is evaluated. The evaluation includes topological quality assessment, geometric distance assessment, and appearance similarity assessment. Topological quality assessment checks for topological issues such as non-manifold edges, holes, and flipped triangles to ensure the model's integrity and renderability. Geometric distance assessment uses methods based on grid alignment or corresponding point distances to calculate the model's... and Deformation errors in three-dimensional space, such as point-to-point distance and the closest distance from a point to a surface, are used to reflect geometric errors; appearance similarity evaluation uses the original model Compared with low-resolution models Images are rendered and generated from the same viewpoint, and their appearance consistency is measured using structural similarity (SSIM), L1 difference, and pixel difference. Finally, this module outputs a joint evaluation function, Loss, to describe the model. The degree of deviation in appearance and geometry.

[0114] Model Appearance Similarity Perception Module: This module is mainly used to collect appearance information of two 3D models from multiple viewpoints. Specifically, the module uses the spatial position and orientation of a preset set of camera viewpoints to compare the appearance of the original 3D model G and the low-resolution model. Simultaneous image capture is performed, generating two corresponding sets of appearance images. These image sets are used for appearance loss calculation, thereby ensuring the model's appearance loss is accurate. The original model can be reproduced well from multiple perspectives. The texture, lighting, and structural details.

[0115] Appearance and geometry iteration optimization module: used for iterative calculation of low-resolution 3D models. The joint evaluation function Loss of G between the original 3D model and the low-resolution 3D model is calculated, and the gradient of the evaluation function Loss value is transferred to the low-resolution 3D model using a differentiable rendering pipeline. In terms of geometry and location, end-to-end model optimization is achieved; based on the gradient of the joint loss function Loss, the model is optimized. The module updates the position, geometry, and possible normal information of the model, gradually approximating the appearance and geometric features of the original model G. After a sufficient number of iterations, or when the loss reaches a specified threshold, the module outputs an optimized, lightweight 3D model that highly approximates the original model G in both appearance and geometry.

[0116] Output Module: This module stores the optimized lightweight realistic 3D model to a specified path. Users can input the save directory to export the final high-fidelity lightweight model in a preset format, such as OBJ, PLY, glTF, b3dm, or a custom format, for subsequent visualization, rendering, or system deployment.

[0117] The real-scene 3D model preprocessing module includes:

[0118] Input unit: Used to read the tile file of the original real-world 3D model file based on the path of the user input; when the 3D model is stored in the form of paginated LOD (Level of Detail), the input unit recursively traverses the model directory, automatically identifies and obtains all leaf node tiles to obtain the highest resolution model data;

[0119] Stitching unit: Used to perform unified transformation and stitching of information such as vertex coordinates, normal vectors, and texture coordinates of multiple input tile models, stitching all tiles into a complete and continuous 3D model G;

[0120] Parsing unit: Used to parse the stitched 3D model G into a {V, T} tuple, where V is the set of vertices and T is the set of face elements. Each face element contains basic information such as vertex index, vertex coordinates, and texture coordinates.

[0121] Output unit: Used to output the complete 3D model {V, T} generated by the analysis, for use by subsequent modules.

[0122] The joint evaluation module for appearance and geometry includes:

[0123] Input unit: Used to input the original 3D model G and the low-resolution 3D model. and related parameters for joint evaluation;

[0124] Geometric evaluation unit: used for calculation Geometric quality indicators such as topological consistency with G, degree of local deformation, and point / surface distance error;

[0125] Appearance evaluation unit: used to calculate based on differentiable rendering results or appearance projection images. The visual differences between G and G include structural similarity in brightness (SSIM), pixel difference (L1), etc.

[0126] Joint evaluation calculation unit: used to integrate geometric evaluation indicators and appearance evaluation indicators to construct a joint evaluation function Loss, which reflects the... Compared to G's overall realism;

[0127] Output unit: Used to output the joint evaluation function Loss and its intermediate indices.

[0128] The inter-model appearance similarity perception module includes:

[0129] Input unit: Used to input the original model G and the low-resolution model. And camera shooting parameters, including viewpoint position, direction, focal length, resolution, etc.;

[0130] Rendering and shooting unit: used to simultaneously capture images from multiple camera perspectives. The image was captured and rendered by G to obtain two sets of corresponding appearance rendering images;

[0131] Appearance feature extraction unit: used to extract features such as brightness, texture, and structure from the captured image, providing basic data for the evaluation module's calculations;

[0132] Output unit: Used to output the original model image set and the low-resolution model image set.

[0133] The low-resolution 3D model appearance and geometry optimization module includes:

[0134] Input unit: Used to input the original model G and the low-resolution model. The calculation parameters of the joint evaluation function Loss;

[0135] Gradient calculation unit: Based on the differentiable rendering pipeline, it performs forward rendering and backpropagation on the joint evaluation function Loss, and calculates the Loss with respect to... Gradients of attributes such as vertex position, normal vector, and texture coordinates;

[0136] Optimize the solution unit: Adjust iteratively using optimization methods (such as Adam, SGD, etc.) based on gradient information. Its geometric structure and appearance properties make It gradually approximates G in appearance and geometry;

[0137] Model update unit: Used to update the low-resolution model with optimized vertex, texture, or pixel information. ;

[0138] Output unit: Outputs the final high-fidelity, lightweight, realistic 3D model. .

[0139] like Figure 8 and Figure 9 As shown, another embodiment of the present invention also provides a high-fidelity lightweight device for real-world 3D models. The device includes a distributed memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for extracting buildings from real-world 3D models, including:

[0140] Step 1: Preprocessing the real-world 3D model. This involves stitching together one or more tiles of the real-world 3D model to form a complete 3D model G, and then parsing it into a single... A tuple, wherein the set of vertices is The set of face elements is . The number of vertices. The number of geometric elements;

[0141] Step 2, initial state estimation of the low-resolution model: extract, sample, or reconstruct a coarse, inaccurate low-resolution real-scene 3D model from the original realistic 3D model G. This serves as the initial state for the final lightweight real-world 3D model.

[0142] Step 3: Construct a joint evaluation function Loss for appearance and geometry to evaluate low-resolution realistic 3D models. The topological quality of the model, as well as its geometrical distance and appearance similarity to the original 3D model G;

[0143] Step 4: Perceiving appearance similarity between models. Using the graphics rendering pipeline, the original 3D model G and the low-resolution model are simultaneously evaluated from different camera viewpoints and directions. Take photos to obtain a series of exterior images;

[0144] Step 5: Optimize the low-resolution 3D model Appearance and geometry, calculating low-resolution 3D models The joint evaluation function Loss between the model and the original 3D model G is used to transfer the gradient of the Loss value to the low-resolution 3D model via a differentiable rendering pipeline. In terms of geometry and location, based on the gradient of the joint loss function Loss, the model is... The position, geometry, and possible normal information are updated to gradually approximate the appearance and geometric features of the original model G;

[0145] Step 6: Iterate through steps 4 and 5 until a low-resolution 3D model is achieved. The joint evaluation function Loss between the model and the original realistic 3D model G has a very small value, thus generating a final high-fidelity lightweight realistic 3D model.

[0146] It should be noted that the high-fidelity lightweight real-scene 3D model simplification device provided in the above embodiments is only illustrated by the division of the above-described program modules when performing lightweighting or simplification of real-scene 3D models. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the specific implementation process of the high-fidelity lightweight real-scene 3D model simplification device and method provided in the above embodiments is detailed in the method embodiments and will not be repeated here.

[0147] The memory in this embodiment of the invention is used to store various types of data to support the operation of the high-fidelity, lightweight, real-world 3D model device. Examples of such data include any computer program used to operate on the high-fidelity, lightweight, real-world 3D model electronic device.

[0148] The high-fidelity lightweight method for real-world 3D models disclosed in this invention can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the high-fidelity lightweight method for real-world 3D models can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in this invention's embodiments. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention's embodiments can be directly represented as execution by a hardware decoding processor, or as execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically a memory. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the high-fidelity lightweight method for real-world 3D models provided in this invention's embodiments.

[0149] In an exemplary embodiment, the high-fidelity real-world 3D model lightweight device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0150] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0151] In an exemplary embodiment, the present invention also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory including a computer program. This computer program can be executed by the processor of the high-fidelity real-scene 3D model lightweighting device, performing the steps of the high-fidelity real-scene 3D model lightweighting method. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0152] In summary, the high-fidelity real-scene 3D model lightweighting method, apparatus, and equipment of the present invention achieve high-fidelity, low-complexity simplification and reconstruction of real-scene 3D models. The number of vertices and face elements in the reconstructed lightweight real-scene 3D model is significantly reduced compared to the original model, while still maintaining a highly consistent appearance and geometric shape with the original real-scene 3D model. Furthermore, it has a very high degree of automation, eliminating the need for manual labeling and significantly reducing the storage volume and network transmission bandwidth of the real-scene 3D model.

[0153] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0154] It should be understood that the above description of the embodiments is quite detailed and should not be construed as limiting the scope of protection of this invention. Those skilled in the art can make various modifications and variations based on the teachings of this invention. Any modifications, equivalent substitutions, improvements, etc., made without departing from the scope of protection of the claims of this invention fall within the protection scope of this invention, and the scope of protection claimed by this invention should be determined by the appended claims.

Claims

1. A method for lightweighting high-fidelity real-scene 3D models, characterized in that, Includes the following steps: Step 1, Preprocessing of the Realistic 3D Model: Piecing together one or more tiles of a real-world 3D model to form a complete original real-world 3D model G, and resolving it into a single... A tuple, wherein the set of vertices is The set of face elements is ,and The number of vertices. The number of geometric elements; Step 2, Low-resolution realistic 3D model Construction: Randomly and uniformly sample from the vertex set of the original real-world 3D model G Each vertex serves as a low-resolution realistic 3D model. initial vertex set position In a given set of points The above uses a triangulation method to generate a triangular mesh, which is then minimized. Deriving the Poisson equation Discretized into a sparse linear system on a finite basis function space After solving, the isosurface is extracted to obtain 3D surface element set This results in a rough, low-resolution 3D model of the real scene. This serves as the initial state for the final lightweight real-world 3D model. Step 3, Construct the joint evaluation function Loss for appearance and geometry: used to evaluate low-resolution realistic 3D models. Topological quality, and low-resolution realistic 3D models Geometric distance and appearance similarity with the original real-world 3D model G; Step 4, Appearance Similarity Perception Between Models: Using the graphics rendering pipeline, the original 3D model G and the low-resolution model are simultaneously compared from different camera viewpoints and directions. Take a picture and obtain an image pair (I, I'); Step 5: Optimize the low-resolution 3D model Appearance and Geometry: Computing Low-Resolution 3D Models The joint evaluation function Loss between the model and the original realistic 3D model G is used to transfer the gradient of the Loss value to the low-resolution 3D model via a differentiable rendering pipeline. In terms of geometry and location, based on the gradient of the joint loss function Loss, the model is... The position, geometry, and possible normal information are updated to gradually approximate the appearance and geometric features of the original model G; Step 6: Iterate through steps 4 and 5 until a low-resolution 3D model is achieved. The value of the joint evaluation function Loss between the model and the original realistic 3D model G is less than a specified threshold or the number of iterations reaches a specified threshold, thereby generating the final high-fidelity lightweight realistic 3D model.

2. The method for lightweighting a high-fidelity real-scene 3D model according to claim 1, characterized in that, The joint evaluation function constructed in step 3 is: ， in, There are two images. To evaluate low-resolution realistic 3D models The similarity between the appearance of the original real-world 3D model G and the actual scene model G from different angles. Evaluation of low-resolution real-world 3D models Geometric similarity between the model and the original real-world 3D model G Used to evaluate low-resolution realistic 3D models Mesh topology quality, , and These are three weighted terms.

3. The method for lightweighting a high-fidelity real-scene 3D model according to claim 2, characterized in that, Geometric position similarity ， in, and These are the corresponding points on the two models, i.e. For the model Above and The point with the shortest distance, where N is the number of sampling points.

4. The method for lightweighting a high-fidelity real-scene 3D model according to claim 2, characterized in that, Appearance similarity ， in, It is calculated by weighting the structural similarity index and the L1 pixel loss, with the weights being: and The L1 pixel loss can be calculated using the mask norm. ， ， in, For brightness, For contrast, It is a structural item.

5. The method for lightweighting a high-fidelity real-scene 3D model according to claim 2, characterized in that, Mesh topology quality ， Each is regularized by Laplace. and side length regularization term Weighted composition, The weights of the Laplace regularization term. The weight of the side length regularization term.

6. The method for lightweighting a high-fidelity real-scene 3D model according to claim 5, characterized in that, The formula for calculating the Laplace regularization term is as follows: ， yes a vertex Laplace coordinates Let v be the set of vertices adjacent to vertex v; the edge length regularization term is used to prevent excessively long edges, and its calculation formula is: .

7. The method for lightweighting a high-fidelity real-scene 3D model according to claim 1, characterized in that, The differentiable rendering pipeline described in step 5 performs a series of differentiable rendering operations. ， Among them, Rasterize is the rasterization operation, Interpolate is the interpolation operation, Texturing is the texture sampling operation, and AntiAlias ​​is the anti-aliasing operation.

8. A device for lightweighting a real-scene 3D model, which implements the method for lightweighting a real-scene 3D model according to any one of claims 1-7, characterized in that, Includes the following modules: Real-world 3D model preprocessing module: Used to input a real-world 3D model, stitch together one or more tiles of the real-world 3D model to form a complete original real-world 3D model G, and parse it into a... tuple; Low-resolution model initial state generation module: Used to extract, sample, or reconstruct a coarse, inaccurate low-resolution realistic 3D model from the original realistic 3D model G. This serves as the initial state for the final lightweight real-world 3D model. Joint evaluation module for appearance and geometry: used to evaluate low-resolution realistic 3D models. The topological quality of G, and its geometrical distance and appearance similarity to the original real-world 3D model; Model Appearance Similarity Perception Module: This module collects appearance information from multiple viewpoints of two 3D models. Specifically, it uses the spatial position and orientation of a preset set of camera viewpoints to compare the appearance of the original real-world 3D model G and the low-resolution real-world 3D model G. Simultaneous shooting is performed to generate two corresponding sets of appearance images; these sets are used for appearance loss calculation, thereby ensuring the low-resolution real-scene 3D model... The original model can be reproduced well from multiple perspectives. Texture, lighting, and structural details; Appearance and geometry iteration optimization module: used for iterative calculation of low-resolution 3D models. The joint evaluation function Loss of G between the model and the original realistic 3D model is calculated, and the gradient of the Loss value of the evaluation function is transferred to the low-resolution 3D model using a differentiable rendering pipeline. In terms of geometry and position, thereby optimizing The appearance and geometry are used to ultimately generate a high-fidelity, lightweight, realistic 3D model. Output module: Used to input the path where the model is saved, and output a high-fidelity, lightweight, realistic 3D model.

9. An electronic device comprising a distributed memory, a processor, and a computer program in the memory and executable in the processor, wherein the processor, when executing the computer program, implements the steps of the high-fidelity lightweight method for real-world 3D models according to any one of claims 1-8.

10. A computer-readable storage medium storing computer software instructions that implement the steps of a high-fidelity lightweight method for real-world 3D models as described in any one of claims 1-8.

Images