A mesh texture simplification algorithm suitable for three-dimensional reconstruction

Abstract

The application discloses a mesh texture simplification algorithm suitable for three-dimensional reconstruction, and comprises the following steps: S1, reference three-dimensional model scene construction; using the recovered scene structure in three-dimensional reconstruction and the calibrated image, performing texture mapping on the original fine three-dimensional mesh to complete the reconstruction of the reference three-dimensional model scene; S2, reference three-dimensional model scene image acquisition; according to the three-dimensional space relative attitude relationship between the reference three-dimensional model scene and the view internal and external parameters, using the inverse projection principle to perform three-dimensional mesh to two-dimensional image rasterization calculation pixel by pixel and view by view, and completing the acquisition of the reference image set; S3, mesh and texture simplification; using the QEM algorithm to simplify the mesh, using the reference image set as a data source and using the texture reconstruction algorithm to perform texture remapping and simplification. The application has the advantages that the algorithm can support mesh and texture simplification together, under different texture simplification parameters, the texture almost has no distortion and can significantly reduce the texture data amount.

Description

Technical Field

[0001] This invention relates to the field of mesh texture simplification technology, and in particular to a mesh texture simplification algorithm suitable for 3D reconstruction. Background Technology

[0002] In recent years, oblique 3D reconstruction technology has overcome the limitations of traditional aerial photography, which can only capture images from a vertical angle. It acquires images from multiple angles and at high resolution, and rapidly reconstructs 3D models using large-scale, automated, and clustered parallel computing, thus becoming a mainstream modeling method. However, with the increasing size of the reconstruction area and the continuous improvement of modeling accuracy, the contradiction between the rapidly growing volume of 3D model data and the limited transmission bandwidth and smooth model rendering has intensified. Mesh simplification is an effective solution.

[0003] Mesh reduction techniques transform high-resolution, accurate original 3D models into coarser approximate mesh models. Currently, scholars both domestically and internationally have made significant progress in this field, with algorithms broadly categorized into two types: geometry-driven reduction algorithms and appearance attribute-driven reduction algorithms. Common geometry-driven reduction algorithms include vertex clustering-based algorithms, vertex extraction-based algorithms, and edge-folding-based algorithms. These algorithms rely solely on geometry in the cost metric, striving to maintain geometric fidelity while reducing vertices, neglecting appearance attributes such as color and texture. When dealing with 3D models containing appearance attributes like texture, the reduction effect is not ideal. In contrast, appearance attribute-driven reduction algorithms consider not only geometry but also appearance attributes, especially texture, to better ensure the appearance of the reduced model. Among these methods, Garland and Heckbert extended their original algorithm by incorporating appearance attributes into the Quadratic Error Metric (QEM); Sporysz et al. performed Canny edge detection on texture maps while considering the ratio of simplified region area to the total area of ​​the 3D mesh, effectively preserving edge information in the mesh; She et al. segmented the surface mesh based on topology and appearance and derived an error metric that considered both geometry and texture, minimizing texture distortion. However, these methods only treat model simplification as a post-processing step in 3D reconstruction and do not consider its close integration with the 3D reconstruction process; furthermore, the simplified models using these methods exhibit some degree of texture distortion, and the lack of texture simplification results in a still large model data volume.

[0004] Appearance attribute-driven simplification algorithms consider both geometric and appearance attributes, especially texture, during the simplification process to better preserve the appearance of the simplified model. Most algorithms use iterative edge folding to simplify the mesh and determine the texture coordinates of each replaced vertex to reduce texture distortion. Garland and Heckbert extended their original algorithm by incorporating attributes into a quadratic error metric (QEM). Cohen et al. further improved their work by introducing a texture deviation metric and finding the texture coordinates of new vertices locally based on this metric. Williams et al. prioritized edge dimensionality reduction based on a perceptual model, considering factors such as texture deviation, lighting contrast, and dynamic lighting. Other algorithms reference texture images or rendered images during mesh simplification. Lindstrom and Turk proposed an image-driven simplification method that compares rendered images of the model before and after simplification from multiple viewpoints and calculates the root mean square error (RMSE) of image pixels, then prioritizes edge folding operations based on the RMSE. Qu and Meyer analyzed the perceptual characteristics of surface signals (e.g., texture, color, light) and used the results to calculate the importance value of each vertex in the model and integrate it into the QEM to guide the simplification process. In addition, Pascual et al. and González et al. proposed a simplified algorithm based on viewpoint entropy and mutual information, which can also reduce texture distortion in the simplified model.

[0005] Other appearance-preserving strategies have been proposed for model simplification with textures. García and Patow proposed a texture technique called Inverse Geometry Texture (IGT), which defines texture coordinates for all vertices in the simplified model to preserve texture details in the high-resolution reference model. Chen and Chuang, as well as Coll and Paradinas, modified the texture image to minimize texture distortion caused by dimensionality reduction at each edge. Notably, the methods of García and Patow, Chen and Chuang, and Coll and Paradinas are not suitable for embedded LOD construction because the texture coordinates or texture image content inherited from vertices change continuously during iterative simplification.

[0006] In mitigating texture distortion and deformation in simplified models, the latest and most effective method in existing research comes from She (She J, Gu X, Tan J, et al. An appearance reserving simplification method for complex 3D building models[J]. Transactions in GIS, 2019, 23), who proposed a novel method for simplifying complex 3D building models, achieving a good balance between geometric fidelity and texture preservation. The basic principle of this method is as follows:

[0007] Step 1: Mesh Segmentation. Based on Breadth-First Search (BFS), the curved mesh is segmented into multiple regions according to geometric similarity and texture features. Based on the segmentation results, a weight is assigned to each edge of the model to enable more simplification operations within the same region;

[0008] Step 2: Cost Calculation and Half-Edge Folding. Traverse the mesh to calculate the initial cost of all edges, and select the edge with the lowest cost as the starting point for half-edge folding simplification. The simplification fully considers the texture information in the model.

[0009] Step 3: Texture Coordinate Update. Adjust the texture coordinates and update the cost of adjacent edges. The method terminates when the simplification rate exceeds a user-specified threshold (simplification rate is defined as the number of triangles deleted divided by the number of triangles in the original model).

[0010] Existing algorithms are mature in mesh simplification, but in texture simplification, they only consider the calculation and updating of the texture coordinates corresponding to the simplified mesh. This can only alleviate texture distortion and deformation to a certain extent, but does not completely solve the problem. Secondly, the algorithm does not simplify the texture content, resulting in a still large amount of data in the simplified model. The following will explain these two aspects in detail:

[0011] (1) Simplified mesh texture distortion and deformation:

[0012] The simplification algorithm performs operations such as mesh vertex deletion, edge folding, and face merging on adjacent faces in 3D space according to certain merging rules (such as quadratic metric formulas). However, due to the fact that the texture patches corresponding to these merged faces are not necessarily continuous in 2D texture space, this discrete distribution phenomenon leads to the inability to correctly calculate the texture coordinates of the newly generated triangular faces. The following combines... Figure 1 and Figure 2 Explanation:

[0013] Based on the existing algorithm (she), the original mesh is... Merge into a simplified grid The merging process takes into account the issues of model texture distortion and deformation: the error metric formula constructed in step 2 of the algorithm tends to merge continuous texture patches, such as... Figure 2 (c) For discretely distributed texture patches, such as Figure 2 (c) The algorithm collects multiple line segment groups before and after folding. If there is no intersection between the line segment pairs, the texture error formula will fail to be calculated. Therefore, it cannot handle this type of case, and the algorithm has certain limitations.

[0014] Meanwhile, in step 3 of the existing algorithm, when adjusting the texture coordinates, the nearest texture coordinate value in the texture space is found and approximated based on the texture coordinate mapping table established before and after simplification of the mesh vertices. For example... Figure 1 (b) Middle piece The vertex to which it belongs Corresponding texture coordinates Then use Figure 1 (a) Mid-section The vertex to which it belongs Corresponding texture coordinates Replacement, note the two here. In texture space, they are very likely not at the same coordinates, therefore... Figure 1 (b) Middle piece The texture patterns are very likely to be distorted or deformed.

[0015] Figure 2 (a) It is the original mesh in 3D space. It consists of 5 adjacent facets; Figure 2 (b) It is a half-fold The simplified grid afterwards; Figure 2 (c) Represents 2D texture space, yes The corresponding pattern in texture space , They are and exist The corresponding point in the middle.

[0016] (2) Simplification of texture content:

[0017] Because existing algorithms directly use the original texture as the texture of the simplified mesh, they only adjust and update the texture coordinates of the relevant vertices. Generally, in a 3D model composed of a mesh and texture, the texture accounts for a large proportion of the total model data. Therefore, the data volume of the simplified model is still very large. Summary of the Invention

[0018] The purpose of this invention is to provide a mesh texture simplification algorithm suitable for 3D reconstruction, thereby solving the aforementioned problems existing in the prior art.

[0019] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0020] A mesh texture simplification algorithm suitable for 3D reconstruction includes the following steps:

[0021] S1. Construction of the reference 3D model scene;

[0022] Using the scene structure recovered in 3D reconstruction and the calibrated image, texture mapping is performed on the original fine 3D mesh to complete the reconstruction of the reference 3D model scene;

[0023] S2. Image acquisition of the reference 3D model scene;

[0024] Based on the relative pose relationship between the reference 3D model scene and the intrinsic and extrinsic parameters of the view, the rasterization calculation from 3D mesh to 2D image is performed pixel by pixel on a view-by-view using the back projection principle to complete the acquisition of the reference image set.

[0025] S3, Mesh and Texture Simplification;

[0026] The QEM algorithm is used to simplify the mesh, and a reference image set is used as the data source. Texture reconstruction algorithm is used to remap and simplify the texture.

[0027] Preferably, step S1 specifically includes the following:

[0028] S11. Select the best view based on the grid patch of the multi-view image;

[0029] The optimal view for each facet in the mesh is obtained by minimizing the energy function of the view using graph cuts and alpha expansion; and texture patches are created to store spatially adjacent faces with the same optimal view into the same texture patch.

[0030] (1)

[0031] (2)

[0032] in, For view The energy function; For data items, , Triangular mesh In the marked image The Soble gradient integral under the given condition represents the node. Select a label image The probability magnitude; For smoothing terms, , indicating adjacent nodes and When selecting images with the same label, the value of the smoothing term is 0; otherwise, it is infinity. For a certain texture patch, It is a facet belonging to the current texture patch. This is the best view corresponding to the current texture patch; It is a sheet of dough; A collection of facets; To and Adjacent facets; and For image, and For numbering;

[0033] S12, Adjust the color at the seam of the texture block;

[0034] Global color adjustment and Poisson editing for local color adjustment were used to adjust the color at the seams of adjacent facets;

[0035] S13, Texture space layout and texture coordinate calculation;

[0036] Based on the projection matrix of the view, the vertex coordinates of the facets stored in the texture patch are back-projected one by one into the view image space, and the bounding box corresponding to the texture patch is calculated. The texture space layout is performed using a two-dimensional box packing algorithm, and the offset value of each bounding box is calculated. The texture coordinate value of each facet is calculated, and the pixel data is extracted to generate the texture of the reference three-dimensional model scene.

[0037] Preferably, step S2 specifically includes the following steps:

[0038] S21. Construct a view based on the intrinsic parameters, extrinsic parameters, and image resolution of the view in the restored scene. The projection matrix, as shown in Equation 3, transforms 3D points in the world coordinate system to pixels in the view image coordinate system; then, an octree index is built for the reference 3D model scene to facilitate fast view searching. Available photos;

[0039] (3)

[0040] in, This is the intrinsic parameter matrix. This is the extrinsic parameter matrix; , These are the x-axis coordinates and y-axis coordinates in the view image coordinate system, respectively; The z-axis coordinate value in the camera coordinate system; The focal length of the camera; , These are the center pixel coordinates; It is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; It is a 3×1 translation vector, which is the offset of the camera coordinate system relative to the world coordinate system; It is a zero vector of size 1×3; These are the components along the x-axis, y-axis, and z-axis in the world coordinate system, respectively.

[0041] S22, Combined View The camera projection matrix is ​​used to create an image with the same resolution as the view image. Depth map and reference image Depth map Located in view In the camera coordinate system, and the depth value is initialized to 0;

[0042] in, and These are the original width and height of the view image, respectively;

[0043] S23, Traversing Views The visible surface is projected using a camera projection matrix. Projecting from 3D space to a 2D reference image And rasterize into triangles ;

[0044] S24, Statistics The occupancy of pixels is determined by extracting color values ​​pixel by pixel from the texture of the reference 3D model scene to construct the reference image. Data content;

[0045] S25. Repeat steps S21-S24 to traverse all views until the reference image set is complete. The data collection task.

[0046] Preferably, step S24 specifically includes the following:

[0047] S241, Dough sheet pixels in The mappings in 3D space and texture space are respectively and ;

[0048] S242, Calculation Point Corresponding depth value and depth map Depth value in In comparison, if Less than Then Stored in the depth map;

[0049] S243, Using triangles Medium pixel with dough pieces noodle pieces With triangle The back projection between them and the texture coordinates of the three vertices of the face. The linear relationship between them is used to calculate the texture coordinates. The calculation formula is as follows:

[0050] (4)

[0051] In this process, back projection is the inverse of formula 3; , They are respectively The x-axis and y-axis coordinates in the view image coordinate system; , They are respectively The x-axis and y-axis coordinates in the view image coordinate system; For dough Projection mapping in image space;

[0052] S244, Based on texture coordinates Extract pixel color values ​​and fill them into the reference image. In At pixel coordinates.

[0053] Preferably, step S3 specifically includes the following:

[0054] S31. Determine the simplification parameters in the QEM algorithm according to actual needs. , The QEM algorithm is used to perform vertex deletion and merging, edge folding, and face deletion and merging operations on the original fine 3D mesh.

[0055] S32. Determine the texture simplification parameters, i.e., the resolution sampling level of the reference image. ;

[0056] S33. Sampling level based on the resolution of the reference image Determine the intrinsic parameter matrix involved in the texture reconstruction algorithm. extrinsic parameter matrix and reference image resolution ;use A new reference image set is obtained by sampling the reference image set. ;

[0057]

[0058]

[0059] (5)

[0060] S34. Using the calculated parameters, image data, and simplified mesh as input, perform texture mapping on the simplified mesh according to the texture reconstruction steps in section S2.

[0061] Preferably, in step S32, the resolution sampling level of the reference image It can be customized according to needs, or simplified parameters can be used. The calculation is as follows:

[0062] (6).

[0063] The beneficial effects of this invention are: 1. When only mesh simplification is performed, the algorithm of this invention basically does not exhibit texture distortion or deformation, and is robust. 2. The algorithm of this invention can support the simplification of mesh and texture together; under different texture simplification parameters, the texture shows almost no distortion and the amount of texture data can be significantly reduced. Attached Figure Description

[0064] Figure 1 This is a simplified diagram of step 3 in the existing simplification algorithm;

[0065] Figure 2 This is a simplified diagram of step 2 in the existing simplification algorithm;

[0066] Figure 3 This is a flowchart illustrating the principle of the mesh texture simplification algorithm in this embodiment of the invention;

[0067] Figure 4 This is a schematic diagram of a texture block with a border in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the three-dimensional model and its scene structure in an embodiment of the present invention;

[0069] Figure 6 This is the optimal view ID of the original mesh patch in the embodiments of the present invention;

[0070] Figure 7 This is the optimal view ID for simplifying the mesh patch in this embodiment of the invention;

[0071] Figure 8 These are images of the same area acquired at different times in this embodiment of the invention;

[0072] Figure 9This is a schematic diagram illustrating the transformation relationship between the three-dimensional mesh, view, and texture in an embodiment of the present invention;

[0073] Figure 10 This is a schematic diagram of experimental data in an embodiment of the present invention;

[0074] Figure 11 This is a schematic diagram comparing the texture quality of the building area model in an embodiment of the present invention;

[0075] Figure 12 This is a schematic diagram comparing the texture quality of the factory area model in an embodiment of the present invention;

[0076] Figure 13 This is a schematic diagram comparing the texture quality of the building area model in an embodiment of the present invention;

[0077] Figure 14 These are schematic diagrams of three-dimensional models of four different regions in an embodiment of the present invention;

[0078] Figure 15 This is a bar chart showing the statistical data volume in an embodiment of the present invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0080] Example 1

[0081] like Figure 3 As shown, this embodiment provides a mesh texture simplification algorithm suitable for 3D reconstruction, including the following steps:

[0082] S1. Construction of the reference 3D model scene;

[0083] Using the scene structure recovered in 3D reconstruction and the calibrated image, texture mapping is performed on the original fine 3D mesh to complete the reconstruction of the reference 3D model scene;

[0084] S2. Image acquisition of the reference 3D model scene;

[0085] Based on the relative pose relationship between the reference 3D model scene and the intrinsic and extrinsic parameters of the view, the rasterization calculation from 3D mesh to 2D image is performed pixel by pixel on a view-by-view using the back projection principle to complete the acquisition of the reference image set.

[0086] S3, Mesh and Texture Simplification;

[0087] The QEM algorithm is used to simplify the mesh, and a reference image set is used as the data source. Texture reconstruction algorithm is used to remap and simplify the texture.

[0088] The following sections will explain the three parts mentioned above:

[0089] I. Construction of the reference 3D model scene

[0090] The following steps are used to reconstruct the texture of the original fine 3D mesh by applying texture mapping, thus completing the reconstruction of the reference 3D model scene. The specific steps are as follows:

[0091] 1. Optimal view selection based on grid patches of multi-view images

[0092] This step essentially involves calculating a label for each facet to record the facet's mouth and the image number for texture mapping.

[0093] The energy function of the view is minimized by graph cuts and alpha expansion to obtain the best view for each facet in the mesh, and texture patches are created. Spatially adjacent faces with the same best view are stored in the same texture patch.

[0094] (2)

[0095] in, For a certain texture patch, It is a facet belonging to the current texture patch. This is the best view corresponding to the current texture patch.

[0096] In the data terms of the energy function The Soble gradient integral is used to solve the image defocusing problem, in the smoothing term of the energy function. The Potts model is used to smooth the view selection of the neighborhood triangulation;

[0097] (1)

[0098] in, For view The energy function; For data items, , Triangular mesh In the labeled image The Soble gradient integral under the given condition represents the node. Select a label image The probability magnitude; For smoothing terms, , indicating adjacent nodes and When selecting images with the same label, the value of the smoothing term is 0; otherwise, it is infinity. It is a sheet of dough; A collection of facets; To and Adjacent facets; For image; , For numbering.

[0099] As a data item, the facet was considered. The study involves two aspects: summing all pixels in the gradient magnitude map of the projected view and image consistency detection. Among these, the patch... The formula for the sum of the projection gradient magnitudes of pixels is: The gradient magnitude is Image consistency detection primarily uses a modified mean-shift algorithm to attempt to remove images containing occlusions such as pedestrians; among which, For image The gradient.

[0100] As a smoothing term, it refers to the difference in texture on the left and right sides of the seam at the joint. A smoothing term is proposed based on the Potts model: This also tends to use compact blocks rather than distant views, and the computation speed is very fast.

[0101] 2. Color adjustment at the seams of texture blocks

[0102] After selecting the best view for a facet, adjacent faces may choose different views as their texture source. Due to objective factors such as shooting angle, lighting, and occlusion during image acquisition, there are differences in pixels near the seams of adjacent faces. Therefore, it is necessary to adjust the colors between adjacent faces. Color adjustment includes two parts: global color adjustment and Poisson editing for local color adjustment.

[0103] (1) Global color adjustment: This mainly involves finding inconsistent color values ​​at vertex projections and along all adjacent seam edges; an cumulative correction is calculated for the color value of each vertex from a global perspective. The formula is expressed in matrix form as follows: In this formula It is a superimposed vector on the left and right sides of a seam; and It is a sparse matrix containing ±1 terms. Calculate an accumulated correction for the vertex.

[0104] Even after global adjustments, the algorithm still cannot eliminate all visible color differences at seams. Therefore, local Poisson image editing is also required. The algorithm sets a 20-pixel-wide border strip, the outer edge of which ( Figure 4 (outer light-colored line) and inner edge ( Figure 4 The inner dark line (as a boundary condition for the Poisson equation) is used as follows: the mean pixel color of the image assigned to the patch and the images of neighboring patches is set to a fixed value for the color of each outer edge pixel. The value of each inner edge pixel is fixed to its current color. If the patch is too small, the inner edges will be omitted.

[0105] S13, Texture space layout and texture coordinate calculation;

[0106] After color adjustment, the vertex coordinates of the faces stored in the texture patches are back-projected one by one into the view image space according to the projection matrix of the view, and the bounding boxes corresponding to the texture patches are calculated. Then, the texture space layout is performed using a 2D box packing algorithm, and the offset value of each bounding box is calculated. Finally, the texture coordinate values ​​of each face are calculated, and pixel data is extracted to generate the texture of the reference 3D model scene. The 2D box packing algorithm is an improved 2D bin packing approximation algorithm.

[0107] II. Image Acquisition of the Reference 3D Model Scene

[0108] To facilitate the description of the method of the present invention, the following provisions are made: using the recovered view pose (e.g.) Figure 5 The set of images generated by the camera's view frustum (represented by a light-colored triangular frame) "taking pictures" of the reference 3D model scene is called the reference image set. The set of images generated by devices such as drones and mobile terminals "taking pictures" of the real objective world is called a real image set. .

[0109] If a real image set is used directly as the data source for the simplified mesh texture reconstruction in the third step, the texture pixel content of the original mesh and the simplified mesh will be inconsistent. The specific reasons are as follows:

[0110] (1) Within the same spatial range, the original mesh and the simplified mesh have different shapes and sizes of faces, causing the original mesh and the simplified mesh to use different views as the best views for the faces in the first stage of texture reconstruction, such as Figure 6 , Figure 7 and Figure 8 As shown. Among them, Figure 6 (a) and Figure 7 (a) are all overall diagrams. Figure 6 (b) and Figure 7 (b) is a magnified view of the corresponding overall view. Figure 6 and Figure 7The dashed box represents the same spatial range. The texture reconstruction algorithm converts each patch into a node of the graph and constructs an energy function based on factors such as patch visibility, gradient magnitude, image consistency detection, and moving object culling. Then, a global graph cut method is performed on the graph to obtain the optimal view of the patch. Figure 6 (b) The best view IDs for the original grid are 48, 87, 115, and 88, while Figure 7 (b) The best view ID for simplifying the grid is only 41.

[0111] (2) Data collection by devices such as drones or mobile terminals is affected by the time dimension. Images collected at different times in the same area will contain different information such as vehicles, pedestrians, dynamic shadows, and changes in road surface wetness, for example... Figure 8 The ( Figure 8 (a) There are cars driving on the road, but no water. Figure 8 (b) There are no cars driving on the road (there is water). Therefore, the pixel content extracted from the same patch from different images may be different.

[0112] Based on the above two aspects, and according to reason (1), the original and simplified meshes within the red box will use different views as the best views for the facets. According to reason (2), the pixel content of the images collected at different times within the same spatial range is different, and the pixel content extracted by the facets from different images is also different. Therefore, this causes inconsistency in the pixel content of the texture.

[0113] To avoid inconsistencies in pixel content between the original and simplified mesh textures, the algorithm extracts pixels from the texture of the reference 3D model scene using information such as camera intrinsics, view extrinsics, and calibrated images from the recovered scene to generate a reference image set. The specific steps are as follows:

[0114] 1. Construct a view based on the intrinsic parameters, extrinsic parameters, and image resolution of the view in the restored scene. The projection matrix, as shown in Equation 3, transforms 3D points in the world coordinate system to pixels in the view image coordinate system; then, an octree index is built for the reference 3D model scene to facilitate fast view searching. Available photos;

[0115] (3)

[0116] in, This is the intrinsic parameter matrix. This is the extrinsic parameter matrix; , These are the x-axis coordinates and y-axis coordinates in the view image coordinate system, respectively; The z-axis coordinate value in the camera coordinate system; The focal length of the camera; , These are the center pixel coordinates; It is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; It is a 3×1 translation vector, which is the offset of the camera coordinate system relative to the world coordinate system; It is a zero vector of size 1×3; These are the components along the x-axis, y-axis, and z-axis in the world coordinate system, respectively.

[0117] 2. Combine with views The camera projection matrix is ​​used to create an image with the same resolution as the view image. Depth map and reference image Depth map Located in view In the camera coordinate system, and the depth value is initialized to 0;

[0118] 3. Traverse the view The visible surface is projected using a camera projection matrix. Projecting from 3D space to a 2D reference image And rasterize into triangles ;

[0119] 4. Statistics The occupancy of pixels is determined by extracting color values ​​pixel by pixel from the texture of the reference 3D model scene to construct the reference image. Data content;

[0120] by Figure 9 ( Figure 9 (a) is the overall view, with the grid at the bottom, the texture at the top left, and the view reference image at the top right; Figure 9 (b) Pixels in the enlarged view of the three spatial relationships Let's take an example to illustrate, where and They are dough pieces pixels in Mapping between 3D space and texture space. First, calculate the points. Corresponding depth value and depth map Depth value in Compare; if Less than Then Store it in the depth map; then, utilize... medium pixel and , and The inverse projection between them (the inverse process of formula (3)) and the texture coordinates of the three vertices of the facet The linear relationship between them is used to calculate the texture coordinates. The formula is (4); finally, based on the texture coordinates Extract pixel color values ​​(usually RGB) and fill them into the reference image. In At pixel coordinates; where, , They are respectively The x-axis and y-axis coordinates in the view image coordinate system; , They are respectively The x-axis and y-axis coordinates in the view image coordinate system; For dough Projection mapping in image space.

[0121]

[0122] 5. Repeat steps S21-S24 to traverse all views until the task of acquiring the reference image set is completed.

[0123] III. Mesh and Texture Simplification

[0124] The simplification of 3D models is divided into two aspects: mesh simplification and texture simplification. This invention uses the QEM algorithm; however, texture simplification currently suffers from texture distortion and deformation problems, and the unsimplified texture results in a still large model data volume. The method proposed in this invention can effectively solve the above problems. The specific steps are as follows:

[0125] 1. Determine the simplification parameters in the QEM algorithm based on actual needs. , The QEM algorithm is used to perform operations such as vertex deletion and merging, edge folding, and face deletion and merging on the original fine 3D mesh.

[0126] 2. Determine the texture simplification parameters, i.e., the resolution sampling level of the reference image. ;

[0127] The value can be determined independently or based on the mesh simplification parameters. The estimated calculation formula proposed in this invention is as follows: Theoretically The range of values ​​is However, in practical applications, an infinite resolution level for an image is meaningless. Therefore, the commonly used range limit is set to... .in addition, The calculation formula can also be customized.

[0128] 3. Sampling level based on the resolution of the reference image Determine the intrinsic parameter matrix involved in the texture reconstruction algorithm. extrinsic parameter matrix and reference image resolution ;use A new reference image set is obtained by sampling the reference image set. ;

[0129]

[0130]

[0131] (5)

[0132] 4. Using the calculated parameters, image data, and simplified mesh as input, perform texture mapping on the simplified mesh according to the texture reconstruction steps in section S2.

[0133] Example 2

[0134] In this embodiment, the IMS automated 3D reconstruction software developed by the Chinese Academy of Surveying and Mapping is used as a reference, and the mesh texture simplification algorithm of this invention, which is applicable to 3D reconstruction, is embedded. The effect is verified by taking the 3D mesh model reconstructed by oblique photogrammetry of a town in Tengzhou City as an example.

[0135] I. Experimental Data

[0136] Detailed parameters of the experimental data are shown in Table 1, and the reconstruction range is as follows: Figure 10 As shown, it covers the main types of 3D models: buildings, vegetation, roads, etc., which has universal significance for experimental verification. The experimental environment is a workstation with Windows 10 64-bit operating system, Intel Core™ i9-10900X CPU (3.70GHz), and 128GB of memory.

[0137] Table 1 Description of 3D mesh data for oblique images

[0138]

[0139] II. Texture Quality Comparison Analysis

[0140] Regarding mesh simplification, the algorithm of this invention is the same as that of traditional algorithms, both based on the QEM algorithm, performing folding and merging operations on the triangular faces of the mesh. Regarding texture simplification, traditional algorithms only recalculate texture coordinates, leading to quality issues such as texture distortion and deformation, while the algorithm of this invention effectively avoids texture distortion and deformation. A comparative experiment was designed in the same environment to compare the algorithm of this invention with a traditional algorithm (She J, Gu X, Tan J, et al. An appearance reserving simplification method for complex 3D building models[J]. Transactions in GIS,2019, 23).

[0141] The experiment selected two types of complex and textured 3D models of different scenarios (building area and factory area). The building area includes the main elements such as building facade, windows and attachments, while the factory area includes the main elements such as oil tanks, road traffic signs and pipelines.

[0142] 1. Mesh simplification parameters Impact on texture quality

[0143] For the two sets of 3D models, only mesh simplification is performed, without texture simplification. The original mesh (i.e. , As the truth value, different mesh reduction parameters are used respectively. Then, the degree of texture distortion and deformation was compared between the algorithm of this invention and the traditional algorithm. The experimental comparison results for building areas and factory areas are as follows: Figure 11 , 12 As shown.

[0144] Comparative analysis of experiments with different mesh simplification parameters: From Figure 11 As can be seen from (a), (c), (e), and (g), the smaller the mesh simplification parameter of the traditional algorithm, the greater the degree of texture distortion and deformation. When the simplification parameter is large, the degree of texture distortion and deformation is small, but as the mesh simplification parameter decreases, the texture distortion and deformation become very obvious. This phenomenon is particularly noticeable at oil tanks, road traffic signs, and pipelines in factory areas, such as... Figure 12 As shown in (c), (e), and (g), this is because as the mesh simplification parameters decrease, the probability of texture patches corresponding to those folded and merged faces being continuous in the two-dimensional texture space decreases. This discrete distribution phenomenon leads to the inability to correctly calculate the texture coordinates of the newly generated triangular faces, thus exacerbating texture distortion and deformation. Figure 11As can be seen from (a), (b), (d), and (f), the algorithm of this invention produces textures in both the building and factory areas without distortion or deformation under different mesh simplification parameters, closely resembling the original mesh.

[0145] Comparative analysis of experiments with the same mesh simplification parameters: Figure 11 (b) and (c) Figure 11 (d) and (e) Figure 11 (f) and (g), Figure 12 (b) and (c) Figure 12 (d) and (e) Figure 12 Experimental results (f) and (g) with the same six sets of mesh simplification parameters show that the number of triangular faces in the algorithm of this invention is equal to that of the traditional algorithm. The texture distortion and deformation of the algorithm of this invention are both less than those of the traditional algorithm, verifying the superiority of the algorithm of this invention in terms of texture quality. Furthermore, the algorithm of this invention shows no distortion or deformation in areas such as building facades, attachments, road traffic signs, pipelines, and oil tanks in both types of models, demonstrating its applicability to models containing different elements and its strong universality.

[0146] Figure 11 In the image, (a) shows the original mesh (truth value): , (a) Number of facets = 109304; (b) Algorithm of the present invention: , Number of facets = 54651; (c) Traditional algorithm: , Number of facets = 54651; (d) Algorithm of this invention: , Number of facets = 27326; (e) Traditional algorithm: , Number of facets = 27326; (f) Algorithm of this invention: , Number of facets = 13662; (g) Traditional algorithm: , Number of pieces = 13662.

[0147] Figure 12 In the image, (a) shows the original mesh (truth value): , (a) Number of facets = 125288; (b) Algorithm of the present invention: , Number of facets = 62643; (c) Traditional algorithm: , Number of facets = 62643; (d) Algorithm of this invention: , Number of facets = 31321; (e) Traditional algorithm: , Number of facets = 31321; (f) Algorithm of this invention: , Number of facets = 15661; (g) Traditional algorithm: , Number of pieces = 15661.

[0148] 2. Texture simplification parameters Impact on texture quality

[0149] Mesh simplification parameters were set in the experiment. According to the calculation formula in step S2 get The algorithm of this invention is executed in the building area according to the above parameters to obtain the simplified result, such as... Figure 13 As shown. Figure 13 In the middle, (a) the algorithm of this invention: , (a) Number of facets = 54651; (b) Algorithm of the present invention: , (c) Algorithm of the present invention: , Number of pieces = 13662.

[0150] from Figure 13 As can be seen from (a), (b), and (c), the larger the texture simplification parameter, the blurrier the texture, but there is almost no difference in texture distortion and deformation. (Comparison) Figure 11 (b) and Figure 13 (a) Figure 11 (d) and Figure 13 (b) Figure 11 (f) and Figure 13 (c) Analysis shows that when the mesh simplification parameters are the same but the texture simplification parameters are different, Figure 12 Textures simplified from textural elements Figure 11 Unsimplified textures show almost no difference in distortion and deformation, but simplified textures are less sharp than unsimplified textures. This indicates that the texture simplification parameters... The magnitude of the value does not affect the distortion and deformation of the texture, but it does affect the clarity of the texture. In extreme cases, exceeding a certain texture simplification parameter threshold makes the texture content difficult to distinguish. This is mainly because the algorithm of this invention performs appropriate downsampling on the images in the reference dataset during texture simplification, resulting in a decrease in texture clarity. However, the algorithm of this invention can meet the normal requirements of model simplification under general circumstances.

[0151] III. Comparative Analysis of Texture Data Volume

[0152] This invention's algorithm not only simplifies the mesh but also reduces the overall data volume of the model through texture simplification. Multiple model tiles (100.0m * 100.0m in size) were randomly selected from four representative regions (residential area, factory area, vegetated area, and road area) as experimental data. Figure 14 ( Figure 14 (a) Residential area; Figure 14 (b) Factory area; Figure 14 (c) Vegetated area; Figure 14 (d) Road area). The simplified parameters of the mesh and texture are shown. The values ​​were set to (1.0, 1.0), (0.5, 2.0), (0.25, 4.0), and (0.125, 8.0) respectively. The data volume of the algorithm of this invention and the traditional algorithm were compared, as shown in Table 2. The data volume (mesh, texture) after simplification by the algorithm of this invention and the traditional algorithm under different simplification parameters is statistically analyzed, with units in Kb. (a) Residential area; (b) Factory area; (c) Vegetated area (taking farmland as an example); (d) Road area (taking highway as an example). The bar chart is shown below. Figure 15 ( Figure 15 The content represented by each four adjacent bars in the figure, from left to right, is as follows: the algorithm grid of this invention, the algorithm texture of this invention, the traditional algorithm grid, and the traditional algorithm texture.

[0153] Table 2 Data Volume Statistics

[0154] (a)

[0155]

[0156] (b)

[0157]

[0158] (c)

[0159]

[0160] (d)

[0161]

[0162] As can be seen from Table 2, the algorithm of this invention can reduce the amount of data in the 3D model through simplification of both mesh and texture, and the amount of texture data increases with... The value increases or decreases by a factor of two, such as and At that time, the model data volume of the algorithm of this invention was only 38.9% and 7.11% of the original 3D model; while the traditional method only reduced the data volume of the mesh, without changing the data volume of the texture. With The decrease in value, The more significant the increase in the value, the more significant the difference in the reduction of 3D model data volume between the algorithm of this invention and traditional algorithms becomes, such as... At the same time, the model data volume of the algorithm of this invention is only 12.9% of that of the traditional algorithm. In summary, the algorithm of this invention can effectively alleviate the contradiction between the rapidly increasing volume of 3D model data and the limited transmission bandwidth and smooth model rendering.

[0163] IV. Conclusion

[0164] Existing 3D mesh simplification algorithms, when performing mesh simplification, are limited to recalculating texture coordinates only, resulting in varying degrees of texture distortion and deformation, and do not perform texture simplification. Therefore, this invention proposes a mesh texture simplification algorithm suitable for 3D reconstruction. First, a reference 3D model scene is constructed based on the original mesh and the recovered camera pose; second, a reference image set is generated based on the reference 3D model scene and the recovered camera pose; finally, the mesh is simplified using the QEM algorithm, and the texture is simplified using an improved texture reconstruction algorithm while avoiding distortion. Through experimental verification and comparative analysis, the following conclusions are drawn: (1) In the case of mesh simplification only, the algorithm of this invention basically does not have texture distortion and deformation, and is robust; (2) The algorithm of this invention can support the simplification of both mesh and texture. Under different texture simplification parameters, the texture has almost no distortion and can significantly reduce the amount of texture data. When the texture simplification parameters are... At that time, the amount of texture data in the algorithm of this invention is only 28.17% of that of the traditional method; when the texture simplification parameter At that time, the amount of texture data in the algorithm of this invention is only 2.40% of that of the traditional method.

[0165] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0166] This invention provides a mesh texture simplification algorithm suitable for 3D reconstruction. When only mesh simplification is performed, the algorithm exhibits virtually no texture distortion or deformation and is robust. The algorithm supports simplification of both mesh and texture simultaneously; under different texture simplification parameters, the texture shows almost no distortion and the amount of texture data is significantly reduced.

[0167] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A mesh texture simplification algorithm suitable for 3D reconstruction, characterized in that: Includes the following steps, S1. Construction of the reference 3D model scene; Using the scene structure recovered in 3D reconstruction and the calibrated image, texture mapping is performed on the original fine 3D mesh to complete the reconstruction of the reference 3D model scene; S2. Image acquisition of the reference 3D model scene; Based on the relative pose relationship between the reference 3D model scene and the intrinsic and extrinsic parameters of the view, the rasterization calculation from 3D mesh to 2D image is performed pixel by pixel on a view-by-view using the back projection principle to complete the acquisition of the reference image set. S3, Mesh and Texture Simplification; The QEM algorithm is used to simplify the mesh, with a reference image set as the data source and a texture reconstruction algorithm used to remap and simplify the texture. Step S1 specifically includes the following: S11. Select the best view based on the grid patch of the multi-view image; The optimal view for each facet in the mesh is obtained by minimizing the energy function of the view using graph cuts and alpha expansion. And create texture patches, storing spatially adjacent patches with the same best view into the same texture patch; （1） （2） In formula (1), For view The energy function; For data items, , The Soble gradient integral of the triangular mesh under the labeled image represents the node. Select a label image The probability magnitude; For smoothing terms, , indicating adjacent nodes and When selecting images with the same label, the value of the smoothing term is 0; otherwise, it is infinity. It is a sheet of dough; A collection of facets; To and Adjacent facets; and For image, and For numbering; In formula (2), For a certain texture patch, It is a facet belonging to the current texture patch; This is the best view corresponding to the current texture patch; S12, Adjust the color at the seam of the texture block; Global color adjustment and Poisson editing for local color adjustment were used to adjust the color at the seams of adjacent facets; S13, Texture space layout and texture coordinate calculation; Based on the projection matrix of the view, the vertex coordinates of the face patches stored in the texture patch are back-projected one by one to the view image space and the bounding box corresponding to the texture patch is calculated. The texture space layout is performed using a two-dimensional box packing algorithm and the offset value of each bounding box is calculated. Calculate the texture coordinates of each facet and extract pixel data to generate the texture of the reference 3D model scene.

2. The mesh texture simplification algorithm for 3D reconstruction according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21. Construct a view based on the intrinsic parameters, extrinsic parameters, and image resolution of the view in the restored scene. The projection matrix, as shown in formula (3), transforms the 3D points in the world coordinate system to pixels in the view image coordinate system; then, an octree index is built for the reference 3D model scene to facilitate fast search of the view. Available photos; （3） in, This is the intrinsic parameter matrix. This is the extrinsic parameter matrix; , These are the x-axis coordinates and y-axis coordinates in the view image coordinate system, respectively; The z-axis coordinate value in the camera coordinate system; The focal length of the camera; , These are the center pixel coordinates; It is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; It is a 3×1 translation vector, which is the offset of the camera coordinate system relative to the world coordinate system; It is a zero vector of size 1×3; These are the components along the x-axis, y-axis, and z-axis in the world coordinate system, respectively. S22, Combined View The camera projection matrix is ​​used to create an image with the same resolution as the view image. Depth map and reference image Depth map Located in view In the camera coordinate system, and the depth value is initialized to 0; in, and These are the original width and height of the view image, respectively; S23, Traversing Views The visible surface is projected using a camera projection matrix. Projecting from 3D space to a 2D reference image And rasterize into triangles ; S24, Statistics The occupancy of pixels is determined by extracting color values ​​pixel by pixel from the texture of the reference 3D model scene to construct the reference image. Data content; S25. Repeat steps S21-S24 to traverse all views until the reference image set is complete. The data collection task.

3. The mesh texture simplification algorithm for 3D reconstruction according to claim 2, characterized in that: Step S24 specifically includes the following: S241, Dough sheet pixels in The mappings in 3D space and texture space are respectively and ; S242, Calculation Point Corresponding depth value and depth map Depth value in In comparison, if Less than Then Stored in the depth map; S243, Using triangles Medium pixel with dough pieces noodle pieces With triangle The back projection between them and the texture coordinates of the three vertices of the face. The linear relationship between them is used to calculate the texture coordinates. ; The calculation formula is as follows: （4） In this process, back projection is the inverse of formula (3); , They are respectively The x-axis and y-axis coordinates in the view image coordinate system; , They are respectively The x-axis and y-axis coordinates in the view image coordinate system; For dough Projection mapping in image space; S244, Based on texture coordinates Extract pixel color values ​​and fill them into the reference image. In At pixel coordinates.

4. The mesh texture simplification algorithm for 3D reconstruction according to claim 3, characterized in that: Step S3 specifically includes the following: S31. Determine the simplification parameters in the QEM algorithm according to actual needs. , The QEM algorithm is used to perform vertex deletion and merging, edge folding, and face deletion and merging operations on the original fine 3D mesh. S32. Determine the texture simplification parameters, i.e., the resolution sampling level of the reference image. ; S33. Sampling level based on the resolution of the reference image Determine the intrinsic parameter matrix involved in the texture reconstruction algorithm. extrinsic parameter matrix and reference image resolution ; use A new reference image set is obtained by sampling the reference image set. ; ； ； （5） S34. Using the calculated parameters, image data, and simplified mesh as input, perform texture mapping on the simplified mesh according to the texture reconstruction steps in section S2.

5. The mesh texture simplification algorithm for 3D reconstruction according to claim 4, characterized in that: In step S32, the resolution sampling level of the reference image It can be customized according to needs, or simplified parameters can be used. The calculation is as follows: （6）。

Images