An automatic texture mapping method for three-dimensional reconstruction, a terminal and a storage medium

Abstract

The application discloses a kind of automatic texture mapping methods for three-dimensional reconstruction, terminal and storage medium, it is related to computer technology field, it solves the technical problem that the chromatic aberration of texture mapping in triangular mesh model is larger, its technical scheme main point is to determine the first optimal projection visual angle of each triangle in triangular mesh model;The second optimal projection visual angle of each vertex of triangle is calculated, obtain 3 second optimal projection visual angle of each triangle about vertex;The UV coordinate of each triangle vertex is calculated, respectively obtain the horizontal and vertical coordinates of three vertices of triangle in two-dimensional texture mapping, so as to obtain the horizontal and vertical coordinates of any point in two-dimensional texture mapping in triangle;According to coordinate, corresponding sampling is carried out on pixel in original photo, and texture mapping is automatically carried out according to the pixel sampled.It significantly reduces the complexity of texture mapping process, reduces the chromatic aberration of adjacent triangle texture, and generates relatively complete and continuous texture.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an automatic texture mapping method, terminal and storage medium for 3D reconstruction. Background Technology

[0002] Texture mapping in 3D reconstruction refers to generating corresponding color information for the surface of a reconstructed white 3D mesh model based on a set of RGB images taken with known camera poses, in order to reproduce the colors of 3D objects in a real scene. Existing texture mapping techniques have the following drawbacks:

[0003] (1) When generating the initial texture, the GraphCut algorithm based on the probabilistic graphical model is often used to select the RGB image of the optimal viewpoint for each triangle of the 3D model. For example, in a 3D model that is a triangular mesh model, since the optimal viewpoints of adjacent triangles are not necessarily the same, their image lighting may be very different, which may result in a large color difference in the texture of adjacent triangles.

[0004] (2) When performing color difference optimization, the color adjustment amount of the vertices is often optimized first, while the adjustment amount of the pixels inside the triangle is interpolated. This will result in a large error in the color adjustment amount when the distance between adjacent vertices is far.

[0005] (3) When generating UV coordinates for texture mapping, conformal or shape-preserving parameterization is often used to parameterize spatial triangles into 2D textures, which can lead to problems such as triangle overlap and requires complex post-processing.

[0006] Therefore, a new texture mapping method is urgently needed to solve the above problems. Summary of the Invention

[0007] This application provides an automatic texture mapping method, terminal, and storage medium for 3D reconstruction. Its technical purpose is to reduce color difference and adjustment error in texture mapping in triangular mesh models and simplify the texture mapping process.

[0008] The above-mentioned technical objective of this application is achieved through the following technical solution:

[0009] An automatic texture mapping method for 3D reconstruction includes:

[0010] S1: Construct a triangular mesh model using the original photograph and camera pose;

[0011] S2: Determine the first optimal projection viewpoint for each triangle in the triangular mesh model;

[0012] S3: Calculate the second optimal projection view for each vertex of the triangle to obtain three second optimal projection views for each triangle about the vertex; where adjacent triangles share vertices, the value of the second optimal projection view is equal to the first optimal projection view that appears most frequently among the adjacent triangles.

[0013] S4: Calculate the UV coordinates of each triangle vertex to obtain the horizontal and vertical coordinates (u,v) of the three vertices of the triangle in the 2D texture map. Based on the triangle vertex coordinates, obtain the horizontal and vertical coordinates (u,v) of any point inside the triangle in the 2D texture map.

[0014] S5: Sample the corresponding pixels in the original photo according to the (u,v) coordinates, and automatically perform texture mapping based on the sampled pixels.

[0015] Furthermore, the method also includes:

[0016] S6: Perform global color difference optimization on the texture map, obtain the color adjustment amount of each pixel, and then update the pixel values ​​of the texture map.

[0017] A terminal includes a processor and a memory, the processor being configured to execute one or more programs stored in the memory to implement the steps of an automatic texture mapping method for 3D reconstruction.

[0018] A storage medium comprising one or more programs executable by one or more processors to implement the steps of an automatic texture mapping method for 3D reconstruction.

[0019] The beneficial effects of this application are as follows: The automatic texture mapping method for 3D reconstruction described in this application constructs a triangular mesh model using the original photograph and camera pose; determines the first optimal projection viewpoint for each triangle in the triangular mesh model; calculates the second optimal projection viewpoint for each vertex of the triangle, obtaining three second optimal projection viewpoints for each triangle about the vertex; calculates the UV coordinates of each triangle vertex, obtaining the horizontal and vertical coordinates (u,v) of the three vertices of the triangle in the 2D texture map; obtains the horizontal and vertical coordinates (u,v) of any point inside the triangle in the 2D texture map based on the triangle vertex coordinates; samples the corresponding pixels in the original photograph based on the (u,v) coordinates, and automatically performs texture mapping based on the sampled pixels. This significantly reduces the complexity of the texture mapping process, reduces the color difference between adjacent triangle textures, and generates a more complete and continuous texture. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method described in this application;

[0021] Figure 2This is a schematic diagram of the projection view of a triangle in the triangular mesh model in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram of a grid block with a radius R of 2 in an embodiment of this application. Detailed Implementation

[0023] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0024] like Figure 1 As shown, the automatic texture mapping method for 3D reconstruction described in this application includes:

[0025] S1: Construct a triangular mesh model using the original photograph and camera pose.

[0026] S2: Determine the first optimal projection viewpoint for each triangle in the triangular mesh model.

[0027] Since a triangle on a triangular mesh model may be visible in photos from multiple different viewpoints, an image with an optimal viewpoint must be selected for each triangle during texture mapping. For example... Figure 2 As shown, in this embodiment, when selecting the optimal projection angle, the vector r formed by the line connecting the camera center and the triangle center is considered. j And the normal vector n of the triangle i The angle between them, when r j With n i When the angle between them is the smallest, vector r j The corresponding camera viewpoint is the first optimal projection viewpoint of the triangle, in order to minimize the triangle distortion caused by the projection transformation.

[0028] S3: Calculate the second optimal projection view for each vertex of the triangle to obtain three second optimal projection views for each triangle about the vertex; where adjacent triangles share vertices, the value of the second optimal projection view is equal to the first optimal projection view that appears most frequently among the adjacent triangles.

[0029] As mentioned above, directly generating the initial texture based on the first optimal projection viewpoint selected for each triangle may result in significant color differences between adjacent faces. This application, after selecting the optimal projection viewpoint for each triangle, utilizes the characteristic that adjacent triangles in a triangular mesh model share vertices to calculate the optimal projection viewpoint v for each vertex. k (i.e., the second optimal projection angle), its value is equal to the angle that appears most frequently among the first optimal projection angles of adjacent triangles. Figure 3For example, the conventional numbers indicate that the distance between the triangle and its vertex is 1 (1-nearest neighbor), where "1" indicates that the first projection view appears most frequently, and "2" indicates that the second projection view appears most frequently. The skewed numbers indicate that the distance between the triangle and its vertex is 2 (2-nearest neighbor), where "2" indicates that the second projection view appears most frequently, and "3" indicates that the third projection view appears most frequently. Where v k =2, indicating that among the triangles adjacent to vertex k, the second projection view appears the most times. In this embodiment, the maximum neighbor distance to the vertex is considered to be 2.

[0030] Following the above method, after determining the optimal projection viewpoints for all vertices, each triangle obtains three projection viewpoints. Finally, the texture of the triangle can be obtained by weighted summation of the colors in the three projection viewpoint images. Since each adjacent triangle shares two identical vertices (viewpoints), this method of texture mapping based on vertex projection viewpoints can significantly reduce color differences between adjacent triangles.

[0031] S4: Calculate the UV coordinates of each triangle vertex to obtain the horizontal and vertical coordinates (u, v) of the three vertices of the triangle in the 2D texture map. Based on the triangle vertex coordinates, obtain the horizontal and vertical coordinates (u, v) of any point inside the triangle in the 2D texture map.

[0032] Specifically, the texture of a 3D model is typically represented using a 2D image and the UV coordinates of the texture map of the three vertices of a triangle. That is, it is necessary to map the triangle in space onto the 2D image and determine the x and y coordinates (u, v) of the three vertices of the triangle in the 2D texture map. Existing mapping methods often involve segmenting the 3D model, followed by conformal or shape-preserving parameterization, and require complex post-processing. This application iteratively finds a mesh patch composed of triangles within a radius R and not exceeding N on the 3D model. By projecting the patch onto the corresponding region in the original image, calculating its bounding rectangle, and adding the patch to the texture map, UV coordinates are assigned to the triangles in each patch. Simultaneously, a certain overlap needs to be maintained between patches to avoid texture seams at patch boundaries.

[0033] In the algorithm for generating the UV coordinates of triangle vertices, the algorithm inputs are as follows:

[0034] F i: The i-th triangle face; S[i]: The optimal projection view of the i-th triangle; A[i, k]: The visibility of the i-th triangle in the k-th view, 0 indicates incomplete visibility, 1 indicates visibility; W, H: The length and width of the texture map; R: The radius of the triangle's neighbors; N: The maximum number of triangles in the same patch; VIS[i]: Whether the triangle face has been visited, 0 indicates unvisited, 1 indicates visited.

[0035] The algorithm outputs: a patch list L containing the UV coordinates of the triangular faces, and the patch index I of each triangular face. i .

[0036] In a specific embodiment, the UV coordinates of each triangle vertex are calculated to obtain the horizontal and vertical coordinates (u, v) of the three vertices of the triangle in the two-dimensional texture map, including:

[0037] S411: Initialize the grid block list L and the triangle projection viewpoint, represented as: L = [], V[:] = -1.

[0038] S412: Randomly select a triangle F that has not been assigned a grid block. i and F i The first optimal projection angle S[F i [This is used as the current projection viewpoint, and the current mesh block is initialized to patch = [F] i ].

[0039] S413: Using triangle F i Centered on the x-axis, for each of its neighbors F′ within radius R i Perform the traversal, if F′ i Not fully visible in the current projection view (i.e., A[F′) i ,S[F i ]] = 0) or F i With F′ i If the absolute value of the dot product of the unit normal vectors is less than the threshold of 0.7, then F′ will not be included. i Incorporated into F i In the centered grid block, otherwise for F′ i Determine whether the site has been accessed:

[0040] When F′ i Visited, i.e., VIS[F′ i ] = 1, if F′ i The first optimal projection angle V[F′ i If the current projection viewpoint is the same, then F′ will not be projected. i Incorporated into F i If the grid is centered, include it; otherwise, include it and update the list p = p.append(F′).i Update the projection angle V[F′) i ] = S[F i ];

[0041] When F′ i Unvisited, i.e., VIS[F′ i If ] = 0, then F′ i Incorporated into F i Within the central grid block, update p = p.append(F′ i Update the projection angle V[F′) i ] = S[F i ], and assign a patch index to it. Mark it as visited VIS[F′ i ] = 1;

[0042] Among them, each F′ included i The length of list L is increased by 1. When the length of list L reaches the maximum value N, the traversal ends and L = L.append(patch) is updated. Then step S32 is repeated until all triangles in the triangular mesh model are assigned mesh blocks, thus obtaining all the mesh blocks of the triangular mesh model. R represents the maximum distance between the neighboring triangle and the central triangle, and N represents the maximum number of triangles contained in each mesh block.

[0043] S414: Determine the bounding rectangle of each grid block from the original image of the projection view corresponding to each grid block. Let the offset of the coordinates of a vertex of a triangle in the grid block relative to the upper left corner of the bounding rectangle be (dx, dy), and the coordinates of the upper left corner of the bounding rectangle mapped to the texture map be (u0, v0). Then the coordinates of the vertex are expressed as: (u, v) = (u0 + dx / W, v0 + dy / H). By analogy, the horizontal and vertical coordinates (u, v) of the three vertices of each triangle in the grid block in the two-dimensional texture map are finally obtained.

[0044] Based on the coordinates of the triangle vertices, the x and y coordinates (u, v) of any point P inside the triangle in the 2D texture map are obtained, including:

[0045] P = αP A +βP B +γP C , 0≤α,β,γ≤1;

[0046]

[0047] S ABC =S PBc +S PAC +S PAB ;

[0048] Among them, P A P B P C S represents the coordinates of the three vertices of the triangle; PBC S PAC S PAB Let S represent the area of ​​the triangle formed by point P and the three sides of the triangle; ABC α represents the area of ​​the triangle itself; α, β, and γ represent the weights of the pixel in question under the second optimal projection viewpoint of the three vertices of the triangle.

[0049] Specifically, once the UV coordinates of the three vertices of a triangle are determined, the positions of all pixels in the texture map for that triangle are also determined. As mentioned earlier, for each pixel within the triangle, its color is obtained by weighting the values ​​of the optimal projection viewpoints of the pixel at the three vertices. Since the coordinates of any point within the triangle can be obtained by weighting the coordinates of the three vertices, this weight can be calculated from the relative position of the pixel to the three vertices of the triangle.

[0050] S5: Sample the corresponding pixels in the original photo according to the (u, v) coordinates, and automatically perform texture mapping based on the sampled pixels.

[0051] S6: Perform global color difference optimization on the texture map, obtain the color adjustment amount of each pixel, and then update the pixel values ​​of the texture map.

[0052] The above texture mapping generation method ensures that the textures of two adjacent triangles share two identical projected view images (two adjacent triangles share two identical vertices). Further reduction is needed to reduce the color difference that occurs at the triangle intersection caused by a third different view.

[0053] Specifically, global color difference optimization is performed on the texture map using a loss function, which is expressed as:

[0054]

[0055] Among them, C p =(r p g p b p ) represents the color of each pixel p. C represents p Color adjustment amount; C q This represents the color of pixel p's neighboring pixel q. C represents q Color adjustment amount; α p ,β p γ pThese represent the weights of pixel p under the second optimal projection viewpoints at the three vertices of the triangle, respectively. p represents the effective set of pixels for a texture map. a p b p c These represent the pixels corresponding to the three vertices of the triangle. They represent p respectively a p b p c Color adjustment amount; λ represents all neighbors of pixel p; λ, μ, and v all represent weight coefficients.

[0056] In a specific embodiment, λ = 1; μ = 0.05; v = 1. The above loss function can be globally iteratively optimized using gradient descent.

[0057] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.

Claims

1. An automatic texture mapping method for 3D reconstruction, characterized in that, include: S1: Construct a triangular mesh model using the original photograph and camera pose; S2: Determine the first optimal projection viewpoint for each triangle in the triangular mesh model; S3: Calculate the second optimal projection view for each vertex of the triangle to obtain three second optimal projection views for each triangle about the vertex; where adjacent triangles share vertices, the value of the second optimal projection view is equal to the first optimal projection view that appears most frequently among the adjacent triangles. S4: Calculate the UV coordinates of each triangle vertex to obtain the horizontal and vertical coordinates (u,v) of the three vertices of the triangle in the 2D texture map. Based on the triangle vertex coordinates, obtain the horizontal and vertical coordinates (u',v') of any point inside the triangle in the 2D texture map. S5: Sample the corresponding pixels in the original photo according to the (u',v') coordinates, and automatically perform texture mapping based on the sampled pixels; In step S4, obtaining the horizontal and vertical coordinates (u, v) of the triangle vertices in the 2D texture map includes: S411: Initialize the grid block list L and the triangle projection view; S412: Randomly select a triangle that has not been assigned a grid block. and will The first optimal projection view is used as the current projection view, and the current mesh block is initialized to patch= ; S413: In the form of a triangle Centered on the integer part of the integer part, consider each of its neighbors within a radius R. Perform traversal, if Not fully visible in the current projection viewpoint or and If the absolute value of the dot product of the unit normal vectors is less than the threshold of 0.7, then it will not be... Incorporated into In the central grid block, otherwise for Determine whether the site has been accessed; when If visited, If the first optimal projection view is the same as the current projection view, then do not project. Incorporated into In the centered grid block, otherwise included; when If it has not been visited, then Incorporated into Within the central grid block; Each of the included 1 The length of list L is increased by 1. When the length of list L reaches the maximum value N, the traversal ends and L = L.append(patch) is updated. Then step S412 is repeated until all triangles in the triangular mesh model are assigned mesh blocks, thus obtaining all the mesh blocks of the triangular mesh model; R represents the maximum distance between the neighboring triangle and the central triangle, and N represents the maximum number of triangles contained in each mesh block. S414: Determine the circumscribed rectangle of each grid block from the original image of the projected viewpoint corresponding to each grid block, and let the offset of the coordinates of a vertex of a triangle within the grid block relative to the upper left corner of the circumscribed rectangle be... The coordinates of the top-left corner of the bounding rectangle mapped to the texture map are... Then the coordinates of the vertex are represented as: (u,v)=( + / W, + / H); and so on, until the x and y coordinates (u, v) of the three vertices of each triangle in the mesh block in the two-dimensional texture map are obtained; where W represents the width of the texture map and H represents the length of the texture map; In step S4, the horizontal and vertical coordinates (u', v') of any point inside the triangle in the 2D texture map are obtained based on the coordinates of the triangle vertices, including: ； ； ； in, These represent the coordinates of the three vertices of the triangle; Representing points respectively The area of ​​the triangle formed by the triangle and its three sides; This represents the area of ​​the triangle itself. These represent the weights of the pixel in question under the second optimal projection viewpoints of the three vertices of the triangle.

2. The method as described in claim 1, characterized in that, The method also includes: S6: Perform global color difference optimization on the texture map, obtain the color adjustment amount of each pixel, and then update the pixel values ​​of the texture map.

3. The method as described in claim 1 or 2, characterized in that, In step S2, the first optimal projection viewpoint for each triangle in the triangular mesh model is determined, including: Obtain the vector formed by the line connecting the camera center and the triangle center. And obtain the normal vector of the triangle. ,when and When the included angle between them is the smallest, the vectors The corresponding camera viewpoint is the first optimal projection viewpoint of the triangle.

4. The method as described in claim 2, characterized in that, In step S6, global color difference optimization is performed on the texture map using a loss function, which is expressed as: ； in, Represents each pixel The color, express Color adjustment amount; Represents pixels neighboring pixels The color, express Color adjustment amount; Representing pixels The weights assigned to the triangle from the second optimal projection viewpoint at its three vertices; The set of valid pixels representing a texture map; These represent the pixels corresponding to the three vertices of the triangle. , , They represent Color adjustment amount; Represents pixels All neighbors; All of these represent weighting coefficients.

5. The method as described in claim 4, characterized in that, ； ； 。 6. A terminal, characterized in that, The terminal includes a processor and a memory, the processor being used to execute one or more programs stored in the memory to implement the steps of an automatic texture mapping method for 3D reconstruction as described in any one of claims 1-5.

7. A storage medium, characterized in that, The storage medium includes one or more programs that can be executed by one or more processors to implement the steps of an automatic texture mapping method for 3D reconstruction as described in any one of claims 1-5.

Images