Texture mapping method and device

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to a texture mapping method and device. The method comprises the following steps: acquiring an initial vectorization model, a discrete image set and at least one camera pose; based on the discrete image set, obtaining an effective patch tag graph of each view angle, and calculating an optimal view angle of each effective patch according to the at least one camera pose and the effective patch tag graph of each view angle; and based on the optimal view angle of each effective surface patch, performing deformation processing on the initial texture regions of all the effective surface patches to obtain distortion-removed texture regions, dispersing the texture regions to generate a texture set, and obtaining a texture-containing vectorization model according to the initial vectorization model and the texture set. Therefore, when the vectorization model containing the texture is used for texture mapping, the problems that a traditional texture mapping method is mainly suitable for a dense triangular mesh model and misplacement, missing, distortion and the like possibly occur to the vectorization model can be solved, and therefore the subjective visualization effect of texture mapping is improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a texture mapping method and device. Background Art

[0002] With the development of 3D reconstruction technology and the rise of the concept of digital twin cities in recent years, 3D reconstruction technology has begun to be widely used in the reconstruction of large cities. The 3D reconstruction models of large cities have strong application value in reality. They not only have a profound impact in the field of surveying and mapping, but also have very wide applications in unmanned driving and smart cities.

[0003] In related technologies, the models generated by three-dimensional reconstruction methods are mostly represented as dense triangular meshes, and the camera posture is used to establish a one-to-one correspondence between image pixels from different perspectives and the vertices of each triangular face, thereby obtaining the final model containing texture.

[0004] However, directly using the texture mapping method of triangular mesh for vector models will introduce problems such as dislocation, missing, and distortion, which will affect the subjective visualization effect and need to be solved urgently. Summary of the Invention

[0005] The present application provides a texture mapping method to solve the problem that traditional texture mapping methods are mainly applicable to dense triangular mesh models, and may cause problems such as dislocation, missing, and distortion for vector models, thereby improving the subjective visualization effect of texture mapping.

[0006] To achieve the above objectives, a first embodiment of the present application provides a texture mapping method, comprising the following steps:

[0007] Obtain an initial vectorized model, a discrete image set, and at least one camera pose;

[0008] Based on the discrete image set, obtaining a valid patch label map for each viewpoint, and calculating an optimal viewpoint for each valid patch according to the at least one camera pose and the valid patch label map for each viewpoint; and

[0009] Based on the optimal viewing angle of each valid facet, the initial texture areas of all valid facets are deformed to obtain texture areas after distortion is removed, and the texture areas are discretized to generate a texture set, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set.

[0010] According to one embodiment of the present application, the deformation processing of the initial texture regions of all valid facets based on the optimal viewing angle of each valid facet to obtain the texture region after the distortion is removed includes:

[0011] Selecting a first target facet having a facet area larger than a preset area from all valid facets, and performing triangulation on the initial texture area of ​​the first target facet to obtain a plurality of triangulated triangles;

[0012] Calculating new vertex coordinates of each subdivided triangle after spatial projection according to the centroid coordinates of each subdivided triangle, and determining a homography matrix of the deformation of each subdivided triangle according to the new vertex coordinates of each subdivided triangle after spatial projection;

[0013] Based on the homography matrix of the deformation of each subdivided triangle, a first texture area after the distortion is removed is mapped and generated, and the texture area after the distortion is removed is obtained according to the first texture area and the second texture area of ​​the remaining valid surface patch.

[0014] According to one embodiment of the present application, before deforming the initial texture regions of all valid facets based on the optimal viewing angle of each valid facet to obtain the texture region after the distortion is removed, the method further includes:

[0015] Calculating the coverage of each valid surface at the corresponding optimal viewing angle, and screening out a second target surface whose coverage is less than a preset threshold;

[0016] Based on a preset homography matrix and a preset greedy heuristic selection strategy, other perspective images are transformed to the perspective corresponding to the second target surface until the coverage of the second target surface at the corresponding optimal perspective is greater than or equal to the preset threshold.

[0017] According to one embodiment of the present application, after the coverage rate of the second target patch corresponding to the optimal viewing angle is greater than or equal to the preset threshold, the method further includes:

[0018] Determining whether there is a third target surface patch that meets a preset repair condition among all the valid surface patches;

[0019] If the third target patch that meets the preset restoration condition exists, a distortion-free and complete texture image corresponding to the third target patch is obtained, and the distortion-free and complete texture image is used to restore the corresponding target patch.

[0020] According to one embodiment of the present application, the first target surface that meets the preset repair condition is a surface with missing texture and / or a surface with a texture stretching degree greater than a preset stretching degree.

[0021] According to one embodiment of the present application, acquiring a discrete image set and at least one camera pose includes:

[0022] Acquire a texture image from at least one perspective, and perform point cloud sampling on multiple facets of the initial vectorized model to obtain first point cloud data of each facet;

[0023] Based on the feature matching relationship between the decoded video images, calculating the pose of each perspective and the second point cloud data of each perspective, wherein the density of the second point cloud data is less than that of the first point cloud data;

[0024] The at least one camera pose and the discrete image set are obtained according to the first point cloud data of each facet, the pose of each viewing angle, and the second point cloud data of each viewing angle.

[0025] According to one embodiment of the present application, obtaining the at least one camera pose and the discrete image set according to the first point cloud data of each patch, the pose of each viewpoint, and the second point cloud data of each viewpoint includes:

[0026] Calculating an initial coordinate system transformation relationship based on the model coordinate system of the initial vectorized model and a preset first matching algorithm;

[0027] Calculating a coordinate system transformation matrix based on a preset second matching algorithm and the initial coordinate system transformation relationship;

[0028] The coordinate system transformation matrix is ​​used to transform the pose of each viewing angle into the model coordinate system of the initial vectorized model to obtain the at least one camera pose.

[0029] According to one embodiment of the present application, obtaining the at least one camera pose and the discrete image set according to the first point cloud data of each patch, the pose of each viewpoint, and the second point cloud data of each viewpoint includes:

[0030] Based on the first point cloud data of each patch and the second point cloud data of each perspective, generating a patch label map for each camera at each perspective;

[0031] Filter out valid patch label images with a pixel resolution greater than a preset resolution threshold in the patch label images of each camera at each viewing angle, and obtain a viewing angle-patch association table;

[0032] A patch-viewing angle association table is converted according to the viewing angle-patch association table, and the discrete image set is obtained according to the patch-viewing angle association table.

[0033] According to one embodiment of the present application, before obtaining the effective patch label map for each perspective based on the discrete image set, the method further includes:

[0034] Extracting two-dimensional line features of the texture image of each viewing angle, and establishing a one-to-one correspondence between the two-dimensional line features of the texture image of each viewing angle and the contour line projection of the viewing angle-face association table;

[0035] Constructing an error function between the contour endpoints of the view-face association table and a two-dimensional line segment matching the contour endpoints of the view-face association table;

[0036] Based on the correspondence and the error function, weighted bundle adjustment is performed on the at least one camera pose.

[0037] According to the texture mapping method proposed in the embodiment of the present application, by obtaining an initial vectorized model, a discrete image set and at least one camera pose, based on the discrete image set, a valid patch label map of each perspective can be obtained, and the optimal perspective of each valid patch is calculated based on at least one camera pose and the valid patch label map of each perspective, and based on the optimal perspective of each valid patch, the initial texture area of ​​all valid patches is deformed to obtain a texture area after removing the distortion, and a texture set is generated by the discrete texture area, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set. Therefore, using the textured vectorized model for texture mapping can solve the problem that the traditional texture mapping method is mainly applicable to dense triangular mesh models, and may cause misalignment, missing, distortion, etc. for vectorized models, thereby improving the subjective visualization effect of texture mapping.

[0038] To achieve the above-mentioned objectives, a second embodiment of the present application provides a texture mapping device, comprising:

[0039] An acquisition module, configured to acquire an initial vectorized model, a discrete image set, and at least one camera pose;

[0040] a calculation module, configured to obtain a valid patch label map for each viewpoint based on the discrete image set, and calculate an optimal viewpoint for each valid patch based on the at least one camera pose and the valid patch label map for each viewpoint; and

[0041] A processing module is used to deform the initial texture areas of all valid facets based on the optimal viewing angle of each valid facet to obtain a texture area after the distortion is removed, discretize the texture area to generate a texture set, and obtain a vectorized model containing texture based on the initial vectorized model and the texture set.

[0042] According to one embodiment of the present application, the processing module is specifically configured to:

[0043] Selecting a first target facet having a facet area larger than a preset area from all valid facets, and performing triangulation on the initial texture area of ​​the first target facet to obtain a plurality of triangulated triangles;

[0044] Calculating new vertex coordinates of each subdivided triangle after spatial projection according to the centroid coordinates of each subdivided triangle, and determining a homography matrix of the deformation of each subdivided triangle according to the new vertex coordinates of each subdivided triangle after spatial projection;

[0045] Based on the homography matrix of the deformation of each subdivided triangle, a first texture area after the distortion is removed is mapped and generated, and the texture area after the distortion is removed is obtained according to the first texture area and the second texture area of ​​the remaining valid surface patch.

[0046] According to one embodiment of the present application, before deforming the initial texture regions of all valid facets based on the optimal viewing angle of each valid facet to obtain the texture regions after dedistortion, the processing module further includes:

[0047] a screening unit, configured to calculate the coverage of each valid surface at the corresponding optimal viewing angle, and screen out a second target surface having a coverage less than a preset threshold;

[0048] A transformation unit is used to transform other perspective images to the perspective corresponding to the second target surface based on a preset homography matrix and a preset greedy heuristic selection strategy, until the coverage of the second target surface at the corresponding optimal perspective is greater than or equal to the preset threshold.

[0049] According to one embodiment of the present application, after the coverage of the second target patch corresponding to the optimal viewing angle is greater than or equal to the preset threshold, the transformation unit is further configured to:

[0050] Determining whether there is a third target surface patch that meets a preset repair condition among all the valid surface patches;

[0051] If the third target patch that meets the preset restoration condition exists, a distortion-free and complete texture image corresponding to the third target patch is obtained, and the distortion-free and complete texture image is used to restore the corresponding target patch.

[0052] According to one embodiment of the present application, the third target surface that meets the preset repair condition is a surface with missing texture and / or a surface with a texture stretching degree greater than a preset stretching degree.

[0053] According to one embodiment of the present application, the acquisition module includes:

[0054] an acquisition unit, configured to acquire a texture image from at least one perspective, and perform point cloud sampling on each of the plurality of facets of the initial vectorized model to obtain first point cloud data of each facet;

[0055] a calculation unit, configured to calculate a pose of each viewing angle and second point cloud data of each viewing angle based on a feature matching relationship between the decoded video images, wherein the density of the second point cloud data is less than that of the first point cloud data;

[0056] A processing unit is configured to obtain the at least one camera pose and the discrete image set based on the first point cloud data of each patch, the pose of each viewing angle, and the second point cloud data of each viewing angle.

[0057] According to one embodiment of the present application, the processing unit is specifically configured to:

[0058] Calculating an initial coordinate system transformation relationship based on the model coordinate system of the initial vectorized model and a preset first matching algorithm;

[0059] Calculating a coordinate system transformation matrix based on a preset second matching algorithm and the initial coordinate system transformation relationship;

[0060] The coordinate system transformation matrix is ​​used to transform the pose of each viewing angle into the model coordinate system of the initial vectorized model to obtain the at least one camera pose.

[0061] According to one embodiment of the present application, the processing unit is specifically configured to:

[0062] Based on the first point cloud data of each patch and the second point cloud data of each perspective, generating a patch label map for each camera at each perspective;

[0063] Filter out valid patch label images with a pixel resolution greater than a preset resolution threshold in the patch label images of each camera at each viewing angle, and obtain a viewing angle-patch association table;

[0064] A patch-viewing angle association table is converted according to the viewing angle-patch association table, and the discrete image set is obtained according to the patch-viewing angle association table.

[0065] According to one embodiment of the present application, before obtaining the effective patch label map for each perspective based on the discrete image set, the calculation module is further configured to:

[0066] Extracting two-dimensional line features of the texture image of each viewing angle, and establishing a one-to-one correspondence between the two-dimensional line features of the texture image of each viewing angle and the contour line projection of the viewing angle-face association table;

[0067] Constructing an error function between the contour endpoints of the view-face association table and a two-dimensional line segment matching the contour endpoints of the view-face association table;

[0068] Based on the correspondence and the error function, weighted bundle adjustment is performed on the at least one camera pose.

[0069] According to the texture mapping device proposed in the embodiment of the present application, by obtaining an initial vectorized model, a discrete image set and at least one camera pose, a valid facet label map of each perspective can be obtained based on the discrete image set, and the optimal perspective of each valid facet is calculated based on at least one camera pose and the valid facet label map of each perspective, and based on the optimal perspective of each valid facet, the initial texture area of ​​all valid faces is deformed to obtain a texture area after removing the distortion, and a texture set is generated by the discrete texture area, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set. Therefore, using the vectorized model containing texture for texture mapping can solve the problem that the traditional texture mapping method is mainly applicable to dense triangular mesh models, and may cause misalignment, missing, distortion, etc. for vectorized models, thereby improving the subjective visualization effect of texture mapping.

[0070] To achieve the above-mentioned objectives, the third aspect of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the texture mapping method as described in the above-mentioned embodiment.

[0071] To achieve the above-mentioned objectives, the fourth embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the texture mapping method as described in the above-mentioned embodiment.

[0072] To achieve the above-mentioned objectives, the fifth embodiment of the present application proposes a computer program product, including a computer program, which, when executed by a processor, is used to implement the texture mapping method described in the above-mentioned embodiment.

[0073] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0075] Figure 1 is a flowchart of a texture mapping method in related art;

[0076] Figure 2 Schematic diagram of traditional dense triangle mesh and low-polygon parameterized mesh;

[0077] Figure 3A flowchart of a texture mapping method provided according to an embodiment of the present application;

[0078] Figure 4 Schematic diagram of the arrangement of a texture acquisition device according to one embodiment of the present application;

[0079] Figure 5 A schematic diagram of performing point cloud sampling on a face according to an embodiment of the present application;

[0080] Figure 6 is a schematic diagram of a visible patch label map according to one embodiment of the present application;

[0081] Figure 7 Schematic diagram of converting a view-face association table into a facet-view association table according to one embodiment of the present application;

[0082] Figure 8 A schematic diagram of constructing the association of face groups on the same plane according to one embodiment of the present application;

[0083] Figure 9 Schematic diagram of patch texture effects of different sizes according to an embodiment of the present application;

[0084] Figure 10 Schematic diagram of the presentation effects of the traditional triangular mesh solution and the solution of the present application (after facet perspective selection);

[0085] Figure 11 is a schematic diagram of a texture region set according to an embodiment of the present application;

[0086] Figure 12 A schematic diagram of a process of deforming a texture region according to an embodiment of the present application;

[0087] Figure 13 Schematic diagram of the presentation effects of the traditional triangular mesh solution and the solution of the present application (after texture deformation processing);

[0088] Figure 14 is a schematic diagram of a texture stitching result according to an embodiment of the present application;

[0089] Figure 15 is a schematic diagram of a texture restoration result according to an embodiment of the present application;

[0090] Figure 16 Schematic diagram of two-dimensional line features of a texture image and surface contour line projection according to one embodiment of the present application;

[0091] Figure 17 Schematic diagram showing comparison of optimized front and rear panel contour line projections according to one embodiment of the present application;

[0092] Figure 18 is a general flow chart of a texture mapping method according to one embodiment of the present application;

[0093] Figure 19 Schematic diagram of a texture mapping result obtained for a vectorized model containing texture according to one embodiment of the present application;

[0094] Figure 20 Schematic block diagram of a texture mapping device according to an embodiment of the present application;

[0095] Figure 21 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0096] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0097] The texture mapping method and device proposed according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0098] Before introducing the texture mapping method proposed in the embodiment of the present application, a brief introduction to the texture mapping method in the related art is first given.

[0099] In related technologies, texture mapping methods for traditional dense triangle meshes are as follows: Figure 1 (a) shows the traditional dense triangular mesh diagram. Figure 2 As shown in (a), the method includes the following steps: ① (initial) vectorized model (without texture); ② camera pose optimization based on discrete image set and camera pose; ③ facet perspective selection; ④ texture color adjustment; ⑤ discrete texture region generation texture set; ⑥ vectorized model with texture is obtained based on (initial) vectorized model and texture region set. Texture mapping method for low facet parameterized network is as follows Figure 1 (b) shows the schematic diagram of the low-face parameterized grid. Figure 2 As shown in (b), the method includes the following steps: ① (initial) vectorized model (without texture); ② optimizing the camera pose by combining the discrete image set and the camera pose; ③ selecting the patch perspective; ④ performing texture stitching; ⑤ generating a texture set from discrete texture areas; ⑥ obtaining a vectorized model with texture based on the (initial) vectorized model and the texture area set.

[0100] However, the related technologies have the following problems: (1) Existing texture mapping technologies are mainly applicable to triangular mesh models, and their application to vectorized models is mainly concentrated on large-scale urban buildings. In addition, traditional texture mapping methods may cause problems such as misalignment, missing, and distortion for vector models, affecting the subjective visualization effect; (2) Existing texture mapping technologies generally assume that the images and poses of each view corresponding to the model required for texture generation have been obtained, while in some applications, only the original three-dimensional model may be provided, and additional image data needs to be collected; (3) Existing texture mapping technologies need to extract a large number of line features when performing image alignment optimization. For some textureless or weakly textured indoor scenes, the number of features may not meet the requirements; (4) Existing texture mapping technologies determine the visibility of the patch by calculating the vertices or center points of the patch, and select the optimal perspective through photometric and perceptual consistency. When the patch area is large, there may be judgment errors; (5) Existing texture mapping technologies do not consider the situation where the optimal perspective does not exist due to the excessive tilt of the shooting perspective. In this case, directly generating a texture image will produce large distortion and blur.

[0101] Based on the above problems, the embodiment of the present application proposes a texture mapping method, which is targeted and improved on the basis of the original solution.

[0102] Figure 3 This is a flowchart of a texture mapping method according to an embodiment of the present application.

[0103] For example, Figure 3 As shown, the texture mapping method includes the following steps:

[0104] In step S301, an initial vectorized model, a discrete image set and at least one camera pose are obtained.

[0105] It can be understood that a vectorized model refers to a two-dimensional spatial model oriented towards spatial entities, and two-dimensional or three-dimensional vector data (such as geometric shapes, direction vectors, etc.) are usually used to represent the model. The vectorized model has higher model accuracy and flexibility, and can better handle complex geometric shapes and details. Among them, the initial vectorized model mentioned in this application refers to a textureless vectorized model; the discrete image set is a set of discrete image data, which are usually stored and represented in digital form; the camera pose includes position and stage, the position can represent the coordinates of the camera in three-dimensional space, and the pose can represent the orientation of the camera, including rotation and translation. It should be noted that in the embodiment of the present application, the number of camera poses is consistent with the number of discrete image sets.

[0106] The following describes in detail how to obtain a discrete image set and at least one camera pose.

[0107] As a possible implementation method, obtaining a discrete image set and at least one camera pose includes: collecting a texture image of at least one perspective, and performing point cloud sampling on multiple facets of the initial vectorized model respectively to obtain first point cloud data of each facet; based on the feature matching relationship between the video decoded images, calculating the pose of each perspective and the second point cloud data of each perspective, wherein the density of the second point cloud data is less than that of the first point cloud data; obtaining at least one camera pose and a discrete image set according to the first point cloud data of each facet, the pose of each perspective and the second point cloud data of each perspective.

[0108] Among them, a texture image is a digital image that represents the surface texture of a three-dimensional object in a two-dimensional form, which is used to enhance or simulate the realism of a three-dimensional object; the feature matching relationship between video decoded images refers to the process of establishing feature matching between adjacent frames by extracting feature information of continuous frame images in the video.

[0109] Specifically, for application scenarios with only a single vectorized model (no texture) input, additional work on texture image acquisition is required. First, use Figure 4 The three-directional camera arrangement shown in (a) constitutes a texture acquisition device for acquiring texture images from at least one viewing angle, wherein the field of view covered by each camera is greater than 90°, as shown in FIG. Figure 4 As shown in (b), since the backward view is blocked by the collector, there is no need to place a camera. The three-way distributed camera arrangement can cover the entire model structure. Due to its own aggregation, the vectorized model can be regarded as a collection of multiple triangular facets. By setting a sampling plan according to actual needs, including determining the number, location and distribution of sampling points, and performing point cloud sampling on multiple facets of the initial vectorized model, the first point cloud data of each facet (in the model coordinate system) can be obtained, as shown in Figure 5As shown; Secondly, according to the feature matching relationship between the video decoded images, the position of each camera perspective and the second point cloud data of each perspective can be calculated, that is, the feature extraction algorithm (such as SIFT (ScaleInvariant FeatureTransform (scale-invariant feature transform, etc.) automatically detects and extracts features (such as color, edge, texture, etc.) of each decoded image frame, generates corresponding feature descriptors, and then finds similar or identical feature points based on the feature descriptors, determines the matching relationship between similar or identical feature points, and uses a feature matching algorithm (such as brute force matching, etc.) to find the best matching feature point pair. Then, using the matched feature point pairs, the displacement and motion vector between adjacent frames can be calculated. Combined with known camera intrinsic parameters (such as focal length, principal point coordinates, etc.), the bundle adjustment method or sparse optimization algorithm can be used to calculate the pose of each camera view (in the camera coordinate system). Based on the pose of each camera view, point cloud data (i.e., second point cloud data) in the camera coordinate system of each view is obtained, where the density of the second point cloud data is less than that of the first point cloud data. After obtaining the first point cloud data of each patch, the pose of each view, and the second point cloud data of each view, at least one camera pose and a discrete image set can be obtained based on the first point cloud data of each patch, the pose of each view, and the second point cloud data of each view.

[0110] For ease of understanding, the following further explains how to obtain at least one camera pose.

[0111] As a possible implementation method, in some embodiments, at least one camera pose and discrete image set are obtained based on the first point cloud data of each facet, the pose of each perspective and the second point cloud data of each perspective, including: calculating the initial coordinate system transformation relationship based on the model coordinate system of the initial vectorized model and a preset first matching algorithm; calculating the coordinate system transformation matrix based on the preset second matching algorithm and the initial coordinate system transformation relationship; and using the coordinate system transformation matrix to convert the pose of each perspective to the model coordinate system of the initial vectorized model to obtain at least one camera pose.

[0112] Specifically, based on the model coordinate system of the initial vectorized model and the preset first matching algorithm (such as NDT (Normal Distributions Transform)), the initial coordinate system transformation relationship after rough alignment can be calculated. This can be used as the initial value, and then combined with the preset second matching algorithm (such as ICP (Iterative Closest Point)), the coordinate system transformation matrix after fine alignment can be calculated. This coordinate system transformation matrix can be used to convert the pose of each perspective in the camera coordinate system to at least one camera pose in the model coordinate system of the initial vectorized model.

[0113] The following further explains how to obtain a discrete image set.

[0114] As a possible implementation method, in other embodiments, at least one camera pose and discrete image set are obtained based on the first point cloud data of each patch, the pose of each viewpoint and the second point cloud data of each viewpoint, including: generating a patch label map for each camera at each viewpoint based on the first point cloud data of each patch and the second point cloud data of each viewpoint; screening out valid patch label maps with a pixel resolution greater than a preset resolution threshold in the patch label map of each camera at each viewpoint to obtain a viewpoint-patch association table; converting the viewpoint-patch association table to obtain a patch-viewpoint association table, and obtaining a discrete image set based on the patch-viewpoint association table.

[0115] The preset resolution threshold may be pre-set by researchers in this field, obtained through a limited number of experiments, or obtained through a limited number of computer simulations, and is not specifically limited here.

[0116] Specifically, based on the first point cloud data of each patch and the second point cloud data of each viewpoint, a BVH (Bounding Volume Hierarchy) tree can be used to generate a label map of patches visible to each camera at each viewpoint, such as Figure 6 As shown in the figure (different color areas in the label map represent the ID of the model patch), and according to formula (1), the patch label maps with pixel resolution greater than the preset resolution threshold in the patch label map of each camera at each viewing angle are selected as valid patch label maps, thereby obtaining the association table of camera viewing angle for patch labels, that is, the viewing angle-patch association table (as shown in Figure 7 (a)), the view-face association table is further converted into the association table of each face for the camera view, that is, the face-view association table (such as Figure 7 (b) Finally, a discrete image set can be obtained based on the patch-view association table. This discrete image set is the union of the patch-view association table. Therefore, by filtering out discrete image sets from a continuous video, the time consumed by subsequent texture data processing can be reduced.

[0117]

[0118] Among them, v i is the vertex of the patch, i is a natural number, Z is the Z coordinate of the vertex, f x is the focal length of the camera in the x direction, f y is the focal length of the camera in the y direction.

[0119] In step S302, based on the discrete image set, a valid patch label map of each viewpoint is obtained, and the optimal viewpoint of each valid patch is calculated according to at least one camera pose and the valid patch label map of each viewpoint.

[0120] Specifically, based on a discrete image set, a BVH tree can also be used to generate a patch label map for each camera at each viewpoint, from which patch label maps with pixel resolution greater than a preset resolution threshold are selected as valid patch label maps for each viewpoint; since each patch may be observed by multiple camera viewpoints, they need to be sorted according to priority to obtain the optimal viewpoint. First, based on at least one camera pose and the valid patch label map for each viewpoint, the association of patch groups belonging to the same plane is constructed, such as Figure 8 As shown, the scaling contrast of the visible area and edge projection at each viewing angle is calculated using equations (2) to (5), the number of samples and the uniformity of the samples at each viewing angle are analyzed, and the Markov random field is solved using the constraint that the labels of the same plane group are consistent, thereby obtaining the optimal viewing angle corresponding to each valid patch. The embodiment of the present application can simultaneously process patch results of different sizes, such as Figure 9 As shown, Figure 9 (a) is the large-scale structural texture effect (wall), Figure 9 (b) is the small-scale structural texture effect (door). Figure 10 As shown, Figure 10 (a) is the rendering effect of the traditional triangular mesh solution. Figure 10 (b) is the presentation effect of the solution of the present application. The embodiment of the present application can significantly improve the visualization effect by selecting the optimal viewing angle of the facet.

[0121]

[0122] stl i =l j ,F i ,F j ∈Plane k ,1≤i,j≤N,1≤k≤M

[0123]

[0124]

[0125]

[0126] Where F is the effective patch label, l is the view label, i and j are both positive integers, A is the projection area of ​​the effective patch under the view, Mask is the binary template generated by the view patch label map, r is the ratio of the patch contour line length to the image projection line length, λn is the sampling weight, λ u is sampling uniformity, λ c is the projection consistency weight.

[0127] In step S303, based on the optimal viewing angle of each valid patch, the initial texture areas of all valid patches are deformed to obtain the texture areas after the distortion is removed, and the texture areas are discretized to generate a texture set, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set.

[0128] It is understandable that in order to avoid the problem of visualization distortion caused by directly selecting the texture area when the area of ​​the effective patch is large, the initial texture area of ​​all effective patches can be deformed based on the optimal viewing angle of each effective patch to obtain the texture area after the distortion is removed, thereby improving the visualization effect. Figure 11 As shown, a 2D binning algorithm (such as best fit, worst fit, and first fit) is used to group the texture regions of each valid patch into a set. This means that the discrete texture regions are placed into a unified texture set framework to generate a texture set. Based on the initial vectorized model and the texture set, a textured vectorized model is obtained. Using a 2D binning algorithm to generate a texture set from discrete texture regions effectively manages and optimizes texture resources, improving rendering efficiency and reducing resource waste.

[0129] Next, we will elaborate on how to deform the initial texture area.

[0130] As a possible implementation method, in some embodiments, based on the optimal viewing angle of each valid facet, the initial texture areas of all valid facets are deformed to obtain the texture area after the distortion is removed, including: screening out a first target facet with a facet area greater than a preset area from all valid facets, and triangulating the initial texture area of ​​the first target facet to obtain multiple subdivided triangles; calculating the new vertex coordinates of each subdivided triangle after spatial projection based on the centroid coordinates of each subdivided triangle, and determining the deformed homography matrix of each subdivided triangle based on the new vertex coordinates after spatial projection of each subdivided triangle; based on the deformed homography matrix of each subdivided triangle, mapping is performed to generate the first texture area after the distortion is removed, and obtaining the texture area after the distortion is removed based on the first texture area and the second texture areas of the remaining valid facets.

[0131] The preset area may be pre-set by researchers in this field, obtained through a limited number of experiments, or obtained through a limited number of computer simulations, and is not specifically limited here.

[0132] Specifically, the process of deforming the initial texture area of ​​all valid patches is as follows: Figure 12As shown, first, the patches with an area greater than a preset area are screened out from all valid patches as the first target patches, and the initial texture areas of all first target patches are subjected to length-constrained Delaunay triangulation to form multiple triangular segments, wherein Delaunay triangulation is a method for dividing a two-dimensional plane into a set of non-intersecting triangles, and the length constraint is to ensure that the side lengths of the formed triangles meet certain preset conditions when triangulating the triangles; secondly, according to the centroid coordinates of each triangular patch, the new vertex coordinates of the triangular space projection of each first target patch are reversely calculated, that is, the new vertex coordinates of the triangular space projection of each first target patch are reversely deduced based on the known centroid coordinates. The vertex coordinates of the triangle are obtained, and then the homography matrix of each subdivided triangle deformation is determined according to the new vertex coordinates after the spatial projection of each subdivided triangle. The homography matrix is ​​a matrix used to describe the geometric relationship between two planes (i.e. translation, rotation and scaling). Here it is used to describe how the subdivided triangle is deformed or distorted; finally, the original texture area is transformed using the homography matrix of each subdivided triangle deformation, and the transformed texture area is mapped to the original texture area through texture mapping technology to generate a first texture area after removing the distortion. The texture area of ​​all valid patches after removing the distortion is obtained according to the first texture area and the second texture area of ​​the remaining valid patches, as shown in FIG. Figure 13 As shown, Figure 13 (a) is the rendering effect of the traditional triangular mesh solution. Figure 13 (b) is the presentation effect of the embodiment scheme of this application.

[0133] Furthermore, in some embodiments, before deforming the initial texture areas of all valid facets based on the optimal viewing angle of each valid facet to obtain the texture area after removing the distortion, it also includes: calculating the coverage of each valid facet at the corresponding optimal viewing angle, and screening out the second target facet whose coverage is less than a preset threshold; based on a preset homography matrix and a preset greedy heuristic selection strategy, transforming other perspective images to the viewing angle corresponding to the second target facet until the coverage of the second target facet at the corresponding optimal viewing angle is greater than or equal to the preset threshold.

[0134] The preset threshold value may be pre-set by researchers in the field, obtained through a limited number of experiments, or obtained through a limited number of computer simulations, and is not specifically limited here. The preset greedy heuristic selection strategy is a heuristic search strategy used to make locally optimal choices when solving optimization problems in the hope of achieving a global optimal solution. The core idea of ​​the greedy heuristic selection strategy is to select the best (optimal) choice in the current state in each selection, so that the result is the best (optimal) solution globally.

[0135] That is to say, after selecting the optimal viewing angle corresponding to each valid patch, the coverage rate of each valid patch at the viewing angle can be further calculated, and the second target patch with a coverage rate less than a preset threshold (such as 100%) can be screened out. The preset homography matrix is ​​used to transform the other viewing angle images to the viewing angle corresponding to the second target patch, and the other viewing angles are greedily heuristically selected in sequence based on the preset greedy heuristic selection strategy until the coverage rate of the second target patch at the corresponding optimal viewing angle is greater than or equal to the preset threshold, as shown in formula (6).

[0136]

[0137] Among them, F is the valid patch label and l is the view label.

[0138] When a valid patch is covered by multiple viewpoints, that is, the texture image of the valid patch is composed of multiple viewpoint images, the local texture area can be spliced ​​into a complete texture area, and the other viewpoint images can be mapped to the optimal viewpoint using the homography matrix (such as Equation (7) and Equation (8)). After exposure compensation, optimal seam line search and multi-band fusion operations, a complete texture area is generated, as shown in Figure 14 shown.

[0139]

[0140]

[0141] Where s is a constant, p is the pixel coordinate, H is the homography matrix, K, R, and t are the camera intrinsic parameters, rotation, and translation, respectively, n is the unit normal vector of the patch parameter equation, d is the distance from the origin to the plane, and T is the transpose operation of the vector.

[0142] Furthermore, in some embodiments, after the coverage of the second target surface corresponding to the optimal viewing angle is greater than or equal to a preset threshold, it also includes: determining whether there is a third target surface that meets the preset repair conditions among all valid surfaces; if there is a third target surface that meets the preset repair conditions, obtaining the distortion-free and complete texture image corresponding to the third target surface, and using the distortion-free and complete texture image to repair the corresponding target surface.

[0143] In some embodiments, the third target surface that meets the preset repair condition is a surface with missing texture and / or a surface with a texture stretching degree greater than a preset stretching degree.

[0144] It is understandable that in order to avoid the situation where the texture area of ​​the selected patch is greatly stretched or missing, it is possible to determine whether there is a third target patch that meets the preset repair conditions among all valid patches, that is, to determine whether there is a texture area with a large stretching or missing problem among all valid patches. When faced with the third target patch that meets the preset repair conditions, relevant technical means such as AIGC (Artificial Intelligence Generated Content) technology can be used to obtain a distortion-free and complete texture image (that is, a texture image with a style similar to the real scene image) corresponding to the third target patch through a generative neural network model, and use the distortion-free and complete texture image to repair the corresponding target patch, such as Figure 15 As shown, Figure 15 (a) is a real scene texture image with stretching. Figure 15 (b) is a non-stretched texture image with similar style generated by the generative neural network model. Figure 15 (c) is the real scene texture image with missing textures. Figure 15 (d) is a texture image with no missing content and similar style generated by the generative neural network model.

[0145] In addition, in some embodiments, before obtaining a valid patch label map for each perspective based on a discrete image set, the method further includes: extracting two-dimensional line features of the texture image of each perspective, and establishing a one-to-one correspondence between the two-dimensional line features of the texture image of each perspective and the contour line projection of the perspective-patch association table; constructing an error function between the contour line endpoints of the perspective-patch association table and the two-dimensional line segments that match the contour line endpoints of the perspective-patch association table; and weightedly adjusting at least one camera pose based on the correspondence and the error function.

[0146] It is understandable that in order to correct the possible errors in the initial camera pose and improve the position accuracy of the texture area, at least one camera pose can be optimized. First, the two-dimensional line features of the texture image of each perspective can be extracted by relevant technical means (such as edge detection, line detection and other algorithms) (such as Figure 16 (a)), such as edge features, line features, etc., and use relevant technical means (such as contour rendering based on contour edge detection, etc.) to calculate the contour projection of the view-face association table (such as Figure 16(b) shows), the projection of the contour line can be regarded as a two-dimensional representation of the patch in a certain direction, such as the intersection line of the wall and the ground. By calculating the similarity or distance between the features (such as cosine similarity or Euclidean distance, etc.), the two-dimensional line features of the texture image of each perspective and the contour line projection of the perspective-patch association table are matched. According to the matching results, a one-to-one correspondence between the two-dimensional line features of the texture image of each perspective and the contour line projection of the perspective-patch association table is established; secondly, based on the two-dimensional line features of the texture image of each perspective and the contour line projection of the perspective-patch association table, an error function between the contour line endpoints of the perspective-patch association table and the two-dimensional line segments matching the contour line endpoints of the perspective-patch association table is constructed, as shown in formulas (9) to (11); finally, based on the correspondence and the error function, at least one camera pose is weighted bundle adjusted to obtain a new camera pose for subsequent texture area selection. Among them, weighted bundle adjustment is an optimization algorithm. The contour line projection of the patch before optimization adjustment is as shown in Figure 17 As shown in (a), the contour line projection of the optimized and adjusted patch is as follows Figure 17 (b) shown.

[0147]

[0148]

[0149]

[0150] Among them, R is the rotation amount to be optimized for the current viewpoint, t is the translation amount to be optimized for the current viewpoint, S is the set of matching pairs of image 2D line segments and patch contour lines, P and Q are the two endpoints of the patch contour line, I is the image 2D line segment parameter, K is the camera intrinsic parameter, π is the coordinate of the spatial point projected onto the image plane, λ' is the weight of the loss function, L is the length of the image 2D line segment, T is the transpose operation of the vector, and σ is a constant.

[0151] To facilitate those skilled in the art to further understand the texture mapping method proposed in the embodiment of the present application, the following is a Figure 18 Further elaboration.

[0152] like Figure 18 As shown, Figure 18 This is a general flow chart of the texture mapping method proposed in the embodiment of the present application.

[0153] The whole scheme is divided into two parts: texture data acquisition and texture data processing. Among them, the first part, texture data acquisition process, is an optional scheme, which is mainly used in scenarios with only a single vectorized model (no texture) input, including (1) point cloud sampling of the vectorized model (no texture) to obtain dense point cloud data in the model coordinate system; (2) RGB data acquisition device (three-dimensionally arranged camera) to obtain image stream; (3) using visual SFM (Structure From Motion, traditional 3D reconstruction) to obtain sparse point cloud data in the camera coordinate system; (4) coordinate system alignment, that is, according to the dense point cloud data in the model coordinate system and the sparse point cloud data in the camera coordinate system, the camera pose in the model coordinate system is obtained; (5) texture image screening to obtain a discrete image set.

[0154] The second part of the texture data processing process is mainly used to input a scene that already contains a vectorized model (without texture), a discrete image set and an initial camera pose, which includes (1) camera pose optimization to obtain a new camera pose; (2) patch label calculation to generate visible patch label maps for each perspective; (3) patch perspective selection to obtain the optimal perspective corresponding to the patch; (4) judging whether the coverage of the patch at the corresponding optimal perspective is complete (i.e., whether it reaches 100%); (5) if not, texture splicing processing is performed to splice the local texture area into a complete texture area, otherwise, it is judged whether the texture area is obviously stretched or missing; (6) if so, texture repair processing is performed to obtain a complete and clear texture image; (7) texture deformation processing is performed to obtain the texture area after removing the distortion; (8) texture set generation; (9) the vectorized model (with texture) is obtained based on the vectorized model (without texture) and the texture area set (texture set).

[0155] That is to say, for the texture mapping method proposed in the embodiment of the present application, the texture data acquisition process is not necessary. If the input is a scene with only a single vectorized model (no texture), it is necessary to first perform a texture data acquisition process to obtain a discrete image set and a camera pose, and then perform a texture data processing process; if the input contains both a vectorized model (no texture) and a discrete image set and an initial camera pose, there is no need to perform a texture data acquisition process, and the texture data processing process can be performed directly.

[0156] The texture mapping results obtained from the vectorized model with texture, such as Figure 19 As shown, Figure 19 (a) is a schematic diagram of texture mapping results of a vectorized model without texture; Figure 19 (b) is a schematic diagram of the texture mapping results of the vectorized model with texture; Figure 19(c) is a schematic diagram of IToF (Indirect Time-of-Flight) of a vectorized model with texture; Figure 19 (d) is a schematic diagram of RGB depth estimation of a vectorized model with texture; Figure 19 (e) is a schematic diagram of the Lidar (true value) results of the vectorized model with texture.

[0157] According to the texture mapping method proposed in the embodiment of the present application, by obtaining an initial vectorized model, a discrete image set and at least one camera pose, based on the discrete image set, a valid patch label map of each perspective can be obtained, and the optimal perspective of each valid patch is calculated based on at least one camera pose and the valid patch label map of each perspective, and based on the optimal perspective of each valid patch, the initial texture area of ​​all valid patches is deformed to obtain a texture area after removing the distortion, and a texture set is generated by the discrete texture area, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set. Therefore, using the textured vectorized model for texture mapping can solve the problem that the traditional texture mapping method is mainly applicable to dense triangular mesh models, and may cause misalignment, missing, distortion, etc. for vectorized models, thereby improving the subjective visualization effect of texture mapping.

[0158] Next, the texture mapping device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0159] Figure 20 It is a block diagram of a texture mapping device according to an embodiment of the present application.

[0160] like Figure 20 As shown, the texture mapping device 10 includes: an acquisition module 100 , a calculation module 200 and a processing module 300 .

[0161] The acquisition module 100 is configured to acquire an initial vectorized model, a discrete image set, and at least one camera pose;

[0162] A calculation module 200 is configured to obtain a valid patch label map for each viewpoint based on a discrete image set, and calculate an optimal viewpoint for each valid patch based on at least one camera pose and the valid patch label map for each viewpoint; and

[0163] The processing module 300 is used to deform the initial texture areas of all valid facets based on the optimal viewing angle of each valid facet to obtain the texture areas after the distortion is removed, and to discretize the texture areas to generate a texture set, and to obtain a vectorized model containing texture based on the initial vectorized model and the texture set.

[0164] Furthermore, in some embodiments, the processing module 300 is specifically configured to:

[0165] Selecting a first target facet with a facet area larger than a preset area from all valid facets, and performing triangulation on the initial texture area of ​​the first target facet to obtain a plurality of triangulated triangles;

[0166] Calculate the new vertex coordinates of each subdivided triangle after spatial projection according to the centroid coordinates of each subdivided triangle, and determine the homography matrix of each subdivided triangle deformation according to the new vertex coordinates of each subdivided triangle after spatial projection;

[0167] Based on the homography matrix of each subdivided triangle deformation, a first texture region after the distortion is removed is generated by mapping, and a texture region after the distortion is removed is obtained according to the first texture region and the second texture region of the remaining valid facets.

[0168] Furthermore, in some embodiments, before deforming the initial texture regions of all valid facets to obtain the dedistorted texture regions based on the optimal viewing angle of each valid facet, the processing module 300 further includes:

[0169] A screening unit, configured to calculate the coverage of each valid facet at the corresponding optimal viewing angle, and screen out a second target facet having a coverage less than a preset threshold;

[0170] A transformation unit is used to transform other perspective images to the perspective corresponding to the second target surface based on a preset homography matrix and a preset greedy heuristic selection strategy, until the coverage of the second target surface at the corresponding optimal perspective is greater than or equal to a preset threshold.

[0171] Furthermore, in some embodiments, after the coverage of the second target patch corresponding to the optimal viewing angle is greater than or equal to a preset threshold, the transformation unit is further configured to:

[0172] Determine whether there is a third target surface that meets the preset repair conditions among all valid surfaces;

[0173] If there is a third target patch that meets the preset restoration condition, a distortion-free and complete texture image corresponding to the third target patch is obtained, and the distortion-free and complete texture image is used to restore the corresponding target patch.

[0174] Furthermore, in some embodiments, the third target patch that meets the preset repair condition is a patch with missing texture and / or a patch with a texture stretching degree greater than a preset stretching degree.

[0175] Furthermore, in some embodiments, the acquisition module 100 includes:

[0176] an acquisition unit, configured to acquire a texture image from at least one perspective, and perform point cloud sampling on a plurality of facets of the initial vectorized model to obtain first point cloud data of each facet;

[0177] a calculation unit, configured to calculate a pose of each viewing angle and second point cloud data of each viewing angle based on a feature matching relationship between the decoded video images, wherein the density of the second point cloud data is less than that of the first point cloud data;

[0178] A processing unit is used to obtain at least one camera pose and a discrete image set based on the first point cloud data of each facet, the pose of each perspective, and the second point cloud data of each perspective.

[0179] Furthermore, in some embodiments, the processing unit is specifically configured to:

[0180] Calculating the initial coordinate system transformation relationship based on the model coordinate system of the initial vectorized model and a preset first matching algorithm;

[0181] Calculating a coordinate system transformation matrix based on a preset second matching algorithm and an initial coordinate system transformation relationship;

[0182] The coordinate system transformation matrix is ​​used to transform the pose of each view into the model coordinate system of the initial vectorized model to obtain at least one camera pose.

[0183] Furthermore, in some embodiments, the processing unit is specifically configured to:

[0184] Based on the first point cloud data of each patch and the second point cloud data of each perspective, generate a patch label map for each camera at each perspective;

[0185] Filter out valid patch label images with pixel resolution greater than a preset resolution threshold in the patch label images of each camera at each viewing angle, and obtain a viewing angle-patch association table;

[0186] The view-face association table is converted into a face-view association table, and a discrete image set is obtained according to the face-view association table.

[0187] Furthermore, in some embodiments, before obtaining a valid patch label map for each viewpoint based on the discrete image set, the computing module 200 is further configured to:

[0188] Extracting the two-dimensional line features of the texture image of each viewpoint, and establishing a one-to-one correspondence between the two-dimensional line features of the texture image of each viewpoint and the contour line projection of the viewpoint-face association table;

[0189] Constructing an error function between the contour endpoints of the view-face association table and the two-dimensional line segments that match the contour endpoints of the view-face association table;

[0190] Based on the correspondences and the error function, a weighted bundle adjustment is performed on at least one camera pose.

[0191] It should be noted that the above explanation of the texture mapping method embodiment is also applicable to the texture mapping device of this embodiment, and will not be repeated here.

[0192] According to the texture mapping device proposed in the embodiment of the present application, by obtaining an initial vectorized model, a discrete image set and at least one camera pose, a valid facet label map of each perspective can be obtained based on the discrete image set, and the optimal perspective of each valid facet is calculated based on at least one camera pose and the valid facet label map of each perspective, and based on the optimal perspective of each valid facet, the initial texture area of ​​all valid faces is deformed to obtain a texture area after removing the distortion, and a texture set is generated by the discrete texture area, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set. Therefore, using the vectorized model containing texture for texture mapping can solve the problem that the traditional texture mapping method is mainly applicable to dense triangular mesh models, and may cause misalignment, missing, distortion, etc. for vectorized models, thereby improving the subjective visualization effect of texture mapping.

[0193] Figure 21 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0194] Memory 2101 , processor 2102 , and computer programs stored in the memory 2101 and executable on the processor 2102 .

[0195] When the processor 2102 executes the program, the texture mapping method provided in the above embodiment is implemented.

[0196] Furthermore, the electronic device further includes:

[0197] The communication interface 2103 is used for communication between the memory 2101 and the processor 2102 .

[0198] The memory 2101 is used to store computer programs that can be run on the processor 2102.

[0199] The memory 2101 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0200] If the memory 2101, the processor 2102, and the communication interface 2103 are implemented independently, the communication interface 2103, the memory 2101, and the processor 2102 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 21 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0201] Optionally, in a specific implementation, if the memory 2101, the processor 2102 and the communication interface 2103 are integrated on a chip, the memory 2101, the processor 2102 and the communication interface 2103 can communicate with each other through an internal interface.

[0202] The processor 2102 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0203] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the above texture mapping method when executed by a processor.

[0204] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above texture mapping method when executed by a processor.

[0205] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0206] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0207] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A texture mapping method, characterized in that: The following steps are involved: Obtain an initial vectorized model, a discrete image set, and at least one camera pose; Based on the discrete image set, obtaining a valid patch label map for each perspective, and calculating the optimal perspective of each valid patch according to the at least one camera pose and the valid patch label map for each perspective; as well as Based on the optimal viewing angle of each valid facet, the initial texture areas of all valid facets are deformed to obtain texture areas after distortion is removed, and the texture areas are discretized to generate a texture set, and a vectorized model containing texture is obtained based on the initial vectorized model and the texture set.

2. The method according to claim 1, characterized in that The step of deforming the initial texture regions of all valid facets based on the optimal viewing angle of each valid facet to obtain a texture region after removing the distortion comprises: Selecting a first target facet having a facet area larger than a preset area from all valid facets, and performing triangulation on the initial texture area of ​​the first target facet to obtain a plurality of triangulated triangles; Calculating new vertex coordinates of each subdivided triangle after spatial projection according to the centroid coordinates of each subdivided triangle, and determining a homography matrix of the deformation of each subdivided triangle according to the new vertex coordinates of each subdivided triangle after spatial projection; Based on the homography matrix of the deformation of each subdivided triangle, a first texture area after the distortion is removed is mapped and generated, and the texture area after the distortion is removed is obtained according to the first texture area and the second texture area of ​​the remaining valid surface patch.

3. The method according to claim 1, characterized in that Before deforming the initial texture regions of all valid facets based on the optimal viewing angle of each valid facet to obtain the texture regions after the distortion is removed, the method further includes: Calculating the coverage of each valid surface at the corresponding optimal viewing angle, and screening out a second target surface whose coverage is less than a preset threshold; Based on a preset homography matrix and a preset greedy heuristic selection strategy, other perspective images are transformed to the perspective corresponding to the second target surface until the coverage of the second target surface at the corresponding optimal perspective is greater than or equal to the preset threshold.

4. The method according to claim 3, characterized in that After the coverage rate of the second target patch corresponding to the optimal viewing angle is greater than or equal to the preset threshold, the method further includes: Determining whether there is a third target surface patch that meets a preset repair condition among all the valid surface patches; If the third target patch that meets the preset restoration condition exists, a distortion-free and complete texture image corresponding to the third target patch is obtained, and the distortion-free and complete texture image is used to restore the corresponding target patch.

5. The method according to claim 4, characterized in that The third target patch that meets the preset repair condition is a patch with missing texture and / or a patch with a texture stretching degree greater than a preset stretching degree.

6. The method according to claim 1, characterized in that The obtaining of a discrete image set and at least one camera pose comprises: Acquire a texture image from at least one perspective, and perform point cloud sampling on multiple facets of the initial vectorized model to obtain first point cloud data of each facet; Based on the feature matching relationship between the decoded video images, calculating the pose of each perspective and the second point cloud data of each perspective, wherein the density of the second point cloud data is less than that of the first point cloud data; The at least one camera pose and the discrete image set are obtained according to the first point cloud data of each facet, the pose of each viewing angle, and the second point cloud data of each viewing angle.

7. The method according to claim 6, characterized in that Obtaining the at least one camera pose and the discrete image set according to the first point cloud data of each patch, the pose of each viewpoint, and the second point cloud data of each viewpoint includes: Calculating an initial coordinate system transformation relationship based on the model coordinate system of the initial vectorized model and a preset first matching algorithm; Calculating a coordinate system transformation matrix based on a preset second matching algorithm and the initial coordinate system transformation relationship; The coordinate system transformation matrix is ​​used to transform the pose of each viewing angle into the model coordinate system of the initial vectorized model to obtain the at least one camera pose.

8. The method according to claim 6, characterized in that Obtaining the at least one camera pose and the discrete image set according to the first point cloud data of each patch, the pose of each viewpoint, and the second point cloud data of each viewpoint includes: Based on the first point cloud data of each patch and the second point cloud data of each perspective, generating a patch label map for each camera at each perspective; Filter out valid patch label images with a pixel resolution greater than a preset resolution threshold in the patch label images of each camera at each viewing angle, and obtain a viewing angle-patch association table; A patch-viewing angle association table is converted according to the viewing angle-patch association table, and the discrete image set is obtained according to the patch-viewing angle association table.

9. The method according to claim 8, characterized in that Before obtaining the effective patch label map of each perspective based on the discrete image set, the method further includes: Extracting two-dimensional line features of the texture image of each viewing angle, and establishing a one-to-one correspondence between the two-dimensional line features of the texture image of each viewing angle and the contour line projection of the viewing angle-face association table; Constructing an error function between the contour endpoints of the view-face association table and a two-dimensional line segment matching the contour endpoints of the view-face association table; Based on the correspondence and the error function, weighted bundle adjustment is performed on the at least one camera pose.

10. A texture mapping device, characterized in that: include: An acquisition module, configured to acquire an initial vectorized model, a discrete image set, and at least one camera pose; A calculation module, configured to obtain a valid patch label map for each perspective based on the discrete image set, and calculate an optimal perspective for each valid patch based on the at least one camera pose and the valid patch label map for each perspective; as well as A processing module is used to deform the initial texture areas of all valid facets based on the optimal viewing angle of each valid facet to obtain a texture area after the distortion is removed, discretize the texture area to generate a texture set, and obtain a vectorized model containing texture based on the initial vectorized model and the texture set.

Images