Texture processing method and device of three-dimensional model, equipment and storage medium

By mixing the pixel values ​​of the common-view area of ​​multi-view pictures in three-dimensional texture reconstruction and taking into account the depth and normal direction, the problem of adjacent patches is solved, and the accuracy of three-dimensional texture reconstruction is improved.

CN119068106APending Publication Date: 2024-12-03TENCENT TECH SHANGHAI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310636792.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing three-dimensional texture reconstruction technology can easily lead to misalignment of adjacent patches when the model is rough or the lighting conditions change, affecting the accuracy of texture reconstruction.

Method used

By obtaining multiple viewing images, mixing the corresponding common-view area pixel values, generating a mixed picture, and building the texture of the object model based on the mixed picture, considering pixel depth and normal direction to weight the blending to ensure color consistency.

Benefits of technology

Improve color consistency at multiple perspectives, avoid misalignment of adjacent patch textures, and improve the accuracy of three-dimensional texture reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068106A_ABST
    Figure CN119068106A_ABST
Patent Text Reader

Abstract

The invention relates to a texture processing method and device for a three-dimensional model, equipment and a storage medium, and relates to the technical field of three-dimensional reconstruction. The method comprises the following steps: acquiring n view angle pictures; the n view angle pictures are pictures of the target object under n camera view angles respectively; based on respective camera visual angles of the n visual angle pictures, pixel values of common-view areas between every two of the n visual angle pictures are mixed, and respective mixed pictures of the n visual angle pictures are obtained; based on the respective mixed picture of the n view angle pictures and the respective camera view angle of the n view angle pictures, obtaining the texture of each surface patch on the surface of the object model; the object model is a geometric model of the target object. According to the scheme, the accuracy of three-dimensional texture reconstruction can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a texture processing method, device, equipment and storage medium for a three-dimensional model. Background Art

[0002] Three-dimensional texture reconstruction refers to the process of reconstructing the surface texture of a three-dimensional model of a person / object by taking a two-dimensional picture of the person / object.

[0003] In the related art, the three-dimensional texture reconstruction process may include: calculating the normal direction of each facet of the model, selecting a camera perspective closest to it from multiple perspectives, and assigning the color of the image taken at this perspective as the texture of the facet.

[0004] However, in the solution shown in the above-mentioned related technology, the reconstruction method in which each facet corresponds to the color under a viewing angle has high requirements on the model's fineness and lighting conditions. If the model is relatively rough or there is a difference in lighting under different viewing angles, it is easy to cause the problem of adjacent facets not being aligned, affecting the accuracy of texture reconstruction. Summary of the invention

[0005] The embodiment of the present application provides a texture processing method, device, equipment and storage medium for a three-dimensional model, which can be applied to the modeling of objects with open boundaries and non-open boundaries, expand the application scope of three-dimensional object modeling, and improve the modeling effect of three-dimensional objects. The technical solution is as follows:

[0006] In one aspect, a texture processing method for a three-dimensional model is provided, the method comprising:

[0007] Obtain n perspective images; the n perspective images are images of the target object under n camera perspectives respectively;

[0008] Based on the camera perspectives of the n perspective pictures, pixel values ​​of common viewing areas between any two of the n perspective pictures are mixed to obtain mixed pictures of the n perspective pictures;

[0009] Based on the mixed images of the n perspective images and the camera perspectives of the n perspective images, textures of each facet on the surface of an object model are acquired; the object model is a geometric model of the target object.

[0010] On the other hand, a texture processing device for a three-dimensional model is provided, the device comprising:

[0011] An image acquisition module is used to acquire n perspective images; the n perspective images are images of the target object under n camera perspectives respectively;

[0012] A pixel mixing module, used for mixing pixel values ​​of common viewing areas between two of the n viewing angle pictures based on the camera viewing angles of the n viewing angle pictures, to obtain mixed pictures of the n viewing angle pictures;

[0013] The texture acquisition module is used to acquire the texture of each facet on the surface of the object model based on the mixed image of each of the n perspective images and the camera perspective of each of the n perspective images; the object model is a geometric model of the target object.

[0014] In a possible implementation manner, the device further includes:

[0015] a depth acquisition module, configured to acquire pixel depth information of each of the n perspective images based on the n perspective images and the object model before the pixel blending module blends pixel values ​​of a common view area between the n perspective images based on the respective camera perspectives of the n perspective images to obtain respective blended images of the n perspective images; the pixel depth information is used to indicate the depth of each pixel in the perspective image;

[0016] A weight acquisition module, used to acquire pixel weight information of each of the n perspective pictures based on the pixel depth information of each of the n perspective pictures; the pixel weight information is used to indicate the mixing weight of each pixel in the perspective picture;

[0017] The pixel mixing module is used to weightedly mix the pixel values ​​of the common viewing area between the n perspective images based on the camera perspectives of each of the n perspective images and the pixel weight information of each of the n perspective images to obtain a mixed image of each of the n perspective images.

[0018] In a possible implementation, the weight acquisition module is used to:

[0019] For a first pixel in a first viewing angle picture, obtaining a first normal direction at a position point on the object model corresponding to the first pixel; the first viewing angle picture is any one viewing angle picture among the n viewing angle pictures; the first pixel is any one pixel in the first viewing angle picture;

[0020] Based on the depth of the first pixel and the angle between the first normal direction and a first viewing direction, a mixing weight of the first pixel is obtained; the first viewing direction is a direction from a camera center to the first pixel under the camera viewing angle of the first viewing image.

[0021] In a possible implementation, the pixel mixing module is used to:

[0022] For the second viewing angle picture, obtain each common viewing area picture of the second viewing angle picture; the common viewing area picture is a viewing angle picture among the n viewing angle pictures, which has a common viewing area with the second viewing angle picture; the second viewing angle picture is any viewing angle picture among the n viewing angle pictures;

[0023] Mapping pixels of the common viewing area between each of the common viewing area pictures and the second viewing angle picture into a pixel coordinate system of the second viewing angle picture to obtain mapping coordinates of pixels of the common viewing area between each of the common viewing area pictures and the second viewing angle picture;

[0024] Based on the pixel coordinates of each pixel in the second perspective picture, the mapping coordinates of the pixels of the common view area between each of the common view area pictures and the second perspective picture, and the mixing weights of the pixels of the common view area between each of the common view area pictures and the second perspective picture, the pixels of the common view area between each of the common view area pictures and the second perspective picture are weighted mixed to obtain a mixed picture of the second perspective picture.

[0025] In a possible implementation, the pixel mixing module is used to:

[0026] Based on the normal direction of each facet on the surface of the object model and the camera perspective of each of the n perspective images, obtaining a score of each facet under the camera perspective of each of the n perspective images;

[0027] Determine a target camera perspective of each facet based on the score of each facet under the camera perspective of each of the n perspective images;

[0028] The texture of each facet is obtained based on the pixel value of each facet in the mixed picture corresponding to the respective target camera perspective.

[0029] In a possible implementation, the pixel mixing module is used to:

[0030] For a projection area of ​​the first face piece in a third viewing angle picture, obtaining a normal direction at a position point on the object model corresponding to each pixel in the projection area; the third viewing angle picture is any one viewing angle picture among the n viewing angle pictures;

[0031] Obtaining the viewing direction of each pixel in the projection area; the viewing direction of each pixel in the projection area is the direction from the camera center to each pixel in the projection area under the camera viewing angle of the third viewing angle image;

[0032] Based on the normal direction of the position point on the object model corresponding to each pixel in the projection area, and the viewing direction of each pixel in the projection area, the score of the first surface under the camera viewing angle of the third viewing angle image is obtained.

[0033] In a possible implementation manner, the device further includes:

[0034] A texture unfolding module, used for unfolding each facet on the surface of the object model into a first UV space, and obtaining the coordinates of the three-dimensional space points on each facet in the first UV space;

[0035] A mapping module, configured to map the coordinates of the three-dimensional space points on each face patch in the first UV space to a second UV space based on the coordinates of the vertices of each face patch in the first UV space, so as to obtain the coordinates of the three-dimensional space points on each face patch in the second UV space;

[0036] The texture atlas generation module is used to generate a texture atlas of each patch on the surface of the object model unfolded into the first UV space based on the coordinates of the three-dimensional space points on each patch in the second UV space.

[0037] In a possible implementation, the mapping module is used to:

[0038] For a first spatial point on a second patch, based on coordinates of the first spatial point in the first UV space and coordinates of vertices of the second patch in the first UV space, obtaining weights of vertices of the second patch to the first spatial point; the second patch is any one of the patches; and the first spatial point is any three-dimensional spatial point on the second patch;

[0039] Based on the weights of the vertices of the second patch to the first spatial point and the coordinates of the vertices of the second patch in the first UV space, the coordinates of the first spatial point in the second UV space are obtained.

[0040] In a possible implementation, the mapping module is used to:

[0041] For a first vertex of the second face patch, obtaining a first area enclosed by the first spatial point and two other vertices on the second face patch except the first vertex in the first UV space; the first vertex is any one of the three vertices of the second face patch;

[0042] Obtain a second area enclosed by three vertices of the second face piece in the first UV space;

[0043] Based on the first area and the second area, a weight of the first vertex to the first spatial point is obtained.

[0044] In one possible implementation, the texture expansion module is used to obtain the color value of the second spatial point in the first UV space by bilinear interpolation based on the color values ​​of surrounding pixels of the pixel of the map corresponding to the coordinates of the second spatial point on each facet in the second UV space, in response to the pixel coordinates of the map corresponding to the coordinates of the second spatial point in the second UV space being non-integer.

[0045] In a possible implementation manner, the device further includes:

[0046] a patch determination module, configured to determine, for a second pixel in the first UV space, a third patch among the patches that is closest to the second pixel in the first UV space in response to a center of the second pixel being outside the patches and a partial area of ​​the second pixel overlapping at least one patch among the patches;

[0047] A color setting module is used to set the color value of the second pixel based on the pixel color of the point on the third surface closest to the center of the second pixel.

[0048] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the above-mentioned texture processing method of the three-dimensional model.

[0049] On the other hand, a computer-readable storage medium is provided, wherein at least one computer instruction is stored in the computer-readable storage medium, and the computer instruction is loaded and executed by a processor to implement the above-mentioned texture processing method of the three-dimensional model.

[0050] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the texture processing method of the three-dimensional model provided in the above various optional implementations.

[0051] The technical solution provided by this application may have the following beneficial effects:

[0052] First, for the n perspective images of the target object under n camera perspectives, the pixel values ​​of the common viewing areas between the n perspective images are mixed to obtain mixed images of the n perspective images. Subsequently, when constructing the texture of the object model, the texture of each facet of the object model is constructed by mixing the images. For each camera perspective, the color value of a facet in the object model in the mixed image under the camera perspective is a color value obtained by mixing the color value of the position corresponding to the facet in the target object when photographed under the camera perspective and the color value of the position of the facet when photographed under other camera perspectives. This can improve the color consistency under multiple perspectives, thereby avoiding the situation where the textures of adjacent faces are not aligned, thereby improving the accuracy of three-dimensional texture reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 is a schematic diagram of a system involved in an exemplary embodiment of the present application;

[0055] Figure 2 is a flow chart of a texture processing method of a three-dimensional model shown in an exemplary embodiment of the present application;

[0056] Figure 3 is a flow chart of a texture processing method of a three-dimensional model shown in an exemplary embodiment of the present application;

[0057] Figure 4 It is a schematic diagram of algorithm-related parameters involved in this application;

[0058] Figure 5 is a flow chart of a texture processing method of a three-dimensional model shown in an exemplary embodiment of the present application;

[0059] Figure 6 is a schematic diagram of texture coordinate mapping involved in this application;

[0060] Figure 7 It is a schematic diagram of the Yanbian expansion involved in this application;

[0061] Figure 8 This is the overall flow chart of the 3D / 4D human body high-quality texture reconstruction solution involved in this application;

[0062] Fig. 9 It is a rendering of the human body reconstruction model involved in this application;

[0063] Fig.10 It is a framework diagram of the human body texture reconstruction scheme involved in this application;

[0064] Fig.11 is a comparison chart of texture reconstruction results involved in this application;

[0065] Fig.12 It is a comparison chart of the texture atlas results involved in this application;

[0066] Fig.13 is a block diagram of a texture processing device for a three-dimensional model provided by an embodiment of the present application;

[0067] Fig.14 A structural block diagram of a computer device shown in an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0069] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0070] The embodiment of the present application provides a texture processing method for a three-dimensional model for three-dimensional texture reconstruction. For ease of understanding, several terms involved in the present application are explained below.

[0071] 1) Texture: The texture of an object's surface includes uneven grooves on the surface of the object, and also includes colored patterns on the smooth surface of the object.

[0072] 2) UV: UV is a two-dimensional texture coordinate point residing on the vertices of the polygonal mesh. They define a two-dimensional texture coordinate system called UV texture space, which uses the letters U and V to define the coordinate axes.

[0073] 3) Texture coordinates: Coordinates in texture space that define the mapping relationship from texture to the surface of a 3D model or any rendering primitive. Each vertex in the model is associated with a texture coordinate, indicating that the vertex needs to read the texture map data from this coordinate.

[0074] 4) Texture map: An image used to store and express the details of an object’s surface.

[0075] 5) Texture mapping: Accurately map each point on the 2D texture map to the surface of the 3D model object.

[0076] 6) Texture atlas: Merge multiple texture maps into one texture.

[0077] Figure 1 A schematic diagram of a system used in a texture processing method for a three-dimensional model provided by an exemplary embodiment of the present application is shown. Figure 1 As shown, the system includes: a server 110 and a terminal 120.

[0078] Among them, the above-mentioned server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0079] The terminal 120 may be a terminal device having network connection function and data processing function. For example, the terminal 120 may be a smart phone, a tablet computer, an e-book reader, smart glasses, a smart watch, a smart TV, a laptop computer, a desktop computer, etc.

[0080] Optionally, the above system includes one or more servers 110 and multiple terminals 120. The embodiment of the present application does not limit the number of servers 110 and terminals 120.

[0081] The terminal and the server are connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0082] The above-mentioned terminal and / or server may store / run application programs such as modeling tools or plug-ins for collecting two-dimensional images or reconstructing textures of models of three-dimensional objects through two-dimensional images.

[0083] For example, the server 110 may be provided with an application installation package, and the terminal 120 may download the application installation package from the server 110 and install the application; subsequently, the terminal 120 uses the two-dimensional image to reconstruct the texture of the three-dimensional object model through the application.

[0084] Figure 1Only one terminal is shown in the figure, but in different embodiments, there are multiple other terminals that can access the server 110. Optionally, there are one or more terminals corresponding to the developer, on which a development and editing platform for developing the above application is installed. The developer can edit and update the above application on the terminal, and transmit the updated application installation package to the server 110 through a wired or wireless network. The terminal 120 can download the application installation package from the server 110 to update the application.

[0085] For another example, the server 110 may be configured to install and run the application, and the terminal 120 may send a two-dimensional image to the server 110 . The server 110 may use the received two-dimensional image to reconstruct the texture of the three-dimensional object model through the application, and return the reconstruction result to the terminal 120 .

[0086] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology. This application is not limited here.

[0087] Figure 2 A flowchart of a texture processing method for a three-dimensional model shown in an exemplary embodiment of the present application is shown. The method is executed by a computer device. The computer device can be implemented as a terminal or a server. The terminal or the server can be Figure 1 The terminal or server shown, such as Figure 2As shown, the texture processing method of the three-dimensional model includes the following steps:

[0088] Step 210: Obtain n perspective images; the n perspective images are images of the target object under n camera perspectives respectively.

[0089] The camera viewing angle mentioned above may refer to information such as the spatial angle, position, field of view (FOV) and the like when the camera photographs the target object.

[0090] The n perspective images mentioned above may be images obtained by respectively photographing the target object at n camera perspectives using an image acquisition device.

[0091] In the above-mentioned shooting process, information such as the camera angle of view and camera parameters of the image acquisition device may be recorded.

[0092] The target object may include any three-dimensional object such as a person, an animal or an object.

[0093] Step 220: Based on the camera perspectives of the n perspective images, pixel values ​​of the common viewing areas between the n perspective images are mixed to obtain mixed images of the n perspective images.

[0094] The existence of a common viewing area between two viewing angle images refers to the existence of image areas corresponding to the same spatial position in the two viewing angle images.

[0095] In the embodiment of the present application, the above-mentioned mixing of pixel values ​​of the common viewing area between each of n perspective images to obtain mixed images of the n perspective images means that for each perspective image, the pixel values ​​in other perspective images corresponding to the same spatial position in the current perspective image are mixed into the current perspective image to obtain a mixed image of the current perspective image.

[0096] For example, for the target object, assuming that perspective image 1 and perspective image 2 respectively contain image areas of facet A in the object model of the target object, then in the above step 220, the computer device can mix the pixel values ​​of the image area corresponding to facet A in perspective image 1 to the corresponding pixel position in perspective image 2. For other perspective images other than perspective image 1 and perspective image 2, if other perspective images also have a common viewing area with perspective image 2, then the pixel values ​​of the common viewing area of ​​other perspective images and perspective image 2 are also mixed into perspective image 2, and finally a mixed image corresponding to perspective image 2 is obtained.

[0097] Correspondingly, the computer device also mixes the pixel values ​​of the image area corresponding to facet A in perspective image 2 into the corresponding pixel position in perspective image 1. For other perspective images other than perspective image 1 and perspective image 2, if other perspective images also have a common viewing area with perspective image 1, the pixel values ​​of the common viewing area between other perspective images and perspective image 1 are also mixed into perspective image 2, and finally a mixed image corresponding to perspective image 1 is obtained.

[0098] That is to say, for each perspective picture, the computer device mixes the pixel values ​​of the common viewing area with the perspective picture in other perspective pictures into the perspective picture to obtain a mixed picture of the perspective picture; therefore, the final n perspective pictures will each have a corresponding mixed picture.

[0099] Step 230: Based on the mixed images of the n perspective images and the camera perspectives of the n perspective images, the textures of each facet on the surface of the object model are obtained; the object model is a geometric model of the target object.

[0100] In an embodiment of the present application, when reconstructing the texture of an object model, a computer device uses the color values ​​corresponding to each facet in a mixed image to reconstruct the texture of each facet.

[0101] In a possible implementation, after obtaining the texture of each patch on the surface of the object model, the computer device may store the texture of each patch on the surface of the object model in the format of a UV map. For example, the computer device may select patches that are connected at the same viewing angle to form a block, and put the image corresponding to each block into the UV space to generate a texture atlas.

[0102] To summarize, the scheme shown in the embodiment of the present application first mixes the pixel values ​​of the common viewing areas between the n viewing images of the target object under n camera viewing angles to obtain mixed images of each of the n viewing images, and then constructs the texture of each facet of the object model through the mixed images when constructing the texture of the object model. For each camera viewing angle, the color value of a facet in the object model in the mixed image under the camera viewing angle is the color value of the position corresponding to the facet in the target object when photographed under the camera viewing angle, and the color value obtained by mixing the color value of the position of the facet when photographed under other camera viewing angles. This can improve the color consistency under multiple viewing angles, thereby avoiding the situation where the textures of adjacent faces are not aligned, thereby improving the accuracy of three-dimensional texture reconstruction.

[0103] Figure 3 A flowchart of a texture processing method for a three-dimensional model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 2In the illustrated embodiment, step 212 and step 214 may be included before step 220, and step 220 may be implemented as step 220a:

[0104] Step 212: Based on the n perspective images and the object model, obtain pixel depth information of each of the n perspective images; the pixel depth information is used to indicate the depth of each pixel in the perspective image.

[0105] In an embodiment of the present application, a computer device may use a ray collision algorithm to calculate the intersection of a ray emitted from the optical center of the camera and a geometric model of the target object corresponding to each pixel for each perspective, thereby obtaining effective facets and pixel depths (i.e., a depth map, corresponding to the above-mentioned pixel depth information) at each perspective.

[0106] The above-mentioned valid patch may refer to a patch on the surface of the geometric model that can be projected onto the current view image. In other words, the above-mentioned valid patch refers to a patch on the surface of the geometric model that can be observed when the geometric model is observed from the camera view corresponding to the current view image; for example, the above-mentioned valid patch refers to a patch on the geometric model that can be hit by a ray emitted from the optical center of the camera view corresponding to the current view image.

[0107] Step 214: based on the pixel depth information of each of the n perspective pictures, obtain the pixel weight information of each of the n perspective pictures; the pixel weight information is used to indicate the mixing weight of each pixel in the perspective picture.

[0108] The surface of an object model is usually composed of several triangular patches, and different patches have different directions. At the same time, for a fixed camera perspective, when collecting different positions in the object model through the camera perspective, the viewing direction is also different. Therefore, when collecting pixels at a position on the surface of the object model through different camera perspectives, the accuracy of the collected pixel values ​​will also be different. Generally speaking, the deeper the depth of the pixel point, the lower the accuracy of the collected pixel value. Conversely, the shallower the depth of the pixel point, the higher the accuracy of the collected pixel value.

[0109] In an embodiment of the present application, a computer device can set the weight of each pixel in each perspective image during mixing according to the depth of the pixel, so as to perform weighted processing when subsequently mixing the pixels, thereby improving the accuracy of the mixed pixel value and further improving the accuracy of texture reconstruction.

[0110] In a possible implementation, based on the pixel depth information of each of the n perspective pictures, obtaining the pixel weight information of each of the n perspective pictures includes:

[0111] For a first pixel in a first-perspective image, obtaining a first normal direction at a position point on the object model corresponding to the first pixel; the first-perspective image is any one of the n perspective images; and the first pixel is any one of the pixels in the first-perspective image;

[0112] Based on the depth of the first pixel and the angle between the first normal direction and the first viewing direction, a mixing weight of the first pixel is obtained; the first viewing direction is a direction from the camera center to the first pixel under the camera viewing angle of the first viewing image.

[0113] Among them, in addition to the influence of pixel depth on the accuracy of pixel values, generally speaking, the smaller the angle between the normal direction of the surface where the pixel point is located and the direction of the camera's viewing angle, the higher the accuracy of the collected pixel value. Conversely, the larger the angle between the normal direction of the surface where the pixel point is located and the direction of the camera's viewing angle, the lower the accuracy of the collected pixel value.

[0114] Therefore, in an embodiment of the present application, when calculating the mixing weight of each pixel in each perspective image, the computer device also considers the angle between the normal direction of the patch where the pixel is located and the direction of the camera perspective, and combines the above angle and pixel depth to set the mixing weight for the pixel, so that weighted processing can be performed when the pixels are subsequently mixed, thereby improving the accuracy of the mixed pixel values, and then improving the accuracy of texture reconstruction.

[0115] For example, the process of calculating the blending weight based on the depth information can be as follows:

[0116]

[0117] in, Figure 4 is a schematic diagram of algorithm-related parameters, where w i,p is the weight of pixel p in the image at view i, d i,p is the camera direction of view i, is the normal direction of the intersection point corresponding to pixel p at viewing angle i, cos<a,b> Represents the cosine of the angle between two vectors, D i,p is the depth value of pixel p at viewing angle i.

[0118] The intersection point corresponding to the pixel p under the viewing angle i refers to the point where a ray from the optical center of the camera through the pixel point p on the imaging plane intersects the surface of the geometric model under the viewing angle i of the camera.

[0119] Step 220a: Based on the camera perspectives of the n perspective images and the pixel weight information of the n perspective images, weightedly mix the pixel values ​​of the common viewing areas between the n perspective images to obtain mixed images of the n perspective images.

[0120] In a possible implementation, based on the camera perspectives of the n perspective pictures and the pixel weight information of the n perspective pictures, pixel values ​​of the common viewing areas between the n perspective pictures are weighted mixed to obtain mixed pictures of the n perspective pictures, including:

[0121] For the second viewing angle picture, obtain each common viewing area picture of the second viewing angle picture; the common viewing area picture is a viewing angle picture among the n viewing angle pictures, which has a common viewing area with the second viewing angle picture; the second viewing angle picture is any viewing angle picture among the n viewing angle pictures;

[0122] Mapping pixels of the common viewing area between each common viewing area picture and the second viewing angle picture to a pixel coordinate system of the second viewing angle picture to obtain mapping coordinates of pixels of the common viewing area between each common viewing area picture and the second viewing angle picture;

[0123] Based on the pixel coordinates of each pixel in the second perspective picture, the mapping coordinates of the pixels of the common view area between each common view area picture and the second perspective picture, and the mixing weights of the pixels of the common view area between each common view area picture and the second perspective picture, the pixels of the common view area between each common view area picture and the second perspective picture are weighted mixed to obtain a mixed picture of the second perspective picture.

[0124] In an embodiment of the present application, when the pixel values ​​of the second perspective image are mixed with other perspective images with a common viewing area, the computer device can map the pixels in the other perspective images with a common viewing area to the pixel space corresponding to the second perspective image, thereby unifying the coordinates of the pixels in the perspective images of different camera perspectives, thereby ensuring the accuracy of pixel value mixing, and thereby improving the accuracy of texture reconstruction.

[0125] After obtaining the mixing weights, the computer device can calculate the remapping relationship for each two perspective images with overlapping visible areas between perspectives. For each image, the mapping of all images containing the common viewing area to the perspective is calculated, and all mapping results to the perspective (including the original perspective) are weighted mixed to obtain a mixed image. The formula is as follows:

[0126]

[0127] Where I′ i is the mixed image of the i-th perspective, w j is the weight map of the jth perspective, R(I j ,I i ) represents the mapping from image j to image i, S i is the set of all perspectives that have a common viewing area with perspective i.

[0128] The mapping R(I j ,I i ) means mapping each pixel in image i to the imaging plane of image j. For example, first convert the pixel p_i(x,y) in the coordinate system of image i to the world coordinate system p_w and then convert it to the pixel coordinate system of image j to obtain p_j.

[0129] In a possible implementation, the texture of each facet of the surface of the object model is obtained based on the mixed image of each of the n perspective images and the camera perspective of each of the n perspective images, including:

[0130] Based on the normal direction of each facet on the surface of the object model and the camera perspective of each of the n perspective images, obtain the score of each facet under the camera perspective of each of the n perspective images;

[0131] Determine the target camera perspective of each facet based on the score of each facet under the camera perspective of each of the n perspective images;

[0132] The texture of each patch is obtained based on the pixel value of each patch in the mixed image corresponding to the respective target camera perspective.

[0133] In the embodiment of the present application, since each camera perspective corresponds to a mixed image, and for a specified patch, the accuracy of the pixel value corresponding to the patch in the mixed images of different camera perspectives is also different. Generally speaking, the smaller the angle between the normal direction of the patch and the direction of the camera perspective, the higher the accuracy of the pixel value corresponding to the patch in the mixed image corresponding to the camera perspective. Conversely, the larger the angle between the normal direction of the patch and the direction of the camera perspective, the lower the accuracy of the pixel value corresponding to the patch in the mixed image corresponding to the camera perspective. In this regard, when generating the texture of the patch, the computer device can score each camera perspective according to the angle between the normal direction of the patch and the direction of the camera perspective, and select the pixel value in the mixed image corresponding to the target camera perspective according to the scoring order as the texture of the patch, thereby improving the accuracy of texture reconstruction.

[0134] In a possible implementation, based on the normal direction of each facet on the surface of the object model and the camera perspective of each of the n perspective images, the score of each facet under the camera perspective of each of the n perspective images is obtained, including:

[0135] For the projection area of ​​the first face piece in the third viewing angle picture, obtain the normal direction of the position point on the object model corresponding to each pixel in the projection area; the third viewing angle picture is any one viewing angle picture among the n viewing angle pictures;

[0136] Obtain the viewing direction of each pixel in the projection area; the viewing direction of each pixel in the projection area is the direction from the camera center to each pixel in the projection area under the camera viewing angle of the third-view image;

[0137] Based on the normal direction at the position point on the object model corresponding to each pixel in the projection area, and the viewing direction of each pixel in the projection area, the score of the first surface under the camera viewing angle of the third viewing angle image is obtained.

[0138] Since a patch may correspond to multiple pixels in a perspective image, the computer device calculates the score of the entire patch for a certain camera perspective based on the normal direction and perspective direction corresponding to each pixel in the perspective image, thereby improving the accuracy of the score and, in turn, the accuracy of texture reconstruction.

[0139] For example, after obtaining the mixed image, the computer device can calculate the score of each face of the model at each viewing angle, and select the color of the mixed image at the viewing angle with the highest score as the texture of the face. The formula is as follows:

[0140]

[0141] Among them, s k,i is the score of the kth face under the i-th viewing angle, A k,i is the projection area of ​​the kth face on the image at the i-th perspective.

[0142] Based on the above Figure 2 or Figure 3 The embodiment shown, Figure 5 A flowchart of a texture processing method for a three-dimensional model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 2 In the illustrated embodiment, step 230 may further include steps 240 to 260:

[0143] Step 240: Expand each facet on the surface of the object model into the first UV space, and obtain the coordinates of the three-dimensional space points on each facet in the first UV space.

[0144] The solution shown in the embodiment of the present application can first use a UV unfolding algorithm (such as the UVAtlas algorithm) to re-divide the UV. For example, the UV unfolding algorithm can minimize the stretching in the UV space by minimizing the geodesic distance distortion between each pair of vertices in the mesh. It can map any mesh to a two-dimensional UV map and obtain a better divided area in the UV space (that is, the first UV space mentioned above, which can also be called the target UV space). That is to say, each block in the divided area occupies a larger area, each triangle face has less distortion, fewer fragmented areas, and a higher space occupancy rate (if a triangle face has a larger stretch in the divided area in the UV space, then its texture will be over-compressed in a certain dimension, resulting in poor texture mapping quality).

[0145] Step 250: Based on the coordinates of the vertices of each face patch in the first UV space, map the coordinates of the three-dimensional space points on each face patch in the first UV space to the second UV space to obtain the coordinates of the three-dimensional space points on each face patch in the second UV space.

[0146] The second UV space may be referred to as an original UV space, that is, a UV space in which triangular faces in a model are mapped according to a selected perspective through a perspective selection operation.

[0147] Through the perspective selection operation, the texture atlas generated by mapping the triangular faces in the model according to the selected perspective is often prone to fragmentation, overlap, and low space occupancy. For example, in the embodiment of the present application, the computer device first generates a texture map (UV map) in the original UV space; specifically, for each mesh in the object model, the perspective corresponding to the highest score can be obtained by calculating the scores under all perspectives, and the image under this perspective is selected, and the mesh vertices are projected from the world coordinates to the pixel coordinates of the perspective to obtain the x, y coordinates of the vertices (if the vertex coordinates are outside the image, the next perspective is selected)

[0148] Then vt (UV texture coordinate) is:

[0149] Vt[0]=(x+0.5) / Img.W

[0150] Vt[1]=1-(y+0.5) / Img.H

[0151] Since the image has multiple perspectives, the UV map obtained by this method is equivalent to the superposition of the UVmap corresponding to each perspective. If it is represented by a texture atlas, since the overlap of the triangular faces in the UV space is not taken into account, serious aliasing will occur; when it is desired to use a texture atlas for texture representation, considering the above-mentioned overlap problem, all the triangular faces can be arranged regularly in the UV space according to the size of the area. However, even if the above scheme takes the overlap into account, it still does not take into account the distortion problem caused by the triangular faces during the projection process, such as the stretching phenomenon of the triangular faces. For example, a triangular face on the geometric model undergoes the perspective selection process, that is, the projection of the triangular face at a certain perspective is selected and mapped to the UV space to obtain a triangle in the UV space. However, since the selected perspective may not be directly facing the triangular face in the model, the triangle in the UV space obtained by projection will produce certain distortion compared with the shape of the triangular face in the model. For example, the texture information in a certain direction (such as horizontal) will be compressed, resulting in a decrease in the quality of the generated texture. In addition, more fragments are also prone to seam problems, reducing the texture quality.

[0152] The scheme shown in the embodiment of the present application, after mapping the triangular faces in the model according to the selected perspective to obtain the texture atlas in the second UV space through the perspective selection operation, further expands the object model to the first UV space. In this step, the process of expanding the object model to the first UV space obtains the layout of the object model expanded to the first UV space. At this time, the layout of the object model expanded to the first UV space still lacks color texture. In order to obtain the texture atlas of the first UV space, it is necessary to assign corresponding colors to the pixel coordinates of the object model expanded to the first UV space, and this color needs to be obtained from the texture atlas of the second UV space; therefore, for any three-dimensional spatial point on the surface of the object model, the computer device can map the coordinates of the three-dimensional spatial point in the first UV space (target UV space) to the coordinates in the second UV space (original UV space), that is, determine the coordinates of the three-dimensional spatial point corresponding to the second UV space, so as to perform subsequent color mapping.

[0153] In a possible implementation, based on the coordinates of the vertices of each patch in the first UV space, the coordinates of the three-dimensional space points on each patch in the first UV space are mapped to the second UV space to obtain the coordinates of the three-dimensional space points on each patch in the second UV space, including:

[0154] For a first spatial point on a second patch, based on the coordinates of the first spatial point in the first UV space and the coordinates of the vertices of the second patch in the first UV space, obtain the weights of the vertices of the second patch to the first spatial point; the second patch is any one of the patches; the first spatial point is any three-dimensional spatial point on the second patch;

[0155] Based on the weights of the vertices of the second patch to the first spatial point and the coordinates of the vertices of the second patch in the first UV space, the coordinates of the first spatial point in the second UV space are obtained.

[0156] In an embodiment of the present application, for a first spatial point, a computer device may first determine the coordinates of the vertices of the patch where the first spatial point is located in the first UV space, determine the weights of the vertices of the patch where the first spatial point is located for the first spatial point, and then determine the coordinates of the first spatial point in the second UV space by weighted processing, thereby ensuring the accuracy of mapping the coordinates of the first spatial point in the first UV space to the second UV space as much as possible, thereby ensuring the accuracy of the color of the new texture atlas obtained subsequently.

[0157] In a possible implementation, for a first spatial point on a second patch, based on coordinates of the first spatial point in a first UV space and coordinates of vertices of the second patch in the first UV space, obtaining weights of vertices of the second patch to the first spatial point includes:

[0158] For the first vertex of the second face piece, obtain a first area enclosed by the first spatial point and two other vertices on the second face piece except the first vertex in the first UV space; the first vertex is any one of the three vertices of the second face piece;

[0159] Get the second area enclosed by the three vertices of the second face in the first UV space;

[0160] Based on the first area and the second area, a weight of the first vertex to the first spatial point is obtained.

[0161] In an embodiment of the present application, in the first UV space, the computer device determines the weight of another vertex on the patch to the first spatial point by using the area enclosed by the first spatial point and the two vertices on the patch where the first spatial point is located and the area enclosed by the three vertices of the second patch, thereby determining the weight by the proportional relationship of the areas, thereby ensuring as much as possible the accuracy of mapping the coordinates of the first spatial point in the first UV space to the second UV space, thereby ensuring the accuracy of the color of the new texture atlas subsequently obtained.

[0162] Since the texture mapping relationship has changed, a new texture atlas needs to be generated by remapping. We use the re-divided UV space as the target UV, map the texture coordinates under the target UV back to the original UV space, and calculate the corresponding texture coordinates on the original UV by weighting the vertex coordinates of the triangle where the target UV is located. The texture coordinate mapping diagram is shown below: Figure 6 shown.

[0163] The above mapping relationship calculation process can refer to the following formula:

[0164]

[0165]

[0166] Among them, UV s (p) is the coordinate of the three-dimensional point P in the original UV, m t,i Indicates the i-th triangle of the target UV, UV t (p) represents the coordinates of point P in the target UV, UV t (v i,k ) represents the kth vertex of the i-th triangle in the target UV, Represents the weight of the vertex coordinates in the target UV. A(a,b,c) represents the area enclosed by the three points a,b,c.

[0167] Step 260: Generate a texture atlas of each patch on the surface of the object model unfolded into the first UV space based on the coordinates of the three-dimensional space points on each patch in the second UV space.

[0168] In an embodiment of the present application, a computer device determines the coordinates of a three-dimensional space point on each facet in the original UV space, and when the coordinates of the three-dimensional space point on each facet in the target UV space are known, the color value of the coordinates of the three-dimensional space point in the target UV space can be set to the color value of the three-dimensional space point in the texture atlas in the target UV space, thereby achieving color assignment of the facets expanded to the target UV space after the object model is expanded to the target UV space, thereby obtaining the texture atlas of the object model in the target UV space. Since the newly obtained texture atlas is a texture atlas assigned color values ​​after being expanded by a UV expansion algorithm, it has the advantages of less distortion of each triangle, less fragmented area, and higher space occupancy.

[0169] In a possible implementation, generating a texture atlas of each patch on the surface of the object model unfolded into the first UV space based on the coordinates of the three-dimensional space points on each patch in the second UV space includes:

[0170] For the coordinates of the second spatial point in the second UV space on each facet, in response to the pixel coordinates of the map corresponding to the coordinates of the second spatial point in the second UV space being non-integer, the color value of the second spatial point in the first UV space is obtained by bilinear interpolation based on the color values ​​of the surrounding pixels of the pixel of the map corresponding to the coordinates of the second spatial point in the second UV space.

[0171] For the case where the calculated pixel coordinates of the texture map may not be integers, the scheme shown in the embodiment of the present application can use bilinear interpolation to calculate the corresponding pixel value through the colors of the surrounding pixel blocks, thereby avoiding the situation where the textures are not aligned and improving the accuracy of texture reconstruction.

[0172] The texture coordinate vt generally mentioned is the result of normalization, and its value is between 0-1. In the embodiment of the present application, the pixel coordinates of the above-mentioned texture map are the results before normalization, that is, the pixel coordinates of the texture map = texture coordinates * the length or width of the texture map.

[0173] For a certain spatial point P, its pixel coordinates in the target UV map (that is, the pixel coordinates in the first UV space) can be obtained through a remapping algorithm (such as the UVAtlas algorithm). Then, its pixel value in the target UV map can be assigned by the pixel value corresponding to its pixel coordinates in the original UV map (that is, the pixel coordinates in the second UV space). When only the pixel values ​​of the integer coordinates in the original UV map (whose coordinates correspond to the pixel coordinates of the image) are known, for pixel values ​​that are not in integer coordinates, they can be obtained by bilinear interpolation through the pixel values ​​of the four points around it.

[0174] In one possible implementation, for a second pixel in a first UV space, in response to the center of the second pixel being outside each patch and a partial area of ​​the second pixel overlapping with at least one of the patches, a third patch among the patches that is closest to the second pixel in the first UV space is determined; and the color value of the second pixel is set based on the pixel color of the point on the third patch that is closest to the center of the second pixel.

[0175] For example, the computer device may set the color of the pixel at the point on the third patch closest to the center of the second pixel to the color value of the second pixel.

[0176] Figure 7 Schematic diagram of Yanbian expansion involved in this application is shown. Figure 7 As shown, if the center of a pixel block 71 in the target UV is not in any triangular face, and part of the pixel block is in one or more triangular faces ( Figure 7 As shown in FIG. 1 , the pixel block is partially in the triangle face 72 and partially in the triangle face 73. If the expansion is not performed, the color of the triangle face will be lost, and finally a seam phenomenon will appear on the model texture. Figure 7 As shown, the solution shown in the embodiment of the present application adopts an edge expansion method, which propagates color by selecting the nearest point of all triangles to the center point of the pixel (the point on the edge of the triangle 73), so that the color transition between the patches is natural.

[0177] The scenarios in which the solutions proposed in the above embodiments of the present application can be applied include but are not limited to 3D / 4D human body high-quality texture reconstruction. Taking the human body high-quality texture reconstruction scenario as an example, please refer to Figure 8 , which shows the overall flow chart of the 3D / 4D human body high-quality texture reconstruction solution involved in this application. Figure 8 As shown, the process may include the following steps:

[0178] S81, input multi-view RGB image;

[0179] S82, input human body geometry model;

[0180] S83, texture mapping calculation;

[0181] S84, texture atlas generation;

[0182] S85, output the textured human body model.

[0183] This solution can be applied to high-quality 3D or 4D human body model reconstruction. Through high-quality texture reconstruction technology, photo-realistic textures can be restored from multiple views for the reconstructed human body geometric model. The human body model with realistic texture generated by this solution can be applied to virtual reality, film and television special effects, model development of virtual characters in games, model generation of player characters in games, etc., realizing the digitization of the human body, which has very considerable application value.

[0184] For example, in a game scene based on virtual reality, the game device can pre-collect multi-perspective RGB images of the player, and then build a textured human body model based on the multi-perspective RGB images and the player's human body geometric model through the scheme shown in the above embodiment of the present application, and the texture in the human body model matches the player's clothing and accessories; during the game, the player can control the human body model to act in the virtual reality scene, thereby improving the simulation effect of the virtual reality scene.

[0185] For another example, in a game development scenario, developers need to develop a model of a virtual character with specified clothing. They can pre-collect multi-perspective RGB images of a real-life model wearing the specified clothing, and then build a textured human body model based on the multi-perspective RGB images and the human body geometry model of the virtual character through the scheme shown in the above embodiment of the present application. At this time, the texture of the obtained human body model matches the specified clothing on the real-life model, thereby obtaining a model of the virtual character with the specified clothing.

[0186] The human body reconstruction model rendering generated by the above solution of the present application can be as follows Fig. 9 shown.

[0187] Please refer to Fig.10, which shows a framework diagram of the human body texture reconstruction solution involved in this application. Fig.10 As shown, the above scheme is mainly divided into two parts:

[0188] S1, texture mapping based on normal weighted fusion, including weight calculation, image fusion and perspective selection steps.

[0189] The input of this part is a multi-view RGB image 1001 and a human body geometric model 1002. The weights of each view are obtained by calculating the depth information and normal information of the geometric model under each camera view, and the images of each view are weighted fused through the mapping relationship between the images. Finally, the color of the corresponding view is selected for each face through view selection. The algorithm uses the information contained in the multi-view image to mix and obtain a multi-view consistent texture to avoid the phenomenon of texture misalignment.

[0190] S2, high-quality texture atlas generation, including UV re-division, remapping, and edge expansion steps.

[0191] This part re-divides the original texture atlas, optimizes the integrity of the image blocks and the occupancy rate in the UV space, converts the texture mapping mapped to the original texture atlas to the mapping to the new texture atlas through remapping, and uses edge expansion to handle possible seam problems to ensure the integrity of the texture information after remapping.

[0192] Finally, by combining the above two steps, a human body model 1003 with texture can be obtained.

[0193] Please refer to Fig.11 , which shows a comparison diagram of texture reconstruction results involved in this application. Fig.11 The left side shows the texture reconstruction result obtained using the traditional MVS-texturing algorithm. Fig.11 As shown, there are seam problems and misaligned textures in regions 1101, 1102, and 1103. Fig.11 The right side shows the reconstruction result of this solution. It can be seen that the result of this solution has no obvious texture misalignment, and there is no obvious seam between the patches, which has a good reconstruction result.

[0194] Please refer to Fig.12 , which shows a comparison chart of the texture atlas results involved in this application. Fig.12 The left side shows the texture atlas result obtained using the traditional MVS-texturing algorithm. Fig.12 The right side shows the texture atlas result of this solution. It can be seen that the image blocks account for a large proportion in the result of this solution, and there are fewer fragmented image blocks, which is more conducive to the texture mapping to present a higher quality effect.

[0195] Fig.13 A block diagram of a texture processing device for a three-dimensional model according to an exemplary embodiment of the present application is shown. The device can be used to perform the following steps: Figure 2 , Figure 3 or Figure 5 All or part of the steps in the method shown; Fig.13 As shown, the device comprises:

[0196] The picture acquisition module 1301 is used to acquire n perspective pictures; the n perspective pictures are pictures of the target object under n camera perspectives respectively;

[0197] A pixel mixing module 1302 is used to mix pixel values ​​of common viewing areas between two of the n viewing angle pictures based on the camera viewing angles of the n viewing angle pictures to obtain mixed pictures of the n viewing angle pictures;

[0198] The texture acquisition module 1303 is used to acquire the texture of each facet on the surface of the object model based on the mixed image of each of the n perspective images and the camera perspective of each of the n perspective images; the object model is a geometric model of the target object.

[0199] In a possible implementation manner, the device further includes:

[0200] a depth acquisition module, configured to obtain pixel depth information of each of the n perspective pictures based on the n perspective pictures and the object model before the pixel blending module 1302 blends pixel values ​​of a common view area between the n perspective pictures based on the respective camera perspectives of the n perspective pictures to obtain respective blended pictures of the n perspective pictures; the pixel depth information is used to indicate the depth of each pixel in the perspective picture;

[0201] A weight acquisition module, used to acquire pixel weight information of each of the n perspective pictures based on the pixel depth information of each of the n perspective pictures; the pixel weight information is used to indicate the mixing weight of each pixel in the perspective picture;

[0202] The pixel mixing module 1302 is used to weightedly mix the pixel values ​​of the common viewing area between the n perspective images based on the camera perspectives of each of the n perspective images and the pixel weight information of each of the n perspective images to obtain a mixed image of each of the n perspective images.

[0203] In a possible implementation, the weight acquisition module is used to:

[0204] For a first pixel in a first viewing angle picture, obtaining a first normal direction at a position point on the object model corresponding to the first pixel; the first viewing angle picture is any one viewing angle picture among the n viewing angle pictures; the first pixel is any one pixel in the first viewing angle picture;

[0205] Based on the depth of the first pixel and the angle between the first normal direction and a first viewing direction, a mixing weight of the first pixel is obtained; the first viewing direction is a direction from a camera center to the first pixel under the camera viewing angle of the first viewing image.

[0206] In a possible implementation, the pixel blending module 1302 is used to:

[0207] For the second viewing angle picture, obtain each common viewing area picture of the second viewing angle picture; the common viewing area picture is a viewing angle picture among the n viewing angle pictures, which has a common viewing area with the second viewing angle picture; the second viewing angle picture is any viewing angle picture among the n viewing angle pictures;

[0208] Mapping pixels of the common viewing area between each of the common viewing area pictures and the second viewing angle picture into a pixel coordinate system of the second viewing angle picture to obtain mapping coordinates of pixels of the common viewing area between each of the common viewing area pictures and the second viewing angle picture;

[0209] Based on the pixel coordinates of each pixel in the second perspective picture, the mapping coordinates of the pixels of the common view area between each of the common view area pictures and the second perspective picture, and the mixing weights of the pixels of the common view area between each of the common view area pictures and the second perspective picture, the pixels of the common view area between each of the common view area pictures and the second perspective picture are weighted mixed to obtain a mixed picture of the second perspective picture.

[0210] In a possible implementation, the pixel blending module 1302 is used to:

[0211] Based on the normal direction of each facet on the surface of the object model and the camera perspective of each of the n perspective images, obtaining a score of each facet under the camera perspective of each of the n perspective images;

[0212] Determine a target camera perspective of each facet based on the score of each facet under the camera perspective of each of the n perspective images;

[0213] The texture of each facet is obtained based on the pixel value of each facet in the mixed picture corresponding to the respective target camera perspective.

[0214] In a possible implementation, the pixel blending module 1302 is used to:

[0215] For a projection area of ​​the first face piece in a third viewing angle picture, obtaining a normal direction at a position point on the object model corresponding to each pixel in the projection area; the third viewing angle picture is any one viewing angle picture among the n viewing angle pictures;

[0216] Obtaining the viewing direction of each pixel in the projection area; the viewing direction of each pixel in the projection area is the direction from the camera center to each pixel in the projection area under the camera viewing angle of the third viewing angle image;

[0217] Based on the normal direction of the position point on the object model corresponding to each pixel in the projection area, and the viewing direction of each pixel in the projection area, the score of the first surface under the camera viewing angle of the third viewing angle image is obtained.

[0218] In a possible implementation manner, the device further includes:

[0219] A texture unfolding module, used for unfolding each facet on the surface of the object model into a first UV space, and obtaining the coordinates of the three-dimensional space points on each facet in the first UV space;

[0220] A mapping module, configured to map the coordinates of the three-dimensional space points on each face patch in the first UV space to a second UV space based on the coordinates of the vertices of each face patch in the first UV space, so as to obtain the coordinates of the three-dimensional space points on each face patch in the second UV space;

[0221] The texture atlas generation module is used to generate a texture atlas of each patch on the surface of the object model unfolded into the first UV space based on the coordinates of the three-dimensional space points on each patch in the second UV space.

[0222] In a possible implementation, the mapping module is used to:

[0223] For a first spatial point on a second patch, based on coordinates of the first spatial point in the first UV space and coordinates of vertices of the second patch in the first UV space, obtaining weights of vertices of the second patch to the first spatial point; the second patch is any one of the patches; and the first spatial point is any three-dimensional spatial point on the second patch;

[0224] Based on the weights of the vertices of the second patch to the first spatial point and the coordinates of the vertices of the second patch in the first UV space, the coordinates of the first spatial point in the second UV space are obtained.

[0225] In a possible implementation, the mapping module is used to:

[0226] For a first vertex of the second face patch, obtaining a first area enclosed by the first spatial point and two other vertices on the second face patch except the first vertex in the first UV space; the first vertex is any one of the three vertices of the second face patch;

[0227] Obtain a second area enclosed by three vertices of the second face piece in the first UV space;

[0228] Based on the first area and the second area, a weight of the first vertex to the first spatial point is obtained.

[0229] In one possible implementation, the texture expansion module is used to obtain the color value of the second spatial point in the first UV space by bilinear interpolation based on the color values ​​of surrounding pixels of the pixel of the map corresponding to the coordinates of the second spatial point on each facet in the second UV space, in response to the pixel coordinates of the map corresponding to the coordinates of the second spatial point in the second UV space being non-integer.

[0230] In a possible implementation manner, the device further includes:

[0231] a patch determination module, configured to determine, for a second pixel in the first UV space, a third patch among the patches that is closest to the second pixel in the first UV space in response to a center of the second pixel being outside the patches and a partial area of ​​the second pixel overlapping at least one patch among the patches;

[0232] A color setting module is used to set the color value of the second pixel based on the pixel color of the point on the third surface closest to the center of the second pixel.

[0233] To summarize, the scheme shown in the embodiment of the present application first mixes the pixel values ​​of the common viewing areas between the n viewing images of the target object under n camera viewing angles to obtain mixed images of each of the n viewing images, and then constructs the texture of each facet of the object model through the mixed images when constructing the texture of the object model. For each camera viewing angle, the color value of a facet in the object model in the mixed image under the camera viewing angle is the color value of the position corresponding to the facet in the target object when photographed under the camera viewing angle, and the color value obtained by mixing the color value of the position of the facet when photographed under other camera viewing angles. This can improve the color consistency under multiple viewing angles, thereby avoiding the situation where the textures of adjacent faces are not aligned, thereby improving the accuracy of three-dimensional texture reconstruction.

[0234] Fig.14 The structural block diagram of a computer device 1400 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as a server in the above-mentioned solution of the present application. The computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory (RAM) 1402 and a read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the central processing unit 1401. The computer device 1400 also includes a mass storage device 1406 for storing an operating system 1409, an application program 1410 and other program modules 1411.

[0235] The mass storage device 1406 is connected to the central processing unit 1401 through a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1406 and its associated computer readable medium provide non-volatile storage for the computer device 1400. That is, the mass storage device 1406 may include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0236] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1404 and mass storage device 1406 can be collectively referred to as memory.

[0237] According to various embodiments of the present disclosure, the computer device 1400 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1400 can be connected to the network 1408 through the network interface unit 1407 connected to the system bus 1405, or the network interface unit 1407 can be used to connect to other types of networks or remote computer systems (not shown).

[0238] The memory also includes at least one computer instruction, which is stored in the memory. The central processing unit 1401 implements all or part of the steps in the methods shown in the above embodiments by executing the at least one computer instruction.

[0239] In an exemplary embodiment, a computer-readable storage medium is also provided, which is used to store at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor to implement all or part of the steps in the above-mentioned texture processing method of a three-dimensional model. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0240] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes all or part of the steps in the above-mentioned texture processing method for a three-dimensional model.

[0241] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0242] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A texture processing method for a three-dimensional model, characterized in that: The method comprises: Obtain n perspective images; the n perspective images are images of the target object under n camera perspectives respectively; Based on the camera perspectives of the n perspective pictures, pixel values ​​of common viewing areas between any two of the n perspective pictures are mixed to obtain mixed pictures of the n perspective pictures; Based on the mixed images of the n perspective images and the camera perspectives of the n perspective images, textures of each facet on the surface of an object model are acquired; the object model is a geometric model of the target object.

2. The method according to claim 1, characterized in that Before mixing pixel values ​​of common viewing areas between the n viewing angle pictures based on the respective camera viewing angles of the n viewing angle pictures to obtain respective mixed pictures of the n viewing angle pictures, the method further includes: Based on the n perspective pictures and the object model, obtaining pixel depth information of each of the n perspective pictures; the pixel depth information is used to indicate the depth of each pixel in the perspective picture; Based on the pixel depth information of each of the n perspective pictures, obtaining pixel weight information of each of the n perspective pictures; the pixel weight information is used to indicate the mixing weight of each pixel in the perspective picture; The mixing of pixel values ​​of common viewing areas between two of the n viewing angle pictures based on the respective camera viewing angles of the n viewing angle pictures to obtain mixed pictures of the n viewing angle pictures comprises: Based on the camera perspectives of the n perspective images and the pixel weight information of the n perspective images, pixel values ​​of the common viewing areas between the n perspective images are weighted mixed to obtain mixed images of the n perspective images.

3. The method according to claim 2, characterized in that The acquiring pixel weight information of each of the n perspective pictures based on the pixel depth information of each of the n perspective pictures comprises: For a first pixel in a first viewing angle picture, obtaining a first normal direction at a position point on the object model corresponding to the first pixel; the first viewing angle picture is any one viewing angle picture among the n viewing angle pictures; the first pixel is any one pixel in the first viewing angle picture; Based on the depth of the first pixel and the angle between the first normal direction and a first viewing direction, a mixing weight of the first pixel is obtained; the first viewing direction is a direction from a camera center to the first pixel under the camera viewing angle of the first viewing image.

4. The method according to claim 2, characterized in that: The step of weighting and mixing pixel values ​​of common viewing areas between two of the n viewing images based on the camera viewing angles of the n viewing images and the pixel weight information of the n viewing images to obtain mixed images of the n viewing images comprises: For the second viewing angle picture, obtain each common viewing area picture of the second viewing angle picture; the common viewing area picture is a viewing angle picture among the n viewing angle pictures, which has a common viewing area with the second viewing angle picture; the second viewing angle picture is any viewing angle picture among the n viewing angle pictures; Mapping pixels of the common viewing area between each of the common viewing area pictures and the second viewing angle picture into a pixel coordinate system of the second viewing angle picture to obtain mapping coordinates of pixels of the common viewing area between each of the common viewing area pictures and the second viewing angle picture; Based on the pixel coordinates of each pixel in the second perspective picture, the mapping coordinates of the pixels of the common view area between each of the common view area pictures and the second perspective picture, and the mixing weights of the pixels of the common view area between each of the common view area pictures and the second perspective picture, the pixels of the common view area between each of the common view area pictures and the second perspective picture are weighted mixed to obtain a mixed picture of the second perspective picture.

5. The method according to claim 1, characterized in that The acquiring the texture of each facet on the surface of the object model based on the mixed picture of each of the n perspective pictures and the camera perspective of each of the n perspective pictures comprises: Based on the normal direction of each facet on the surface of the object model and the camera perspective of each of the n perspective images, obtaining a score of each facet under the camera perspective of each of the n perspective images; Determine a target camera perspective of each facet based on the score of each facet under the camera perspective of each of the n perspective images; The texture of each facet is obtained based on the pixel value of each facet in the mixed picture corresponding to the respective target camera perspective.

6. The method according to claim 5, characterized in that The step of obtaining the scores of the respective facets under the respective camera perspectives of the n perspective pictures based on the normal directions of the respective facets on the surface of the object model and the respective camera perspectives of the n perspective pictures comprises: For a projection area of ​​the first face piece in a third viewing angle picture, obtaining a normal direction at a position point on the object model corresponding to each pixel in the projection area; the third viewing angle picture is any one viewing angle picture among the n viewing angle pictures; Obtaining the viewing direction of each pixel in the projection area; the viewing direction of each pixel in the projection area is the direction from the camera center to each pixel in the projection area under the camera viewing angle of the third viewing angle image; Based on the normal direction of the position point on the object model corresponding to each pixel in the projection area, and the viewing direction of each pixel in the projection area, the score of the first surface under the camera viewing angle of the third viewing angle image is obtained.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Expand each facet on the surface of the object model into a first UV space to obtain coordinates of a three-dimensional space point on each facet in the first UV space; Based on the coordinates of the vertices of each face patch in the first UV space, mapping the coordinates of the three-dimensional space points on each face patch in the first UV space to the second UV space, to obtain the coordinates of the three-dimensional space points on each face patch in the second UV space; Based on the coordinates of the three-dimensional space points on the respective patches in the second UV space, a texture atlas is generated in which the respective patches on the surface of the object model are unfolded into the first UV space.

8. The method according to claim 7, characterized in that The mapping the coordinates of the three-dimensional space points on each face patch in the first UV space to the second UV space based on the coordinates of the vertices of each face patch in the first UV space to obtain the coordinates of the three-dimensional space points on each face patch in the second UV space includes: For a first spatial point on a second patch, based on coordinates of the first spatial point in the first UV space and coordinates of vertices of the second patch in the first UV space, obtaining weights of vertices of the second patch to the first spatial point; the second patch is any one of the patches; and the first spatial point is any three-dimensional spatial point on the second patch; Based on the weights of the vertices of the second patch to the first spatial point and the coordinates of the vertices of the second patch in the first UV space, the coordinates of the first spatial point in the second UV space are obtained.

9. The method according to claim 8, characterized in that The step of obtaining, for a first spatial point on the second surface patch, weights of vertices of the second surface patch to the first spatial point based on coordinates of the first spatial point in the first UV space and coordinates of vertices of the second surface patch in the first UV space includes: For a first vertex of the second face patch, obtaining a first area enclosed by the first spatial point and two other vertices on the second face patch except the first vertex in the first UV space; the first vertex is any one of the three vertices of the second face patch; Obtain a second area enclosed by three vertices of the second face piece in the first UV space; Based on the first area and the second area, a weight of the first vertex to the first spatial point is obtained.

10. The method according to claim 7, characterized in that The step of generating a texture atlas of each patch on the surface of the object model unfolded into the first UV space based on the coordinates of the three-dimensional space points on each patch in the second UV space comprises: For the coordinates of the second spatial point on each facet in the second UV space, in response to the pixel coordinates of the map corresponding to the coordinates of the second spatial point in the second UV space being non-integer, based on the color values ​​of the surrounding pixels of the pixel of the map corresponding to the coordinates of the second spatial point in the second UV space, the color value of the second spatial point in the first UV space is obtained by bilinear interpolation.

11. The method according to claim 7, characterized in that For a second pixel in the first UV space, in response to a center of the second pixel being outside the patches and a partial area of ​​the second pixel overlapping with at least one patch among the patches, determining a third patch among the patches that is closest to the second pixel in the first UV space; The color value of the second pixel is set based on the color of the pixel at the point on the third patch closest to the center of the second pixel.

12. A texture processing device for a three-dimensional model, characterized in that: The device comprises: An image acquisition module is used to acquire n perspective images; the n perspective images are images of the target object under n camera perspectives respectively; A pixel mixing module, configured to mix pixel values ​​of common viewing areas between two of the n viewing angle pictures based on the camera viewing angles of the n viewing angle pictures, to obtain mixed pictures of the n viewing angle pictures; The texture acquisition module is used to acquire the texture of each facet on the surface of the object model based on the mixed image of each of the n perspective images and the camera perspective of each of the n perspective images; the object model is a geometric model of the target object.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the texture processing method of the three-dimensional model as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the computer instruction is loaded and executed by a processor to implement the texture processing method of a three-dimensional model as described in any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the texture processing method of a three-dimensional model as described in any one of claims 1 to 11.