Texture map generation method and device, electronic device and storage medium

By obtaining multi-angle color and depth maps, reconstructing the three-dimensional head shape and generating texture maps, combining identity and expression vectors for rendering, the problem of blurred texture maps in the existing technology is solved, and clearer and more realistic texture expression is achieved.

CN114549729BActive Publication Date: 2024-11-22BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210307589.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-25
Publication Date
2024-11-22
Estimated Expiration
2042-03-25

AI Technical Summary

Technical Problem

In the three-dimensional head reconstruction based on a single picture, the generated texture maps are blurred and cannot accurately express the user's three-dimensional image, resulting in unclear textures and inability to reflect the user's real situation.

Method used

By obtaining the color map and depth map of the target user at multiple angles, reconstructing the three-dimensional head shape of the target user, and using the shape and color map to generate texture maps, combining identity vectors and expression vectors for rendering and optimization, ensuring texture maps clarity and authenticity.

Benefits of technology

The generated texture map not only fits the head shape of the target user, has clear texture quality, can more accurately reflect the user's texture characteristics, and solves the problem of unclear texture.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549729B_ABST
    Figure CN114549729B_ABST
Patent Text Reader

Abstract

The present application provides a method and device for generating a texture map, an electronic device and a storage medium, wherein the method comprises: obtaining a color map and a depth map corresponding to a target user at multiple angles; reconstructing the three-dimensional head shape of the target user using the color map and the depth map to obtain a target three-dimensional head shape of the target user; and generating a texture map using the target three-dimensional head shape and the color map. The present application solves the problem in the related art that the texture of the map corresponding to the generated three-dimensional image of the user is not clear and cannot reflect the true texture condition of the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for generating a texture map, an electronic device, and a storage medium. Background Art

[0002] At present, in the task of three-dimensional head reconstruction based on a single image, a generative adversarial network is often used to generate corresponding texture maps according to the input image. Although this method can retain the personalized characteristics of the input face, the generated image is relatively blurry and fails to accurately express the user's three-dimensional image.

[0003] Therefore, there is a problem in the related art that the texture of the map corresponding to the generated three-dimensional image of the user is not clear and cannot reflect the real texture condition of the user. Summary of the invention

[0004] The present application provides a method and device for generating a texture map, an electronic device and a storage medium, so as to at least solve the problem in the related art that the generated map texture corresponding to the user's three-dimensional image is not clear and cannot reflect the user's true texture condition.

[0005] According to one aspect of an embodiment of the present application, a method for generating a texture map is provided, the method comprising:

[0006] Obtain the color map and depth map generated corresponding to the target user at multiple angles;

[0007] Reconstructing the three-dimensional head shape of the target user by using the color map and the depth map to obtain a target three-dimensional head shape of the target user;

[0008] A texture map is generated using the target three-dimensional head shape and the color map.

[0009] According to another aspect of an embodiment of the present application, a texture map generation device is also provided, the device comprising:

[0010] An acquisition unit, used to acquire a color map and a depth map corresponding to a target user generated at multiple angles;

[0011] A first obtaining unit is used to reconstruct the three-dimensional head shape of the target user by using the color map and the depth map to obtain a target three-dimensional head shape of the target user;

[0012] A generating unit is used to generate a texture map using the target three-dimensional head shape and the color map.

[0013] Optionally, the first obtaining unit includes:

[0014] A first obtaining module is used to obtain an initial three-dimensional head shape of the target user according to the color map and the depth map;

[0015] A rendering module, used to render the initial three-dimensional head shape at a target angle using an identity vector and an expression vector to obtain a rendered reference depth map at the target angle, wherein the identity vector and the expression vector are used to describe an identity dimension coefficient and an expression dimension coefficient of the target user;

[0016] The second obtaining module is used to obtain the target three-dimensional head shape by using the color map, the depth map and the reference depth map.

[0017] Optionally, the second obtaining module includes:

[0018] A determination subunit, configured to determine an error value using the color map, the depth map, and the reference depth map;

[0019] a comparing subunit, configured to compare the error value with a first threshold, wherein the first threshold is a critical value for stopping a cyclic comparison between the error value and the first threshold;

[0020] The obtaining subunit is used to obtain the target three-dimensional head shape when the error value is less than the first threshold.

[0021] Optionally, the determining subunit includes:

[0022] A detection submodule, used for detecting key points on the color map;

[0023] An acquisition submodule, used to acquire first key point coordinates of the key point on the color map, acquire second key point coordinates of the key point on the three-dimensional image corresponding to the target three-dimensional head shape, and acquire third key point coordinates on the two-dimensional image mapped by the second key point coordinates using a mapping function;

[0024] A determination submodule is used to determine the error value according to the depth error weight, the depth map, the reference depth map, the key point error weight, the first key point coordinates, the third key point coordinates, the regularization term weight, the identity vector and the expression vector, wherein the depth error weight is used to adjust the contribution value of the depth error function composed of the depth map and the reference depth map, the key point error weight is used to adjust the contribution value of the key point error function composed of the first key point coordinates and the third key point coordinates, and the regularization term weight is used to adjust the contribution value of the regularized error function composed of the identity vector and the expression vector.

[0025] Optionally, the generating unit includes:

[0026] An acquisition module, used for acquiring a two-dimensional image corresponding to the color map;

[0027] A mapping module, used to map the three-dimensional geometric shape points corresponding to the target three-dimensional head shape onto the two-dimensional image to obtain the point coordinates mapped onto the two-dimensional image;

[0028] The sampling module is used to perform image sampling on the two-dimensional image through the point coordinates to obtain the corresponding texture map.

[0029] Optionally, there are multiple texture maps, and the device further includes:

[0030] A fusion unit, configured to fuse a plurality of texture maps to obtain a first texture map after generating the texture map by using the target three-dimensional head shape and the color map;

[0031] The second obtaining unit is used to perform illumination balance and texture detail optimization processing on the first texture map to obtain a processed texture map.

[0032] Optionally, the second obtaining unit includes:

[0033] A first input module, used for inputting the first texture map into a first model to obtain a second texture map, wherein the first model is trained using facial texture map sample data with balanced illumination and texture map sample data with uneven illumination, and is used for generating a texture map with balanced illumination;

[0034] A second input module is used to input the second texture map into a second model to generate the processed texture map, wherein the second model is trained using sample data with texture map clarity lower than a second threshold and sample data with texture map clarity higher than the second threshold, and is used to optimize the texture of the map in detail, and the second threshold is used to indicate that the texture map is a minimum value of clarity.

[0035] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps in any of the above embodiments by running the computer program stored in the memory.

[0036] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the method steps in any of the above embodiments when executed.

[0037] In an embodiment of the present application, a color map and a depth map corresponding to the target user at multiple angles are obtained; the three-dimensional head shape of the target user is reconstructed using the color map and the depth map to obtain the target three-dimensional head shape of the target user; and a texture map is generated using the target three-dimensional head shape and the color map. Since the embodiment of the present application utilizes the color map (i.e., RGB map) and the depth map of the target user obtained at multiple angles, the color map and the depth map are combined to reconstruct the three-dimensional head shape to generate the target three-dimensional head shape of the target user, and then the target three-dimensional head shape and the RGB map of the target user are used to obtain a texture map. The texture map generated in this way not only fits the head shape of the target user, but also has clear texture quality, which solves the problem in the related art that the texture of the map corresponding to the generated three-dimensional image of the user is not clear and cannot reflect the true texture condition of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 is a schematic diagram of a hardware environment of an optional method for generating a texture map according to an embodiment of the present invention;

[0041] Figure 2 is a schematic flow chart of an optional method for generating a texture map according to an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of the overall process of an optional texture map generation method according to an embodiment of the present application;

[0043] Figure 4 is a structural block diagram of an optional texture map generation device according to an embodiment of the present application;

[0044] Figure 5 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] According to one aspect of an embodiment of the present application, a method for generating a texture map is provided. Optionally, in this embodiment, the method for generating a texture map can be applied to Figure 1 In the hardware environment shown in the figure. Figure 1 As shown, the terminal 102 may include a memory 104, a processor 106, and a display 108 (optional component). The terminal 102 may be connected to a server 112 via a network 110 for communication. The server 112 may be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database 114 may be set on the server 112 or independently of the server 112 to provide data storage services for the server 112. In addition, a processing engine 116 may be running in the server 112, and the processing engine 116 may be used to execute the steps executed by the server 112.

[0048] Optionally, the terminal 102 may be, but is not limited to, a terminal that can calculate data, such as a mobile terminal (e.g., a mobile phone, a tablet computer), a laptop, a PC (Personal Computer), etc. The above network may include, but is not limited to, a wireless network or a wired network. The wireless network includes: Bluetooth, WIFI (Wireless Fidelity) and other networks that implement wireless communication. The above wired network may include, but is not limited to: a wide area network, a metropolitan area network, and a local area network. The above server 112 may include, but is not limited to, any hardware device that can perform calculations.

[0049] In addition, in this embodiment, the above-mentioned texture map generation method can also be applied to, but not limited to, an independent processing device with a relatively powerful processing capability without data interaction. For example, the processing device can be, but not limited to, a terminal device with a relatively powerful processing capability, that is, each operation in the above-mentioned texture map generation method can be integrated in an independent processing device. The above is only an example, and this embodiment does not make any limitation to this.

[0050] Optionally, in this embodiment, the texture map generation method can be executed by the server 112, or by the terminal 102, or by both the server 112 and the terminal 102. The terminal 102 can execute the texture map generation method of the embodiment of the present application, or it can be executed by a client installed thereon.

[0051] Take running on the server as an example, Figure 2 is a flow chart of an optional method for generating a texture map according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps:

[0052] Step S201, obtaining color maps and depth maps corresponding to the target user at multiple angles.

[0053] Optionally, the embodiment of the present application uses a camera device, such as a smart phone, a camera, etc., to shoot the target user from multiple angles, such as rotating 30 degrees from 5 angles, top, bottom, left, right, and front, to collect images of the target user's head, and obtain color maps and depth maps generated at multiple angles. The color map is an RGB map, which is used to characterize the target user's skin color, eyebrows, and other colorful colors, and the depth map is used to characterize the target user's three-dimensional contour.

[0054] Step S202: reconstruct the three-dimensional head shape of the target user using the color map and the depth map to obtain a target three-dimensional head shape of the target user.

[0055] Optionally, in an embodiment of the present application, the color map and the depth map are combined to reconstruct the three-dimensional head shape of the target user, and the target three-dimensional head shape of the target user is obtained after reconstruction.

[0056] In addition, the embodiment of the present application can use the model to generate the three-dimensional head shape of the target user. For example, a bilinear three-dimensional deformable model (3DMorphable Model, 3DMM) based on PCA can be used as a base model, that is, the 3DMM model is used as an initial model, wherein the initial model is used to reconstruct the three-dimensional head shape of the target user. At this time, the obtained color map and depth map are first input into the initial model to obtain an output head shape state, and then the head shape is rendered from different angles, and the model parameters of the initial model are continuously adjusted to determine the definite model, and the definite model is used as the final model that can output the three-dimensional head shape that conforms to the target user. Using the definite model, the three-dimensional head shape that conforms to the real head shape of the target user can be output as the target three-dimensional head shape. Here, the target three-dimensional head shape obtained is more personalized, more accurate, and more in line with the three-dimensional image of the target user.

[0057] Step S203, generating a texture map using the target three-dimensional head shape and color map.

[0058] Optionally, after obtaining the target three-dimensional head shape, the target three-dimensional head shape and the two-dimensional image are mapped and projected using a mapping function in combination with the color map, thereby generating a texture map.

[0059] In an embodiment of the present application, a color map and a depth map corresponding to the target user at multiple angles are obtained; the three-dimensional head shape of the target user is reconstructed using the color map and the depth map to obtain the target three-dimensional head shape of the target user; and a texture map is generated using the target three-dimensional head shape and the color map. Since the embodiment of the present application utilizes the color map (i.e., RGB map) and the depth map of the target user obtained at multiple angles, the color map and the depth map are combined to reconstruct the three-dimensional head shape to generate the target three-dimensional head shape of the target user, and then the target three-dimensional head shape and the RGB map of the target user are used to obtain a texture map. The texture map generated in this way not only fits the head shape of the target user, but also has clear texture quality, which solves the problem in the related art that the texture of the map corresponding to the generated three-dimensional image of the user is not clear and cannot reflect the true texture condition of the user.

[0060] As an optional embodiment, reconstructing the three-dimensional head shape of the target user by using the color map and the depth map to obtain the target three-dimensional head shape of the target user includes:

[0061] According to the color map and the depth map, the initial three-dimensional head shape of the target user is obtained;

[0062] Rendering an initial three-dimensional head shape at a target angle using an identity vector and an expression vector to obtain a reference depth map at the target angle after rendering, wherein the identity vector and the expression vector are used to describe an identity dimension coefficient and an expression dimension coefficient of the target user;

[0063] The target 3D head shape is obtained using the color map, depth map and reference depth map.

[0064] Optionally, in the embodiment of the present application, taking the construction of an initial model and adjusting the model parameters of the initial model to obtain the target three-dimensional head shape by the determination model as an example, the following description is expanded: the color map and depth map of the target user at multiple angles can be input into the initial model to obtain the initial three-dimensional head shape of the target user, and then the parameters of the 3DMM are predicted to reconstruct the three-dimensional head model of the user. The formula is expressed as formula (1):

[0065]

[0066] Among them, S reconstructs the user's three-dimensional head shape, is the average shape of 3DMM, B id ,B exp are the identity PCA basis and expression PCA basis of the 3DMM model, respectively. i ,c e are the predicted 3DMM identity dimension coefficients and expression dimension coefficients. In addition, in the process of fitting 3DMM parameters, it is also necessary to predict the posture parameters p at each angle i , i∈(1,2,3,4,5).

[0067] Here, the gradient descent algorithm is used as the optimization algorithm, and the model parameter to be optimized is Δ={c i ,c e ,p i}, at each iteration of the algorithm, the parameter c is first used i , c e The user's three-dimensional head shape S is reconstructed, and then it is input into the differentiable renderer together with the posture parameters of each angle to obtain the rendered reference depth map. Finally, the prediction error is calculated, and the error is back-propagated to the parameter Δ, and the parameter Δ is updated to finally obtain the determined model.

[0068] At the same time, in the process of backpropagating the error to the parameter Δ and updating the parameter Δ, it is necessary to set a first threshold (such as 0.1) in advance, and compare the numerical size of the error value obtained each time with the first threshold, wherein the first threshold is the critical value for stopping the cyclic comparison of the error value with the first threshold. When it is determined that the current error value is less than the first threshold, the updating of the parameter Δ can be stopped. The current 3DMM model is the determined model, and the target three-dimensional head shape is output according to the determined model.

[0069] In an embodiment of the present application, the model parameters are continuously optimized and adjusted by using the error value between the rendered reference depth map and the depth map of the target user to obtain a determined model, so that the three-dimensional head shape output by the determined model is more consistent with the actual head shape of the target user.

[0070] As an optional embodiment, using the color map, the depth map, and the reference depth map to determine the error value includes:

[0071] Detect key points on the color map;

[0072] Obtaining first key point coordinates of the key point on the color map, obtaining second key point coordinates of the key point on the three-dimensional image corresponding to the target three-dimensional head shape, and obtaining third key point coordinates on the two-dimensional image mapped to the second key point coordinates by a mapping function;

[0073] The error value is determined according to the depth error weight, the depth map, the reference depth map, the key point error weight, the first key point coordinates, the third key point coordinates, the regularization term weight, the identity vector and the expression vector, wherein the depth error weight is used to adjust the contribution value of the depth error function composed of the depth map and the reference depth map, the key point error weight is used to adjust the contribution value of the key point error function composed of the first key point coordinates and the third key point coordinates, and the regularization term weight is used to adjust the contribution value of the regularized error function composed of the identity vector and the expression vector.

[0074] Optionally, in an embodiment of the present application, a method for determining the error value is specifically described as follows:

[0075] Set the RGB image of the target user at the i-th angle to The depth map is set to The rendered depth map is set to The overall error function can be defined as:

[0076] τ=λ dep τ dep +λ lan τ lan +λ reg τ reg (2)

[0077] Among them, λ dep is the depth error weight, τ dep is the depth error function, τ dep Define the L2 distance mean of the depth map and rendered depth map at each angle:

[0078]

[0079] λ lan is the key point error weight, τ lan is the key point error function, τ lan It is defined as the mean L2 distance between the key points detected on the RGB images at each angle (i.e., the coordinates of the first key points) and their corresponding key points on the target three-dimensional head shape S (i.e., the coordinates of the second key points):

[0080]

[0081] Here ψ is the key point detection neural network, which is used to detect the key point coordinates on the RGB image. lan is a subset of the three-dimensional shape S, S lan The index meaning of each point in , for example, the first point represents the eye corner. Π is a mapping function that projects the points on the target three-dimensional head shape onto a two-dimensional plane. That is, the eye corner can be used as a key point, and the second key coordinate point of the key point on the target three-dimensional head shape can be projected onto the two-dimensional plane to obtain the coordinates of the third key point.

[0082] λ reg is the regularization term weight, τ reg is the regularized error function, τ reg Positioned as c i and c e The L2 norm of is:

[0083]

[0084] ω i and ω e These are weight values ​​used to adjust c i and c e The numerical value of c i and c e Contribution value is balanced.

[0085] In addition, the above λ dep , lan , reg The purpose of is to adjust the contribution value of its product function to achieve the purpose of balance.

[0086] Finally, referring to formula (2), the above-obtained τdep The value obtained by the function is the same as λ dep The product of the depth error weight, τ lan The value obtained by the function is the same as λ lan The product of the key point error weights, and τ reg The value obtained by the function is the same as λ reg The products of the regularization item weights are then summed up to obtain the error value.

[0087] In an embodiment of the present application, by calculating a color map and a depth map at any angle, and calculating a linear error value between the color map and the depth map obtained by rendering, the model parameters are continuously adjusted to obtain an accurate three-dimensional head shape output model.

[0088] As an optional embodiment, using the target three-dimensional head shape and color map to generate a texture map includes:

[0089] Get the two-dimensional image corresponding to the color map;

[0090] Mapping the three-dimensional geometric shape points corresponding to the target three-dimensional head shape onto the two-dimensional image to obtain the coordinates of the points mapped onto the two-dimensional image;

[0091] Image sampling is performed on the two-dimensional image through point coordinates to obtain the corresponding texture map.

[0092] Optionally, texture sampling is a commonly used texture map acquisition method in three-dimensional reconstruction. In the embodiment of the present application, after reconstructing the target three-dimensional head shape, the points on the target three-dimensional head shape can be mapped back to the two-dimensional image plane through the posture coefficient and the camera coefficient. The formula is described as:

[0093] V 2d =Π(S) (6)

[0094] As in the above embodiment, Π is a mapping function that projects points on the target three-dimensional head shape onto a two-dimensional plane. 2d To obtain the point coordinates of the target 3D head shape on the 2D image plane. Then, by sampling these point coordinates on the image, we can get the texture map. The formula is described as:

[0095] T=F(V 2d ) (7)

[0096] Obtained texture map. It should be noted that, in the embodiment of the present application, the number of obtained texture maps is multiple, and they are texture maps obtained from multiple angles.

[0097] As an optional embodiment, the number of texture maps is multiple, and after the texture maps are generated by using the target three-dimensional head shape and the color map, the method further includes:

[0098] Merging multiple texture maps to obtain a first texture map;

[0099] The first texture map is processed for illumination balance and texture detail optimization to obtain a processed texture map.

[0100] Optionally, after obtaining the generated texture map that matches the target three-dimensional head shape of the target user, in order to obtain a clearer and higher quality texture map, the texture map can be subjected to illumination equalization processing and texture detail optimization processing to obtain a processed texture map.

[0101] For the texture maps obtained at multiple angles, image pyramid fusion is used to fuse them into a pre-made map template. This is because the texture map at a single angle only contains an incomplete facial area. Fusion of texture maps from multiple perspectives can obtain a complete facial map, which can be fused with the template to complete the map area outside the face (for example, neck, ears, etc.) to obtain the first texture map.

[0102] Then, based on the first texture map, illumination equalization and texture detail optimization are processed to obtain a processed texture map. See 3, which includes: a multi-view data acquisition module, a multi-view head reconstruction module, a multi-view texture map fusion module, a map illumination equalization module, and a map detail optimization module.

[0103] In an embodiment of the present application, by fusing, equalizing, and optimizing the texture details of the multiple texture maps obtained, the processed texture map removes the uneven lighting in the map, thereby avoiding the adverse effects of these uneven lighting on subsequent applications, while increasing the skin texture details of the map and improving the resolution of the map.

[0104] As an optional embodiment, the first texture map is processed for illumination balance and texture detail optimization, and the processed texture map includes:

[0105] Inputting the first texture map into the first model to obtain the second texture map, wherein the first model is trained using the texture map sample data with balanced illumination and the texture map sample data with uneven illumination of the face, and is used to generate the texture map with balanced illumination;

[0106] The second texture map is input into the second model to generate a processed texture map, wherein the second model is trained using sample data with texture map clarity lower than a second threshold and sample data with texture map clarity higher than the second threshold, and is used to optimize the texture of the map in detail, and the second threshold is used to indicate the minimum value of clarity of the texture map.

[0107] Optionally, in an embodiment of the present application, two models are provided, one is a first model, which may also be a texture illumination equalization model, and the other is a second model, which may also be a texture detail optimization model. In an embodiment of the present application, the first texture map obtained after fusing multiple texture maps is subjected to illumination equalization and texture detail optimization processing, wherein the illumination equalization processing is specifically: an illumination equalization neural network based on a generative adversarial neural network is used to remove uneven illumination in the map.

[0108] Firstly, we use the existing face relighting method to generate a batch of lighting equalization neural network training data sets. Each pair of data in the data set contains a texture map with balanced lighting and a corresponding texture map with unbalanced lighting.

[0109] The data is then used to train the illumination equalization neural network, which consists of a generator and a discriminator. The generator consists of two encoder structures and a decoder structure, which uses the U-Net structure. The image encoder is used to extract feature information on the RGB image at the front angle, and the map encoder is used to extract feature information on the texture map. The discriminator uses the Patch-GAN structure, which is only used during training and will be removed during application, leaving only the generator.

[0110] The trained model is used as the first model, and the first texture map is input into the first model to obtain the second texture map.

[0111] The texture detail optimization process is as follows: based on the internal m high-quality texture maps (e.g. 800), the training data is further expanded to about n (e.g. 2000) by using methods such as texture color mapping and texture area replacement. These high-quality texture maps are then downsampled, and some random unbalanced illumination is added, and input into the first model. The final data is regarded as low-quality texture maps, and is paired with high-quality texture maps to form a texture detail optimization neural network training data set. Among them, the clarity of the low-quality texture map is lower than the second threshold, and the clarity of the high-quality texture map is higher than the second threshold. The second threshold is used to indicate that the texture map is the minimum value of clarity.

[0112] This dataset is used to train the texture detail optimization neural network. Here, the texture detail optimization neural network directly adopts the pixel2pixelHD generative adversarial network, which has the advantages of generating images with high resolution, richer details and higher realism.

[0113] The trained map detail optimization neural network model is used as the second model, and the second texture map is input into the second model to generate a processed texture map.

[0114] In the embodiment of the present application, the texture map processed by the first model and the second model has high resolution, richer details and higher realism. At the same time, since the obtained target map has high quality, it can enrich the user's entertainment experience in the virtual world and improve the commercial value.

[0115] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0117] According to another aspect of an embodiment of the present application, a texture map generating device for implementing the above-mentioned texture map generating method is also provided. Figure 4 is a structural block diagram of an optional texture map generation device according to an embodiment of the present application, such as Figure 4 As shown, the device may include:

[0118] An acquisition unit 401 is used to acquire a color map and a depth map corresponding to a target user generated at multiple angles;

[0119] A first obtaining unit 402 is connected to the obtaining unit 401 and is used to reconstruct the three-dimensional head shape of the target user by using the color map and the depth map to obtain a target three-dimensional head shape of the target user;

[0120] The generating unit 403 is connected to the first obtaining unit 402 and is used to generate a texture map using the target three-dimensional head shape and the color map.

[0121] It should be noted that the acquisition unit 401 in this embodiment can be used to execute the above step S201, the first obtaining unit 402 in this embodiment can be used to execute the above step S202, and the generation unit 403 in this embodiment can be used to execute the above step S203.

[0122] Through the above modules, the color map (i.e., RGB map) and depth map of the target user obtained at multiple angles are used to combine the color map and the depth map to reconstruct the three-dimensional head shape, generate the target three-dimensional head shape of the target user, and then use the target three-dimensional head shape and the RGB map of the target user to obtain a texture map, and further refine the texture map to finally determine the target texture map. The target texture map generated in this way not only fits the head shape of the target user, but also has clear texture quality, which solves the problem in the related art that the texture of the map corresponding to the generated three-dimensional image of the user is not clear and cannot reflect the true texture condition of the user.

[0123] As an optional embodiment, the first obtaining unit includes:

[0124] The first obtaining module is used to obtain the initial three-dimensional head shape of the target user according to the color map and the depth map;

[0125] A rendering module, used to render an initial three-dimensional head shape at a target angle using an identity vector and an expression vector to obtain a reference depth map at the target angle after rendering, wherein the identity vector and the expression vector are used to describe an identity dimension coefficient and an expression dimension coefficient of a target user;

[0126] The second obtaining module is used to obtain the target three-dimensional head shape using the color map, the depth map and the reference depth map.

[0127] As an optional embodiment, the second obtaining module includes:

[0128] A determination subunit, configured to determine an error value using the color map, the depth map, and the reference depth map;

[0129] A comparison subunit, used to compare the error value with a first threshold value, wherein the first threshold value is a critical value for stopping the cyclic comparison between the error value and the first threshold value;

[0130] A subunit is obtained, which is used to obtain a target three-dimensional head shape when the error value is less than a first threshold.

[0131] As an optional embodiment, the determining subunit includes:

[0132] The detection submodule is used to detect key points on the color map;

[0133] An acquisition submodule is used to acquire a first key point coordinate of a key point on a color map, acquire a second key point coordinate of a key point on a three-dimensional image corresponding to a target three-dimensional head shape, and acquire a third key point coordinate on a two-dimensional image mapped from the second key point coordinate using a mapping function;

[0134] A determination submodule is used to determine the error value according to the depth error weight, the depth map, the reference depth map, the key point error weight, the first key point coordinates, the third key point coordinates, the regularization term weight, the identity vector and the expression vector, wherein the depth error weight is used to adjust the contribution value of the depth error function composed of the depth map and the reference depth map, the key point error weight is used to adjust the contribution value of the key point error function composed of the first key point coordinates and the third key point coordinates, and the regularization term weight is used to adjust the contribution value of the regularized error function composed of the identity vector and the expression vector.

[0135] As an optional embodiment, the generating unit includes:

[0136] An acquisition module, used for acquiring a two-dimensional image corresponding to the color map;

[0137] A mapping module, used for mapping the three-dimensional geometric shape points corresponding to the target three-dimensional head shape onto the two-dimensional image to obtain the coordinates of the points mapped onto the two-dimensional image;

[0138] The sampling module is used to perform image sampling on a two-dimensional image through point coordinates to obtain a corresponding texture map.

[0139] As an optional embodiment, the number of texture maps is multiple, and the device further includes:

[0140] A fusion unit, for fusing a plurality of texture maps to obtain a first texture map after generating the texture map using the target three-dimensional head shape and the color map;

[0141] The second obtaining unit is used to perform illumination balance and texture detail optimization processing on the first texture map to obtain a processed texture map.

[0142] As an optional embodiment, the second obtaining unit includes:

[0143] A first input module, used to input the first texture map into a first model to obtain a second texture map, wherein the first model is trained using facial texture map sample data with balanced illumination and texture map sample data with uneven illumination, and is used to generate a texture map with balanced illumination;

[0144] The second input module is used to input the second texture map into the second model to generate a processed texture map, wherein the second model is trained using sample data with texture map clarity lower than a second threshold and sample data with texture map clarity higher than the second threshold, and is used to optimize the texture of the map in detail, and the second threshold is used to indicate the minimum value of the texture map being clear.

[0145] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0146] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned texture map generation method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0147] Figure 5 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 communicate with each other through the communication bus 504, wherein,

[0148] Memory 503, used for storing computer programs;

[0149] The processor 501 is used to execute the computer program stored in the memory 503 to implement the following steps:

[0150] Obtain the color map and depth map generated corresponding to the target user at multiple angles;

[0151] Reconstructing the three-dimensional head shape of the target user using the color map and the depth map to obtain the target three-dimensional head shape of the target user;

[0152] Using the target 3D head shape and color map, a texture map is generated.

[0153] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0154] The communication interface is used for communication between the above electronic device and other devices.

[0155] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0156] As an example, Figure 5 As shown, the memory 503 may include but is not limited to the acquisition unit 401, the first obtaining unit 402, and the generating unit 403 in the texture map generating device. In addition, it may also include but is not limited to other module units in the texture map generating device, which will not be repeated in this example.

[0157] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0158] In addition, the electronic device mentioned above further includes: a display for displaying the generation result of the texture map.

[0159] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0160] It can be understood by those skilled in the art that Figure 5 The structure shown is for illustration only. The device for implementing the above-mentioned texture map generation method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 5 It does not limit the structure of the above electronic device. For example, the terminal device may also include Figure 5More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations shown.

[0161] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0162] According to another aspect of the embodiment of the present application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of the method for generating a texture map.

[0163] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0164] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0165] Obtain the color map and depth map generated corresponding to the target user at multiple angles;

[0166] Reconstructing the three-dimensional head shape of the target user using the color map and the depth map to obtain the target three-dimensional head shape of the target user;

[0167] Using the target 3D head shape and color map, a texture map is generated.

[0168] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0169] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0170] According to another aspect of the embodiments of the present application, a computer program product or a computer program is also provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the texture map generation method in any of the above embodiments.

[0171] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0172] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the texture map generation method of each embodiment of the present application.

[0173] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a texture map, characterized in that: The method comprises: Obtain the color map and depth map generated corresponding to the target user at multiple angles; Reconstructing the three-dimensional head shape of the target user by using the color map and the depth map to obtain the target three-dimensional head shape of the target user, including: obtaining the initial three-dimensional head shape of the target user according to the color map and the depth map; rendering the initial three-dimensional head shape by using the identity vector and the expression vector at the target angle to obtain the rendered reference depth map at the target angle, wherein the identity vector and the expression vector are used to describe the identity dimension coefficient and the expression dimension coefficient of the target user; determining an error value by using the color map, the depth map and the reference depth map, comparing the error value with a first threshold, wherein the first threshold is a critical value for stopping the cyclic comparison of the error value with the first threshold, and obtaining the target three-dimensional head shape when the error value is less than the first threshold; A texture map is generated using the target three-dimensional head shape and the color map.

2. The method according to claim 1, characterized in that The determining the error value by using the color map, the depth map, and the reference depth map comprises: Detecting key points on the color map; Acquire a first key point coordinate of the key point on the color map, acquire a second key point coordinate of the key point on the three-dimensional image corresponding to the target three-dimensional head shape, and acquire a third key point coordinate on the two-dimensional image mapped by the second key point coordinate using a mapping function; The error value is determined according to the depth error weight, the depth map, the reference depth map, the key point error weight, the first key point coordinates, the third key point coordinates, the regularization term weight, the identity vector and the expression vector, wherein the depth error weight is used to adjust the contribution value of the depth error function composed of the depth map and the reference depth map, the key point error weight is used to adjust the contribution value of the key point error function composed of the first key point coordinates and the third key point coordinates, and the regularization term weight is used to adjust the contribution value of the regularized error function composed of the identity vector and the expression vector.

3. The method according to claim 1, characterized in that The step of generating a texture map by using the target three-dimensional head shape and the color map comprises: Obtaining a two-dimensional image corresponding to the color map; Mapping the three-dimensional geometric shape points corresponding to the target three-dimensional head shape onto the two-dimensional image to obtain the point coordinates mapped onto the two-dimensional image; Image sampling is performed on the two-dimensional image using the point coordinates to obtain the corresponding texture map.

4. The method according to claim 1, characterized in that: The number of the texture maps is multiple. After the texture maps are generated by using the target three-dimensional head shape and the color map, the method further includes: Merging the plurality of texture maps to obtain a first texture map; The first texture map is processed for illumination balance and texture detail optimization to obtain a processed texture map.

5. The method according to claim 4, characterized in that The processing of performing illumination balance and texture detail optimization on the first texture map to obtain the processed texture map comprises: Inputting the first texture map into a first model to obtain a second texture map, wherein the first model is trained using facial texture map sample data with balanced illumination and texture map sample data with uneven illumination, and is used to generate a texture map with balanced illumination; The second texture map is input into a second model to generate the processed texture map, wherein the second model is trained using sample data with texture map clarity lower than a second threshold and sample data with texture map clarity higher than the second threshold, and is used to optimize the details of the texture of the map, and the second threshold is used to indicate the minimum value of clarity of the texture map.

6. A texture map generation device, characterized in that: The device comprises: An acquisition unit, used to acquire a color map and a depth map corresponding to a target user generated at multiple angles; The first obtaining unit is used to reconstruct the three-dimensional head shape of the target user by using the color map and the depth map to obtain the target three-dimensional head shape of the target user, including: obtaining the initial three-dimensional head shape of the target user according to the color map and the depth map; rendering the initial three-dimensional head shape by using the identity vector and the expression vector at a target angle to obtain a reference depth map at the target angle after rendering, wherein the identity vector and the expression vector are used to describe the identity dimension coefficient and the expression dimension coefficient of the target user; determining an error value by using the color map, the depth map and the reference depth map, comparing the error value with a first threshold, wherein the first threshold is a critical value for stopping the cyclic comparison of the error value with the first threshold, and obtaining the target three-dimensional head shape when the error value is less than the first threshold; A generating unit is used to generate a texture map using the target three-dimensional head shape and the color map.

7. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method steps of any one of claims 1 to 5 by running the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program implements the method steps described in any one of claims 1 to 5 when executed by a processor.