A method, device and electronic device for generating texture of three-dimensional head model

Abstract

The present application relates to computer graphics, and more specifically to a method, device and electronic device for generating texture of a three-dimensional head model. The method comprises: obtaining an original facial image in RGB format, converting the original facial image into an image of a Tensor object; segmenting the image of the Tensor object according to different parts of the face, obtaining at least two region images from the image segmentation result, and superimposing the at least two region images to obtain a mixed region image; sampling the color of the image of the Tensor object through a UV mapping relationship and mapping it to a UV texture map; performing template replacement on the UV texture map according to a predefined template image, a mask image and the mixed region image to obtain a replaced UV texture map. The embodiment of the present application realizes the combination of traditional image processing methods and depth algorithms. Template replacement makes the generated three-dimensional head model texture closer to the real object, solves the problems existing in the prior art, and improves the efficiency of texture generation.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and more specifically, to a method, device and electronic device for generating texture of a three-dimensional head model. Background Art

[0002] With the rapid development of digital media and immersive technology, the demand for three-dimensional head models is growing, especially models that can accurately reproduce human facial details, and texture is a top priority. The texture of a three-dimensional head model involves mapping a two-dimensional image onto the surface of a three-dimensional head model, using color, pattern, bumps and other details to enhance the realism and realistic appearance of the object. The generation of the texture of the three-dimensional model can be achieved by using at least one image editing software.

[0003] The existing 3D head model texture generation technology still cannot highly restore the original facial image on the 3D head model. Summary of the invention

[0004] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a method, device and electronic device for generating texture of a three-dimensional head model, so as to achieve the effect of highly restoring the original facial image on the three-dimensional head model.

[0005] According to a first aspect of the present application, a method for generating a 3D head model texture is provided, comprising:

[0006] Obtain an original facial image in RGB format, and convert the original facial image into an image of a Tensor object;

[0007] Segment the image of the Tensor object according to different parts of the face to obtain an image segmentation result;

[0008] Acquire at least two regional images from the image segmentation result, and superimpose the at least two regional images to obtain a mixed regional image;

[0009] The image of the Tensor object is sampled and mapped to the UV texture map through the UV mapping relationship;

[0010] The UV texture map is subjected to template replacement according to the predefined template image, the mask image and the mixed region image to obtain a replaced UV texture map.

[0011] The three-dimensional head model texture generation method can enable the original facial image to complete the texture reproduction on the head model, realize personalized customization, improve the expression of details, and enhance the realism of the three-dimensional head model.

[0012] Optionally, converting the original facial image into an image of a Tensor object, the method includes:

[0013] Preprocessing the original facial image to obtain a preprocessed facial image;

[0014] The preprocessing includes one or more of filtering processing, brightness correction processing, and normalization processing;

[0015] Convert the original facial image into an image of a Tensor object, specifically including:

[0016] The preprocessed image of the original facial image is converted into an image of a Tensor object.

[0017] The original facial image is preprocessed before being converted into an image of a Tensor object. The quality of the preprocessed original facial image is improved, noise contained in the image is removed, and the pixel value range is unified. The preprocessed image is then converted into an image of a Tensor object, thereby improving the stability and performance of the deep algorithm.

[0018] Optionally, segmenting the image of the Tensor object according to different parts of the face to obtain an image segmentation result specifically includes:

[0019] The image of the Tensor object is input into the face segmentation network for segmentation to obtain an image segmentation result.

[0020] By segmenting the image of the Tensor object through the face segmentation network, segmentation result images of different parts of the face can be obtained, the required parts can be effectively extracted and irrelevant parts in the image can be shielded.

[0021] Optionally, processing the image segmentation result of the Tensor object, the method includes:

[0022] Binarizing the image segmentation result to obtain binary regional images of different parts of the face;

[0023] The acquiring at least two regional images from the image segmentation result specifically includes:

[0024] Acquire at least two region images from the binary region image corresponding to the image segmentation result, wherein the at least two region images include a facial skin region;

[0025] The at least two region images are superimposed to obtain a mixed region image.

[0026] Optionally, the image of the Tensor object samples its color through a UV mapping relationship and maps it to a UV texture map.

[0027] The image of the Tensor object defines the mapping relationship between the 2D texture image and the 3D vertices through UV mapping;

[0028] The image of the Tensor object obtains the mapping relationship between the 3D vertices and the image pixels of the Tensor object through the projection model;

[0029] According to the mapping relationship between the 3D vertices and the image pixels of the Tensor object, the corresponding positions are found on the UV texture map for sampling, and the sampling results are mapped to the UV texture map.

[0030] The color of the image of the Tensor object is sampled and mapped to the UV texture map through the UV mapping relationship. The position and proportion of the texture of the original facial image on the model are highly detailed. Multiple different textures are applied to the same model at the same time, which improves the generation efficiency of the head model texture.

[0031] Optionally, the sampled and mapped UV texture map is processed, specifically including:

[0032] Perform lighting balance on the sampled and mapped UV texture map;

[0033] Input the UV texture image after illumination equalization into the facial shadow segmentation network to obtain the small shadow areas in the image;

[0034] Performing a repair operation on a small shadow area in the acquired image to obtain a UV texture map after the shadow is eliminated;

[0035] The step of performing template replacement on the UV texture map according to the predefined template image, the mask image and the mixed area image to obtain the replaced UV texture map is specifically as follows:

[0036] The UV texture image after the shadow is eliminated is subjected to template replacement according to the predefined template image, the mask image and the mixed area image to obtain the replaced UV texture image.

[0037] The UV texture image is subjected to illumination balancing and shadow elimination processing to improve image consistency and reduce the influence of noise. The illumination conditions of the processed image are all good and the image readability is higher.

[0038] Optionally, the predefined template image, mask image and mixed area image perform template replacement on the UV texture map to obtain a replaced UV texture map,

[0039] Multiplying the UV texture map with the mixed area image to obtain a UV texture map of a valid area of ​​the mixed area image;

[0040] Multiplying the predefined template image with the mixed area image to obtain a texture map of the template valid area;

[0041] Calculate the superposition weight of the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template;

[0042] Superimposing the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template according to the superposition weight to obtain a superimposed image;

[0043] multiplying the superimposed image and the predefined mask image to obtain a first image;

[0044] Multiplying the predefined template image and the inverse mask image corresponding to the predefined mask image to obtain a second image;

[0045] The first image and the second image are mixed to obtain a replaced UV texture map.

[0046] The template of the UV texture map is replaced to reproduce the blocked part of the texture map, thereby improving the personalization and recognition of the model.

[0047] Optionally, the image of the Tensor object is processed, specifically including:

[0048] Performing one or more of super-resolution and face restoration on the image of the Tensor object to obtain a third image;

[0049] The color of the image of the Tensor object is sampled and mapped to the UV texture map through the UV mapping relationship, specifically: the color of the third image is sampled and mapped to the UV texture map through the UV mapping relationship;

[0050] And / or, processing the replaced UV texture map to obtain a processed image, specifically including:

[0051] The replaced UV texture map is input into the GFPGAN network to perform face repair to obtain a repaired UV texture map.

[0052] The image of the Tensor object is processed by super-resolution, face repair, etc., and the resolution and clarity of the processed image are improved, which can not only remove noise well but also retain the detailed information on the face; the replaced UV texture map is repaired, and the repaired image effectively improves the final visual effect of the model and achieves a high degree of texture restoration.

[0053] According to a second aspect of the present application, a three-dimensional head model texture generation device is provided, comprising:

[0054] A conversion module, used to obtain an original facial image in RGB format, and convert the original facial image into an image of a Tensor object;

[0055] A segmentation module, used to segment the image of the Tensor object according to different parts of the face to obtain an image segmentation result;

[0056] An image superposition module, used for obtaining at least two regional images from the image segmentation result, and superimposing the at least two regional images to obtain a mixed regional image;

[0057] A mapping module, used to sample the color of the image of the Tensor object through a UV mapping relationship and map it to a UV texture map;

[0058] The template replacement module is used to perform template replacement on the UV texture image according to a predefined template image, a mask image and the mixed area image to obtain a replaced UV texture image.

[0059] In the three-dimensional head model texture generation device, each module is an indispensable part of the whole process, and the modules cooperate with each other. The original facial image is converted into an image format through the conversion module; then through the segmentation module, the segmentation result image is superimposed and enters the next processing module. The processed original facial image passes through the mapping module, and its color is texture sampled using the UV mapping relationship to obtain a UV texture map. The UV texture map is template replaced in the template replacement module, and the UV texture map is template replaced according to the predefined template image, the mask image and the mixed area image to obtain the replaced UV texture map.

[0060] According to a third aspect of the present application, an electronic device is provided, including:

[0061] a memory for storing one or more computer programs;

[0062] The processor implements the three-dimensional head model texture generation method described in the first aspect when the one or more computer programs are executed by the processor.

[0063] The electronic device provides a complete system for generating texture of a three-dimensional head model, stores images, processes the stored images, and the processed images are stored in the memory.

[0064] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the three-dimensional head model texture generation method described in the first aspect when executed.

[0065] Based on any of the above aspects, the three-dimensional head model texture generation method, device, and electronic device provided in the embodiments of the present application enable the texture of the original facial image to be highly reproduced on the three-dimensional head model. The format of the original facial image is converted into an image of a Tensor object, which is conducive to deep algorithm training to process the image. The image of the Tensor object is segmented according to different parts of the face to obtain an image segmentation result. Segmenting the image can effectively extract the required parts of the original image and shield irrelevant parts of the image. At least two regional images are obtained from the image segmentation result, and then the at least two regional images are superimposed to obtain a mixed regional image, which is used as a component of the template replacement link. The image of the Tensor object samples its color through a UV mapping relationship and maps it to a UV texture map. The UV coordinates can accurately correspond the pixels of the original image to the surface of the model object, and process the gaps between points. The details of the original facial image can be projected onto the texture map, which is essentially a process of sampling and reproducing the original image. The UV texture map is replaced by a template according to a predefined template image, a mask image and the mixed area image to obtain a replaced UV texture map. The template replacement is combined with the image restoration technology to replace the UV texture map. The replaced UV texture map is consistent with the original facial image in terms of lighting conditions and personalized features are restored. This solves the defects of the current texture generation technology and improves production efficiency.

[0066] The original facial image is preprocessed, and the quality of the original facial image after the preprocessing is improved, the noise contained in the image is removed, and the pixel value range is unified.

[0067] The UV texture map is subjected to illumination balancing and shadow elimination processing to improve image consistency and reduce the influence of noise. After the processing, the illumination conditions of the image are good both overall and locally, and the image readability is higher.

[0068] The UV texture map after template replacement is processed, and the repaired image effectively improves the final visual effect of the model and achieves a high degree of texture restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 A schematic application scenario diagram of the three-dimensional head model texture generation method provided in this embodiment.

[0071] Figure 2 A schematic diagram of a flow chart of a method for generating texture of a three-dimensional head model provided in this embodiment.

[0072] Figure 3 A template replacement flow chart of a 3D head model texture generation method provided in this embodiment.

[0073] Figure 4 A schematic diagram of the functional modules of a three-dimensional head model texture generation device provided in this embodiment.

[0074] Figure 5 A schematic diagram of an electronic device for a method for generating texture of a three-dimensional head model provided in this embodiment. DETAILED DESCRIPTION

[0075] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0077] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0078] Existing texture generation technology has defects. When sampling the texture of the original facial image, the changes in environmental parameters such as the position and intensity of the light source may cause the generated texture map to be inconsistent under different lighting conditions, and shadows may appear in some parts. At the same time, noise and artifacts in the original image, such as pixelation, blur or scan lines, will be retained in the texture map if not effectively processed, affecting the realism and visual effect of the final model. When dealing with occluded or defective areas such as behind the ears and the back of the head, the existing technology lacks an effective solution, which may lead to incomplete texture or inconsistent texture problems. For complex facial features, such as wrinkles, moles, scars and other personalized marks, the existing technology may not be able to accurately capture and express them, reducing the personalization and recognizability of the model.

[0079] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0080] For example, a schematic diagram of an application scenario of a three-dimensional head model texture generation method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100, the server 100 has an image processing function; the terminal device 200 has a streaming media playback function and may also have an image processing function.

[0081] It is understandable that the server 100 can be an independent electronic device or a cluster composed of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, etc., but is not limited thereto.

[0082] In an practicable manner, the server 100 and the terminal 200 may respectively execute the three-dimensional head model texture generation method provided in the embodiment of the present application, or, optionally, the three-dimensional head model texture generation method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0083] like Figure 2 As shown, this embodiment provides a method for generating a 3D head model texture, which may include the following steps:

[0084] S110 . Obtain an original facial image in RGB format, and convert the original facial image into an image of a Tensor object.

[0085] In this embodiment, the original facial image may be a facial image of a real person, or a facial image of a digital person, an animated character, or the like.

[0086] The Tensor refers to a tensor, which is actually expressed as a multidimensional array. The original facial image is converted into an image of a Tensor object to facilitate subsequent image processing.

[0087] In a preferred implementation, before converting the original facial image into an image of a Tensor object, the original facial image is preprocessed to improve the image quality of the original facial image.

[0088] Optionally, the preprocessing of the original facial image specifically includes:

[0089] Preprocessing the original facial image to obtain a preprocessed facial image;

[0090] The preprocessing includes one or more of filtering processing, brightness correction processing, and normalization processing;

[0091] It can be understood that filtering the original facial image can reduce noise and eliminate defects; brightness correction can improve image quality; and normalization can unify the pixel value range.

[0092] In an optional implementation, the filtering process may adopt bilateral filtering, and bilateral filtering is performed on the original facial image to eliminate small defects. In another optional implementation, the filtering process may also adopt other filtering processing methods, which are not limited here.

[0093] In an optional implementation, the brightness correction processing may be performed by performing gamma correction on the original facial image or the filtered image of the original facial image to preliminarily balance the brightness, thereby further improving the image quality.

[0094] In an optional implementation, the normalization process may be performed by normalizing the pixel values ​​of the original facial image or the image after the original facial image is filtered or the image after the original facial image is brightness corrected from [0, 255] to [0, 1].

[0095] Optionally, converting the original facial image into an image of a Tensor object specifically includes:

[0096] The preprocessed image of the original facial image is converted into an image of a Tensor object.

[0097] It can be understood that converting the preprocessed original facial image into an image of a Tensor object can represent data of any dimension, which is conducive to deep training.

[0098] In an optional implementation, the conversion may use the torchvision.transforms function of PyTorch to transform the original facial image, but the conversion method is not limited thereto.

[0099] S120, segmenting the image of the Tensor object according to different parts of the face to obtain an image segmentation result.

[0100] In this embodiment, the image of the Tensor object is segmented, and the segmentation methods mainly include threshold-based, edge detection and region-based methods and deep learning algorithm-based methods. Through segmentation, the required parts can be effectively extracted and irrelevant parts in the image can be shielded.

[0101] The different parts of the face may include a frontal area, a temporal area, an orbital area, a nose area, a facial skin area and other areas.

[0102] In an optional implementation, the image of the Tensor object is input into a face segmentation network for segmentation, and segmentation result images of different parts of the face can be obtained.

[0103] S130: Obtain at least two regional images from the image segmentation result, and superimpose the at least two regional images to obtain a mixed image.

[0104] The at least two area images may be a facial skin area and a nose, or a combination of a facial skin area and any of the five facial features.

[0105] In an optional embodiment, the image segmentation result is first binarized to obtain binary region images of different parts of the face, at least two region images are obtained from the binary region images corresponding to the image segmentation result, and the at least two region images obtained are mixed to obtain a mixed region image.

[0106] The at least two region images may be region images corresponding to the facial skin region and the nose region but are not limited to the two region images of the facial skin region and the nose region.

[0107] S140: The image of the Tensor object samples its color through a UV mapping relationship and maps it to a UV texture map.

[0108] In an optional implementation, the image of the Tensor object is processed before its color is sampled and mapped onto a UV texture map through a UV mapping relationship.

[0109] Optionally, the image of the Tensor object is processed by one or more of super-resolution and face restoration to obtain a third image;

[0110] The super-resolution refers to super-resolution technology, which is the process of improving the resolution of images or video frames through hardware or software methods, and obtaining high-resolution images through a series of low-resolution images. Commonly used methods include traditional interpolation methods and deep learning-based methods.

[0111] The face restoration includes traditional image restoration methods and methods based on generative adversarial networks. Traditional image restoration methods include but are not limited to diffusion-based and patch-based methods; methods based on generative adversarial networks include but are not limited to single-stage restoration, progressive image restoration and restoration based on prior knowledge.

[0112] In an optional implementation, the image of the Tensor object is super-resolved based on a deep learning algorithm, and the resolution and clarity of the image after being input into the super-resolved network are improved, which can not only achieve a good noise removal effect, but also retain the detailed information on the face. The super-resolved image is input into the GFPGAN network for face restoration to further eliminate defects, and the processing steps include but are not limited to this.

[0113] The color sampling of the image of the Tensor object through the UV mapping relationship and mapping it to the UV texture map specifically involves sampling the color of the third image through the UV mapping relationship and mapping it to the UV texture map;

[0114] In this embodiment, the UV mapping refers to the process of transferring the 3D mesh of the 3D model to the 2D space in order to further create textures for the model. By projecting the model, the mapping relationship between the 3D vertices and the image of the Tensor object or the third image element is obtained. Reading the UV coordinates can map the pixels, textures, details, etc. of the image. Since there are gaps between the coordinate points, the gap positions need to be smoothed to achieve the reproduction of pixels, textures, and details on the model.

[0115] As for the interpolation processing of UV coordinates, since there are many situations of UV coordinates, a common processing method is an interpolation method.

[0116] In an optional implementation, the image of the Tensor object or the third image is sampled and mapped to a UV texture map through UV mapping, the UV coordinates are read, and the mapping relationship between the 2D texture map and the 3D vertices is defined. Since the UV coordinates may not be integer pixel positions, a bilinear interpolation method is used to obtain a smoother texture mapping result. The image of the Tensor object or the third image is then used to map the vertices of the 3D grid to the 2D image plane through a camera projection model to obtain a mapping relationship between the 3D vertices and the pixels of the 2D original image. According to the mapping relationship between the 3D vertices and the image pixels of the Tensor object, the corresponding position is found on the UV texture map for sampling, and the sampling result is mapped to the UV texture map. The mapping can understand the position and proportion of the texture of the original facial image on the model in high detail, and multiple different textures are applied to the same model at the same time, thereby improving the efficiency of generating the texture of the head model. The texture sampling method is not limited to this.

[0117] S150, performing template replacement on the UV texture map according to a predefined template image, a mask image and the mixed region image to obtain a replaced UV texture map.

[0118] In an optional implementation, before performing template replacement on the UV texture map, the sampled and mapped UV texture map is first processed.

[0119] In this embodiment, the processing of the sampled and mapped UV texture image includes lighting balancing processing and / or repairing.

[0120] The illumination equalization processing includes histogram equalization, self-quotient image, phase map, illumination model-based method, etc.;

[0121] The restoration includes traditional image restoration methods and methods based on generative adversarial networks. Traditional image restoration methods include but are not limited to diffusion-based and patch-based methods; methods based on generative adversarial networks include but are not limited to single-stage restoration, progressive image restoration, and restoration based on prior knowledge.

[0122] In an optional implementation, a sliding window illumination equalization algorithm is used to perform illumination equalization processing on the texture image, and the texture image is transformed into the YUV space, and the Y channel thereof is taken out. According to the left-right symmetry of the texture image, the windows corresponding to the Y channel images of the left and right faces are sequentially taken out using the sliding window method. The brightness mean of the two windows is calculated, and the mixed brightness is obtained according to the ratio of light to dark of 9 to 1. The brightness of the two windows is adjusted to be close to the mixed brightness, and the aforementioned steps are repeated until the image is processed completely, so that the illumination conditions of the left and right parts of the face are consistent, and the illumination unevenness of the large-scale left and right range is eliminated. The image after illumination equalization processing is input into the face shadow segmentation network to obtain the small shadow area in the image, and the detected shadow area is filled with black, and the circumscribed rectangular frame of the shadow area is calculated at the same time, and the shadow area is inpainted using the image restoration algorithm, so as to remove the small shadow, retain the personalized features, and eliminate the shadow. The processing method includes but is not limited to this.

[0123] In this embodiment, the predefined template can quickly find the location of the target in the image.

[0124] The mask image can perform control area processing on the image. The opposite of the mask image is the anti-mask image.

[0125] In an optional embodiment, illustratively, Figure 3 A schematic diagram of a template replacement process of a three-dimensional head model texture generation method provided in an embodiment of the present application, wherein the step S140 performs template replacement on the UV texture map according to the predefined template image, the mask image and the mixed area image to obtain the replaced UV texture map, and the specific steps may include:

[0126] S151, multiplying the UV texture map with the mixed area image to obtain a UV texture map of a valid area of ​​the mixed area image; multiplying the predefined template image with the mixed area image to obtain a texture map of a valid area of ​​the template;

[0127] S152, calculating the superposition weight of the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template;

[0128] In an optional embodiment, the UV texture map is multiplied by the mixed area image to obtain a UV texture map of the effective area of ​​the mixed area image, which is denoted as A; the predefined template image is multiplied by the mixed area image to obtain a texture map of the effective area of ​​the template, which is denoted as B.

[0129] The superposition weight includes a first weight K and a second weight b; the first weight K and the second weight b are specifically calculated as follows:

[0130] Calculate the mean and variance of pixels in A and B respectively and record them as:

[0131] The pixel mean of A is M1, and the pixel variance is S1;

[0132] The pixel mean M2 and pixel variance S2 of B.

[0133] By (1)

[0134] (2)

[0135] The above two equations (1) and (2) can be used to solve the first weight K and the second weight b.

[0136] S153. Superimpose the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template according to the superposition weight to obtain a superimposed image; multiply the superimposed image with the predefined mask image to obtain a first image; and multiply the predefined template image with the inverse mask image corresponding to the predefined mask image to obtain a second image.

[0137] In this embodiment, the UV texture map of the effective area of ​​the mixed area image and the texture map pixels of the effective area of ​​the template are linearly superimposed. The UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template are superimposed according to the superposition weight to obtain a superimposed image, which is recorded as A".

[0138] In an optional embodiment, the superimposed image A":

[0139]

[0140] S154: Mix the first image and the second image to obtain a replaced UV texture map.

[0141] In this embodiment, the overlay image is multiplied by the predefined mask image to obtain a first image, which is denoted as C1; the predefined template image is multiplied by the inverse mask image corresponding to the predefined mask image to obtain a second image, which is denoted as C2; the replaced UV texture image is denoted as C.

[0142] In an optional implementation, the first image and the second image are mixed to obtain a replaced UV texture map, the replaced UV texture map C:

[0143]

[0144] The template of the UV texture map is replaced to reproduce the blocked part of the texture map, thereby improving the personalization and recognition of the model.

[0145] For example, Figure 4 A schematic diagram of the module flow of a method for generating texture of a three-dimensional head model provided in an embodiment of the present application.

[0146] Optionally, the 3D head model texture generating device 210 may include a conversion module 211, a segmentation module 212, an image overlay module 213, a mapping module 214, and a template replacement module 215, wherein:

[0147] The conversion module 211 is used to obtain an original facial image in RGB format and convert the original facial image into an image of a Tensor object;

[0148] In this embodiment, the conversion module 211 can be used to perform Figure 2 As shown in step S110 , for a detailed description of the conversion module 211 , reference may be made to the description of step S110 .

[0149] A segmentation module 212, configured to segment the image of the Tensor object according to different parts of the face to obtain an image segmentation result;

[0150] In this embodiment, the segmentation module 212 may be used to perform Figure 2 As shown in step S120, for the detailed description of the segmentation module 212, reference may be made to the description of step S120.

[0151] An image superposition module 213, configured to obtain at least two regional images from the image segmentation result, and superimpose the at least two regional images to obtain a mixed regional image;

[0152] In this embodiment, the image overlay module 213 may be used to perform Figure 2 As shown in step S130 , for a detailed description of the image overlay module 213 , reference may be made to the description of step S130 .

[0153] A mapping module 214 is used to sample the color of the image of the Tensor object through a UV mapping relationship and map it to a UV texture map;

[0154] In this embodiment, the mapping module 214 may be used to perform Figure 2 As shown in step S140, for the detailed description of the mapping module 214, reference may be made to the description of the step S140.

[0155] The template replacement module 215 is used to perform template replacement on the UV texture image according to a predefined template image, a mask image and the mixed area image to obtain a replaced UV texture image.

[0156] In this embodiment, the template replacement module 215 can be used to perform Figure 2 As shown in step S150, for the detailed description of the template replacement module 215, reference may be made to the description of step S150.

[0157] It can be understood that the above-mentioned device embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the device embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A three-dimensional head model texture generation device provided in an embodiment of the present application can execute a three-dimensional head model texture generation method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the three-dimensional head model texture generation device can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0158] Specifically, each step of the method embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor. The steps of the three-dimensional head model texture generation method in combination with the embodiment of the present application can be directly embodied as a hardware encoding processor, or a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0159] The present application embodiment provides an electronic device 310, whose structure is as follows: Figure 5 The electronic device 310 may be Figure 1 The server 100 or the terminal 200 is shown.

[0160] like Figure 5 As shown, the electronic device 310 includes a memory 311, a processor 312, a communication module 313 and an input / output interface 314, etc. Optionally, the memory 311, the processor 312, the communication module 313 and the input / output interface 314 can be connected and communicated through a bus 315.

[0161] The memory 311 is used to store one or more computer programs and transmit the code of the computer program to the processor 312; when the one or more computer programs are executed by the processor 312, the three-dimensional head model texture generation method in the embodiment of the present application is implemented.

[0162] Optionally, the electronic device 310 can be connected to a network via a communication module 313 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 310 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 310 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0163] Optionally, the electronic device 310 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 314. The electronic device 310 itself can have a display device, and can also be connected to other display devices through the input / output interface 314. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 314, so that data in the electronic device 310 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 311. It can be understood that the input / output interface 314 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 314 can be a component of the electronic device 310, or it can be an external device connected to the electronic device 310 when needed.

[0164] Optionally, the memory 311 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0165] Optionally, the computer program stored in the processor 312 may be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 310.

[0166] Optionally, the processor 312 may be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 312 executes various methods and processes of this embodiment, illustratively, such as a three-dimensional head model texture generation method in an embodiment of the present application.

[0167] Optionally, the bus 315 may include a path for transmitting information. The bus 315 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 315 may be divided into an address bus, a data bus, a control bus, etc.

[0168] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 311 of the electronic device 310. When the computer program is executed by the processor 312, one or more steps of a three-dimensional head model texture generation method of the embodiment of the present application can be performed.

[0169] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0170] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for generating texture of a three-dimensional head model, characterized in that: include: Obtain an original facial image in RGB format, and convert the original facial image into an image of a Tensor object; Segment the image of the Tensor object according to different parts of the face to obtain an image segmentation result; Acquire at least two regional images from the image segmentation result, and superimpose the at least two regional images to obtain a mixed regional image; The image of the Tensor object is sampled and mapped to the UV texture map through the UV mapping relationship; Performing template replacement on the UV texture map according to a predefined template image, a mask image and the mixed area image to obtain a replaced UV texture map; Binarizing the image segmentation result to obtain binary regional images of different parts of the face; The acquiring at least two regional images from the image segmentation result specifically includes: Acquire at least two region images from the binary region image corresponding to the image segmentation result, wherein the at least two region images include a facial skin region; Superimposing the at least two region images to obtain a mixed region image; The predefined template image, mask image and mixed area image perform template replacement on the UV texture map to obtain a replaced UV texture map, specifically comprising: Multiplying the UV texture map with the mixed area image to obtain a UV texture map of a valid area of ​​the mixed area image; Multiplying the predefined template image with the mixed area image to obtain a texture map of the template valid area; Calculate the superposition weight of the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template; Superimposing the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template according to the superposition weight to obtain a superimposed image; multiplying the superimposed image and the predefined mask image to obtain a first image; Multiplying the predefined template image and the inverse mask image corresponding to the predefined mask image to obtain a second image; The first image and the second image are mixed to obtain a replaced UV texture map.

2. A method for generating a 3D head model texture according to claim 1, characterized in that: The method further comprises: Preprocessing the original facial image to obtain a preprocessed facial image; The preprocessing includes one or more of filtering processing, brightness correction processing, and normalization processing; Convert the original facial image into an image of a Tensor object, specifically including: The preprocessed image of the original facial image is converted into an image of a Tensor object.

3. The method for generating a 3D head model texture according to claim 1, characterized in that: The image of the Tensor object is segmented according to different parts of the face to obtain image segmentation results, specifically including: The image of the Tensor object is input into the face segmentation network for segmentation to obtain an image segmentation result.

4. The method for generating a 3D head model texture according to claim 1, characterized in that: The color of the image of the Tensor object is sampled and mapped to the UV texture map through the UV mapping relationship, specifically including: The image of the Tensor object defines the mapping relationship between the 2D texture image and the 3D vertices through UV mapping; The image of the Tensor object obtains the mapping relationship between the 3D vertices and the image pixels of the Tensor object through the projection model; According to the mapping relationship between the 3D vertices and the image pixels of the Tensor object, the corresponding positions are found on the UV texture map for sampling, and the sampling results are mapped to the UV texture map.

5. The method for generating a 3D head model texture according to claim 1, characterized in that: Process the sampled and mapped UV texture map, including: Perform lighting balance on the sampled and mapped UV texture map; Input the UV texture image after illumination equalization into the facial shadow segmentation network to obtain the small shadow areas in the image; Performing a repair operation on a small shadow area in the acquired image to obtain a UV texture map after the shadow is eliminated; The step of performing template replacement on the UV texture map according to the predefined template image, the mask image and the mixed area image to obtain the replaced UV texture map is specifically as follows: The UV texture image after the shadow is eliminated is subjected to template replacement according to the predefined template image, the mask image and the mixed area image to obtain the replaced UV texture image.

6. A method for generating a 3D head model texture according to claim 2, characterized in that: The image of the Tensor object is processed as follows: Performing one or more of super-resolution and face restoration on the image of the Tensor object to obtain a third image; The color of the image of the Tensor object is sampled and mapped to the UV texture map through the UV mapping relationship, specifically: the color of the third image is sampled and mapped to the UV texture map through the UV mapping relationship; And / or, processing the replaced UV texture map to obtain a processed image, specifically including: The replaced UV texture map is input into the GFPGAN network to perform face repair to obtain a repaired UV texture map.

7. A three-dimensional head model texture generation device, characterized in that: include: A conversion module, used to obtain an original facial image in RGB format, and convert the original facial image into an image of a Tensor object; A segmentation module, used to segment the image of the Tensor object according to different parts of the face to obtain an image segmentation result; An image superposition module, used for obtaining at least two regional images from the image segmentation result, and superimposing the at least two regional images to obtain a mixed regional image; Binarizing the image segmentation result to obtain binary regional images of different parts of the face; The acquiring at least two regional images from the image segmentation result specifically includes: Acquire at least two region images from the binary region image corresponding to the image segmentation result, wherein the at least two region images include a facial skin region; Superimposing the at least two region images to obtain a mixed region image; A mapping module, used to sample the color of the image of the Tensor object through a UV mapping relationship and map it to a UV texture map; A template replacement module, used for performing template replacement on the UV texture image according to a predefined template image, a mask image and the mixed area image to obtain a replaced UV texture image; The predefined template image, mask image and mixed area image perform template replacement on the UV texture map to obtain a replaced UV texture map, specifically comprising: Multiplying the UV texture map with the mixed area image to obtain a UV texture map of a valid area of ​​the mixed area image; Multiplying the predefined template image with the mixed area image to obtain a texture map of the template valid area; Calculate the superposition weight of the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template; Superimposing the UV texture map of the effective area of ​​the mixed area image and the texture map of the effective area of ​​the template according to the superposition weight to obtain a superimposed image; multiplying the superimposed image and the predefined mask image to obtain a first image; Multiplying the predefined template image and the inverse mask image corresponding to the predefined mask image to obtain a second image; The first image and the second image are mixed to obtain a replaced UV texture map.

8. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the three-dimensional head model texture generation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the three-dimensional head model texture generation method according to any one of claims 1 to 8 when executed.

Images