Map generation method, device, electronic device and readable storage medium

Abstract

The present invention relates to a map generation method, device, electronic device and readable storage medium. The method includes the following steps: generating a second face image with uneven illumination based on a first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model; receiving a third face image and generating a third texture map corresponding to the third face image; obtaining the trained illumination balance model, and using the third face image and the third texture map as inputs of the trained illumination balance model; running the trained illumination balance model to obtain a target texture map with balanced illumination. The third texture map is subjected to illumination correction through the trained illumination balance model, so that a target texture map with balanced illumination can be obtained, thereby reducing the influence of ambient illumination on the face image, and then a virtual image with good display can be generated based on the target texture map.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a texture generation method, device, electronic device and readable storage medium. Background Art

[0002] With the development of artificial intelligence, VR (Virtual Reality), AR (Augmented Reality) and other technologies, more and more application software supports users to create personalized virtual images, which brings users more diverse and immersive entertainment and social methods. Among them, an indispensable technology is 3D head reconstruction; in 3D head reconstruction, a texture map corresponding to the user's face is usually generated, and then the corresponding virtual image of the user is obtained based on the texture map; however, the texture map obtained in this way is easily affected by the ambient light, and the generated texture usually has uneven lighting such as shadows and highlights, which makes the virtual image display abnormal. Summary of the invention

[0003] The present invention provides a texture generation method, device, electronic device and readable storage medium, aiming to solve the technical problem in the prior art that the texture map is affected by the ambient light, resulting in abnormal display of the virtual image.

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides a map generation method, which comprises the steps of:

[0005] Generate a second face image with uneven illumination based on the first face image with balanced illumination, and train an initial illumination balance model using the first face image and the second face image to obtain a trained illumination balance model;

[0006] Receiving a third facial image and generating a third texture map corresponding to the third facial image;

[0007] Acquire a trained illumination balance model, and use the third face image and the third texture map as inputs of the trained illumination balance model;

[0008] The trained illumination balance model is run to obtain an illumination-balanced target texture map.

[0009] Optionally, the step of generating a second face image with uneven illumination based on the first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model includes:

[0010] Acquire a first face image with balanced lighting from a database, and perform a relighting operation on the first face image to obtain a second face image with uneven lighting;

[0011] Generate a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and construct a training sample using the first texture map, the second face image and the second texture map;

[0012] An initial illumination balance model is obtained, and the initial illumination balance model is trained using the training samples to obtain a trained illumination balance model.

[0013] Optionally, the step of acquiring the first face image with balanced illumination from the database includes:

[0014] For each fourth face image in the database, calculating a brightness mean value of the fourth face image;

[0015] Determine whether the brightness mean is greater than a preset brightness threshold;

[0016] If the brightness mean is greater than a preset brightness threshold, the fourth face image corresponding to the brightness mean is used as the first face image.

[0017] Optionally, the step of calculating the brightness mean of the fourth face image includes:

[0018] Calculating the brightness value corresponding to each pixel in the fourth face image respectively;

[0019] The brightness mean is obtained by calculating the average value of each brightness value.

[0020] Optionally, the step of calculating the brightness mean of the fourth face image includes:

[0021] Calculating the brightness value corresponding to each pixel in the fourth face image respectively;

[0022] The standard deviation of each brightness value is calculated to obtain the brightness mean value.

[0023] Optionally, the step of training the initial illumination balance model using the training samples includes:

[0024] Using the second face image and the second texture map as inputs of the initial illumination balance model, and obtaining a fourth texture map through the initial illumination balance model;

[0025] Obtain a preset loss function, and bring the first texture map and the fourth texture map into the preset loss function to calculate the adversarial loss;

[0026] The initial illumination balance model is optimized through the adversarial loss.

[0027] Optionally, the step of generating a third texture map corresponding to the third facial image includes:

[0028] Performing a reconstruction operation on the third facial image to obtain a first three-dimensional facial model corresponding to the third facial image;

[0029] The third texture map is obtained by performing two-dimensional mapping on the pixels in the first three-dimensional face model.

[0030] To achieve the above object, the present invention further provides a map generation device, the map generation device comprising:

[0031] A first training module is used to generate a second face image with uneven illumination based on the first face image with balanced illumination, and train an initial illumination balance model using the first face image and the second face image to obtain a trained illumination balance model;

[0032] A first generating module, configured to receive a third facial image and generate a third texture map corresponding to the third facial image;

[0033] A first acquisition module is used to acquire a trained illumination balance model, and use the third face image and the third texture map as inputs of the trained illumination balance model;

[0034] The first execution module is used to run the trained illumination balance model to obtain an illumination balanced target texture map.

[0035] Optionally, the first training module includes:

[0036] A first acquisition submodule is used to acquire a first face image with balanced illumination from a database, and to perform a relighting operation on the first face image to obtain a second face image with uneven illumination;

[0037] A first generating submodule, configured to generate a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and construct a training sample using the first texture map, the second face image and the second texture map;

[0038] The first training submodule is used to obtain an initial illumination balance model, and train the initial illumination balance model through the training samples to obtain a trained illumination balance model.

[0039] Optionally, the first acquisition submodule includes:

[0040] A first calculation unit, configured to calculate, for each fourth face image in the database, a brightness mean of the fourth face image;

[0041] A first judging unit, used to judge whether the brightness mean is greater than a preset brightness threshold;

[0042] The first execution unit is configured to use the fourth face image corresponding to the brightness mean as the first face image if the brightness mean is greater than a preset brightness threshold.

[0043] Optionally, the first computing unit includes:

[0044] A first calculation subunit, used to respectively calculate the brightness value corresponding to each pixel point in the fourth face image;

[0045] The second calculation subunit is used to calculate the average value of each brightness value to obtain the brightness mean value.

[0046] Optionally, the first computing unit includes:

[0047] A fourth calculation subunit, used to respectively calculate the brightness value corresponding to each pixel point in the fourth face image;

[0048] The fifth calculation subunit calculates the standard deviation of each brightness value to obtain the brightness mean value.

[0049] Optionally, the first training submodule includes:

[0050] a second execution unit, configured to use the second face image and the second texture map as inputs of the initial illumination balance model, and obtain a fourth texture map through the initial illumination balance model;

[0051] A second calculation unit is used to obtain a preset loss function, and bring the first texture map and the fourth texture map into the preset loss function to calculate the adversarial loss;

[0052] The second execution unit is used to optimize the initial illumination balance model through the adversarial loss.

[0053] Optionally, the first generating module includes:

[0054] A first execution submodule, configured to perform a reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;

[0055] The second execution submodule is used to perform two-dimensional mapping on the pixel points in the first three-dimensional face model to obtain the third texture map.

[0056] To achieve the above objectives, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned texture generation method when executed by the processor.

[0057] To achieve the above object, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned texture generation method are implemented.

[0058] The present invention proposes a map generation method, device, electronic device and readable storage medium, which generates a second face image with uneven illumination based on a first face image with balanced illumination, and trains an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model; receives a third face image, and generates a third texture map corresponding to the third face image; obtains the trained illumination balance model, and uses the third face image and the third texture map as inputs of the trained illumination balance model; runs the trained illumination balance model to obtain a target texture map with balanced illumination. The third texture map corresponding to the third face image is subjected to illumination correction through the trained illumination balance model, so that a target texture map with balanced illumination can be obtained, thereby reducing the influence of ambient illumination on the face image, and then a virtual image with good display can be generated based on the target texture map. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0061] Figure 1 It is a flowchart of a first embodiment of a method for generating a texture according to the present invention;

[0062] Figure 2 This is a schematic diagram of the model training process in the second embodiment of the map generation method of the present invention;

[0063] Figure 3 It is a structural schematic diagram of the illumination balance model in the map generation method of the present invention;

[0064] Figure 4 A schematic diagram of the selection process of the first face image in the texture generation method of the present invention;

[0065] Figure 5 It is a schematic diagram of the module structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0067] The technical solutions in the embodiments are described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0068] The present invention provides a method for generating a map, referring to Figure 1 , Figure 1 This is a flow chart of a first embodiment of a method for generating a texture according to the present invention, wherein the method comprises the following steps:

[0069] Step S10, generating a second face image with uneven illumination based on the first face image with balanced illumination, and training an initial illumination balance model using the first face image and the second face image to obtain a trained illumination balance model;

[0070] Lighting balance means that the difference in brightness between the various parts of a facial image is small. Parameters can be set according to the actual application scenario to determine whether the facial image is lighting balanced, such as the average value or standard deviation of each pixel in the facial image. For specific settings, please refer to the subsequent embodiments. The first facial image is a pre-selected facial image that meets the lighting balance requirements; the lighting parameters of the first facial image are adjusted so that the adjusted first facial image does not meet the lighting balance requirements, thereby obtaining a second facial image. The initial lighting balance model is trained by the first facial image and the second facial image, so that the lighting balance model has the ability to convert facial images with uneven lighting into facial images with balanced lighting.

[0071] Step S20, receiving a third face image, and generating a third texture map corresponding to the third face image;

[0072] The third face image can be an image selected and input by the user, or an image collected in real time by an image acquisition device such as a camera; it is understandable that the input or collected image usually includes environmental information in addition to the face, so it is necessary to perform a face recognition operation on the image to extract pixels containing only the face from the image, and then generate the third face image based on the pixels containing only the face. The specific face recognition method can be set according to the actual application scenario and needs, and is not limited here.

[0073] The texture generation method of this embodiment is applied to the establishment of a virtual image; the virtual image is obtained by creating a three-dimensional model of the character and covering the surface of the three-dimensional model with a texture; since the surface of the three-dimensional model of the character is three-dimensional and the third face image is flat, it is necessary to convert the third face image into a third texture map and then cover it on the surface of the three-dimensional model of the human body to obtain the virtual image.

[0074] The step S20 of generating the third texture map corresponding to the third face image comprises the following steps:

[0075] Step S21, performing a reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;

[0076] Step S22: Perform two-dimensional mapping on the pixels in the first three-dimensional face model to obtain the third texture map.

[0077] The reconstruction operation refers to the operation of converting a planar face image into a three-dimensional face model. It should be noted that the specific reconstruction operation can be selected according to the actual application scenario and needs. For example, in this embodiment, a third face image is reconstructed by a 3DMM (3DMorphable Model), such as a BFM (Basel Face Model); specifically, a three-dimensional deformable model coefficient regression neural network is obtained, and the three-dimensional deformable model coefficient regression neural network can be selected from existing public models; the third face image is input into the three-dimensional deformable model coefficient regression neural network to obtain the predicted BFM coefficient, wherein: the BFM coefficient includes (c i , c e , c t , p,γ)∈R 257 , where R 257 is a 257-dimensional vector, i.e., it includes 257 unit data; c i is the identity coefficient, c i ∈R 80 , that is, the identity coefficient corresponds to an 80-dimensional vector, including 80 unit data; c e is the expression coefficient, c e ∈R64 , that is, the expression coefficient corresponds to a 64-dimensional vector, including 64 unit data; c t is the texture mapping coefficient, c t ∈R 80 , that is, the texture mapping coefficient corresponds to an 80-dimensional vector, including 80 unit data; p is the head posture coefficient, p∈R 6 , that is, the head posture coefficient corresponds to a 6-dimensional vector, including 6 unit data; γ is the illumination coefficient, γ∈R 27 , that is, the illumination coefficient corresponds to a 27-dimensional vector, including 27 unit data; it should be noted that the specific BFM coefficient can be set according to the actual application model and needs, and the above description only provides an optional solution; the obtained BFM coefficient is input into the BFM geometric shape formula S; the specific description is:

[0078]

[0079] in, is the average shape of BFM, B id is the identity base of BFM, B exp It is the expression base of BFM; B id and B exp All are preset parameters of the geometric shape formula S. The first three-dimensional face model can be obtained through the geometric shape formula.

[0080] Two-dimensional mapping refers to the operation of mapping points in a three-dimensional model to a two-dimensional plane. It should be noted that the specific mapping method can be selected according to the actual application scenario and needs. For example, in this embodiment, the first three-dimensional face model is mapped in two dimensions by texture sampling TextureSampling to obtain the third texture map. Specifically, the formula is described as:

[0081] V 2d =Pr·(R·+t 3d )

[0082] Where: V 2d is the coordinate of the point on the two-dimensional plane mapped by the first three-dimensional face model; Pr is the orthogonal projection matrix, i.e., the camera coefficient; t 3d is the head posture coefficient, which is converted from p. 2d Afterwards, by comparing the first 3D face model with V 2d The corresponding position is sampled, and the set of sampled pixels is used as the third texture map; specifically, the formula is described as:

[0083] T=F(V 2d )

[0084] Wherein, T is the third texture map, and F is the sampling function; it should be noted that the sampling function can be selected and set according to the actual application scenario and needs, and is not limited here. The third texture map corresponding to the third face image can be obtained through the above steps.

[0085] Step S30, obtaining a trained illumination balance model, and using the third face image and the third texture map as inputs of the trained illumination balance model;

[0086] Step S40, running the trained illumination balance model to obtain an illumination balanced target texture map.

[0087] The illumination balance model is used to balance the third texture map to convert the third texture map with uneven illumination into a target texture map with uniform illumination. Specifically, the illumination balance model in this embodiment adopts a generative adversarial network model. The generative adversarial network model is a deep learning model. It should be noted that the model can also be selected based on actual application scenarios and needs.

[0088] This embodiment performs illumination correction on the third texture map corresponding to the third facial image through the trained illumination balance model, so that a target texture map with balanced illumination can be obtained, thereby reducing the influence of ambient illumination on the facial image, and further generating a virtual image with good display based on the target texture map.

[0089] Further, see Figure 2 In the second embodiment of the map generation method of the present invention proposed based on the first embodiment of the present invention, the step S10 includes the steps of:

[0090] Step S11, obtaining a first face image with balanced illumination from a database, and performing a relighting operation on the first face image to obtain a second face image with uneven illumination;

[0091] There are multiple face images stored in the database, namely the fourth face image; the fourth face image can be obtained in a variety of ways, such as obtaining from the network or server, or collecting through an image acquisition device, etc. The relighting operation is to perform a superposition operation of simulated lighting on the image so that the image has lighting characteristics; it should be noted that the specific relighting operation method can be selected according to the actual application scenario and needs, such as the DPR (Deep Single Image Portrait Relighting, single image portrait relighting) neural network is used to perform the relighting operation; specifically, multiple groups of lighting parameters are preset, and the lighting parameters include different lighting parameters, such as direction, brightness, etc.; when the relighting operation is required, a group of lighting parameters is randomly selected, and the selected lighting parameters and the first face image are used as the input of the DPR neural network, and the DPR neural network outputs a second face image superimposed with the lighting parameters based on the lighting parameters and the first face image.

[0092] Step S12, generating a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and constructing a training sample using the first texture map, the second face image and the second texture map;

[0093] The generation method of the first texture map and the second texture map can be performed by analogy with the generation method of the third texture map, which will not be described in detail here. It should be noted that since the first face image and the second face image differ only in terms of lighting characteristics, the reconstruction results of the first face image and the second face image should be the same, so only the first face image can be reconstructed, and then the first texture map can be obtained based on the second three-dimensional face model obtained by the reconstruction operation; at the same time, the second face image is referenced to the second three-dimensional face model to obtain the second texture map.

[0094] It should be noted that a group of training samples includes a first texture map, a second face image and a second texture map. There are multiple training samples, and a single training sample is used to train the initial illumination balance model once.

[0095] Step S13, obtaining an initial illumination balance model, and training the initial illumination balance model using the training samples to obtain a trained illumination balance model.

[0096] See also Figure 3, the illumination balance model in this embodiment is a generative adversarial network model, which includes a generator and a discriminator, wherein the generator is composed of two encoders and a decoder, the encoder includes an image encoder and a texture encoder, the image encoder is used to extract feature information of the face image, the texture encoder is used to extract feature information of the texture map, the feature information extracted by the image encoder and the texture encoder is cascaded through the feature channel and input into the decoder to obtain the texture map; the generator in this embodiment adopts the U-Net structure, and the discriminator adopts the Patch-GAN structure. During the training process, the generator is used to generate a predicted texture map, i.e., the fourth texture map, according to the second face image with uneven illumination and the corresponding second texture map; the discriminator is used to compare the fourth texture map with the first texture map to correct the deviation; during the use process after the training is completed, the third face image and the third texture map are input into the generator to obtain the target texture map.

[0097] In this embodiment, a first face image with balanced lighting is selected, and a second face image with uneven lighting is obtained through a re-lighting operation; then, an initial lighting balance model is trained based on the first face image with balanced lighting and the second face image with uneven lighting to obtain a trained lighting balance model.

[0098] Further, see Figure 4 In the third embodiment of the texture generation method of the present invention proposed based on the second embodiment of the present invention, the step S40 includes the steps of:

[0099] Step S111, for each fourth face image in the database, calculating the brightness mean of the fourth face image;

[0100] It should be noted that the selection of the first facial image can be carried out in the image acquisition stage, that is, whenever a facial image is acquired, it is judged whether its brightness mean is greater than the preset brightness threshold. Only when the brightness mean is greater than the preset brightness threshold is it stored in the database. At this time, all facial images in the database are first facial images; it can also be carried out in the training sample acquisition stage, that is, various types of facial images are stored in the database. When training samples are needed, the fourth facial image with a brightness mean greater than the preset brightness threshold is selected from the database as the first facial image.

[0101] The brightness mean is used to reflect the illumination uniformity of the face image. First, the brightness value corresponding to each pixel in the fourth face image is obtained, and then the mean operation is performed on all the brightness values ​​to obtain the brightness mean corresponding to the fourth face image. The mean operation can be selected according to the actual application scenario and needs. For example, in this embodiment, the mean value and the standard deviation are respectively used as the mean operation. Specifically, the mean value is taken as an example. The formula is described as:

[0102]

[0103] Where Lum is the average brightness; i is the pixel number; I i r is the color value corresponding to the red channel of the i-th pixel in the RGB color mode; I i g is the color value corresponding to the green channel of the i-th pixel in the RGB color mode; I i b is the color value corresponding to the blue channel of the i-th pixel in the RGB color mode; M is the number of pixels in the fourth face image.

[0104] The square root part represents the brightness value corresponding to the i-th pixel. It can also be replaced by other brightness value calculation formulas according to actual application needs, such as:

[0105]

[0106] At this time, the formula for the average brightness is described as:

[0107]

[0108] The method for calculating the brightness value of a single pixel is as described above, and the mean operation of the standard deviation can be performed in a similar manner, which will not be repeated here. The average value will be used for explanation in subsequent embodiments, and the standard deviation scheme can be performed in a similar manner, which will not be repeated here.

[0109] Step S112, determining whether the brightness mean is greater than a preset brightness threshold;

[0110] Step S113: If the brightness mean is greater than a preset brightness threshold, the fourth face image corresponding to the brightness mean is used as the first face image.

[0111] The preset brightness threshold is a value used to characterize whether the face image is illuminated evenly; the specific preset brightness threshold can be set according to the actual application scenario and needs, such as the environment; in addition, for ease of comparison, the brightness mean is further normalized. From the aforementioned average value calculation formula, it can be seen that the range of the brightness mean is 0 to 255. The brightness mean is divided by 255 so that the range of the brightness mean is between 0 and 1, and then judged by the preset brightness threshold; for example, in this embodiment, the preset brightness threshold is set to 0.98; it can be understood that the preset brightness threshold can also be directly set with 0 to 255 as the range of the brightness mean.

[0112] When the brightness mean is greater than the preset brightness threshold, the fourth face image is considered to be a face image with balanced lighting, and the fourth face image is used as the first face image. When the brightness mean is less than or equal to the preset brightness threshold, the fourth face image is considered to be a face image with uneven lighting, and the fourth face image is filtered.

[0113] This embodiment can accurately select the fourth facial image with balanced illumination as the first facial image.

[0114] Further, in a fourth embodiment of the map generation method of the present invention proposed based on the second embodiment of the present invention, step S13 includes the steps of:

[0115] Step S131, using the second face image and the second texture map as inputs of the initial illumination balance model, and obtaining a fourth texture map through the initial illumination balance model;

[0116] Step S132, obtaining a preset loss function, and bringing the first texture map and the fourth texture map into the preset loss function to calculate the adversarial loss;

[0117] Step S133: optimizing the initial illumination balance model by using the adversarial loss.

[0118] The fourth texture map is a lighting balanced texture map predicted by the initial lighting balanced model; the first texture map is a lighting balanced texture map corresponding to the second texture map; that is, the first texture map has been determined to be lighting balanced, and therefore, the first texture map can be used as a reference to determine whether the fourth texture map is lighting balanced.

[0119] The loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. Specifically, the loss function can be set according to the actual application scenario and needs; in this embodiment, a combination of multiple loss functions is used to optimize the initial illumination balance model; specifically, the loss function includes a pixel loss function, a perceptual loss function, a style loss function, and an adversarial loss function; among which:

[0120] The pixel loss function formula is described as follows:

[0121]

[0122] Among them, L rec () is the pixel loss function; T pred is the fourth texture map; T1 is the first texture map; N is the number of pixels in the fourth texture map or the first texture map; i is the sequence number of the pixel.

[0123] Since the color values ​​between the first texture map and the fourth texture map should be similar, a pixel loss function is introduced to measure the pixel-level difference between the first texture map and the fourth texture map, which is specifically expressed as the L1 distance between the pixels corresponding to the first texture map and the fourth texture map, that is, the Manhattan distance.

[0124] The pixel loss function alone cannot generate high-frequency information of the texture map, such as image content, spatial structure, etc.; in other words, the pixel loss function cannot measure the perceptual difference between the first texture map and the fourth texture map. Therefore, the perceptual loss function is introduced to make the fourth texture map have high-level semantic features similar to the first texture map, so that the similarity of images can be measured more robustly when training the illumination balance model. The perceptual loss function extracts the convolution features of each layer of the first texture map and the fourth texture map through the VGG19 neural network, and expresses the similarity between the convolution features of each layer as the L1 distance between the first texture map and the fourth texture map; the perceptual loss function formula is described as follows:

[0125]

[0126] Among them, L perc () is the perceptual loss function; φ i is the i-th layer convolution feature of the VGG19 neural network.

[0127] In order to further ensure the uniformity of the style between the first texture map and the fourth texture map and avoid large style differences between the first texture map and the fourth texture map, such as color, texture, etc., a style loss function is introduced. The style loss function is used to measure the difference in style between the first texture map and the fourth texture map, which is specifically expressed as the L1 distance of the Gram matrix of each layer of convolution features between the first texture map and the fourth texture map; the style loss function formula is described as follows:

[0128]

[0129] In order to generate high-quality stylized texture maps, the first texture map and the fourth texture map are input into the discriminator respectively, and the resulting adversarial loss function is:

[0130]

[0131] Among them, L gen () is the generated adversarial loss function; D() is the confidence.

[0132]

[0133] Among them, L dis () is the discriminative adversarial loss function.

[0134] It should be noted that the above loss function is only an illustration of a feasible solution. Other loss functions or function combinations can be selected to optimize the illumination balance model according to actual application scenarios and needs.

[0135] It should be noted that when training the initial illumination balance model, it is necessary to set training completion conditions, such as the number of training times or the reliability of the model, etc. When the initial illumination balance model meets the training completion conditions, the initial illumination balance model is used as the trained illumination balance model. It is understandable that the trained illumination balance model can also be self-optimized through the actual processed texture maps and face images during actual use.

[0136] This embodiment can optimize the initial illumination balance model by reasonably setting the loss function to obtain a trained illumination balance model.

[0137] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0139] The present invention also provides a texture generation device for implementing the texture generation method, the texture generation device comprising:

[0140] A first training module is used to generate a second face image with uneven illumination based on the first face image with balanced illumination, and train an initial illumination balance model using the first face image and the second face image to obtain a trained illumination balance model;

[0141] A first generating module, configured to receive a third facial image and generate a third texture map corresponding to the third facial image;

[0142] A first acquisition module is used to acquire a trained illumination balance model, and use the third face image and the third texture map as inputs of the trained illumination balance model;

[0143] The first execution module is used to run the trained illumination balance model to obtain an illumination balanced target texture map.

[0144] The map generation device performs illumination correction on the third texture map corresponding to the third facial image through the trained illumination balance model, so as to obtain an illumination-balanced target texture map, thereby reducing the influence of ambient light on the facial image, and further generating a virtual image with good display based on the target texture map.

[0145] It should be noted that the first training module in this embodiment can be used to execute step S10 in the embodiment of the present invention, the first generation module in this embodiment can be used to execute step S20 in the embodiment of the present invention, the first acquisition module in this embodiment can be used to execute step S30 in the embodiment of the present invention, and the first execution module in this embodiment can be used to execute step S40 in the embodiment of the present invention.

[0146] Furthermore, the first training module includes:

[0147] A first acquisition submodule is used to acquire a first face image with balanced illumination from a database, and to perform a relighting operation on the first face image to obtain a second face image with uneven illumination;

[0148] A first generating submodule, configured to generate a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and construct a training sample using the first texture map, the second face image and the second texture map;

[0149] The first training submodule is used to obtain an initial illumination balance model, and train the initial illumination balance model through the training samples to obtain a trained illumination balance model.

[0150] Furthermore, the first acquisition submodule includes:

[0151] A first calculation unit, configured to calculate, for each fourth face image in the database, a brightness mean of the fourth face image;

[0152] A first judging unit, used to judge whether the brightness mean is greater than a preset brightness threshold;

[0153] The first execution unit is configured to use the fourth face image corresponding to the brightness mean as the first face image if the brightness mean is greater than a preset brightness threshold.

[0154] Furthermore, the first computing unit includes:

[0155] A first calculation subunit, used to respectively calculate the brightness value corresponding to each pixel point in the fourth face image;

[0156] The second calculation subunit is used to calculate the average value of each brightness value to obtain the brightness mean value.

[0157] Furthermore, the first computing unit includes:

[0158] A fourth calculation subunit, used to respectively calculate the brightness value corresponding to each pixel point in the fourth face image;

[0159] The fifth calculation subunit calculates the standard deviation of each brightness value to obtain the brightness mean value.

[0160] Furthermore, the first training module includes:

[0161] a second execution unit, configured to use the second face image and the second texture map as inputs of the initial illumination balance model, and obtain a fourth texture map through the initial illumination balance model;

[0162] A second calculation unit is used to obtain a preset loss function, and bring the first texture map and the fourth texture map into the preset loss function to calculate the adversarial loss;

[0163] The second execution unit is used to optimize the initial illumination balance model through the adversarial loss.

[0164] Furthermore, the first generating module includes:

[0165] A first execution subunit, configured to perform a reconstruction operation on the third face image to obtain a first three-dimensional face model corresponding to the third face image;

[0166] The second execution subunit is used to perform two-dimensional mapping on the pixels in the first three-dimensional face model to obtain the third texture map.

[0167] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0168] Reference Figure 5 In terms of hardware structure, the electronic device may include components such as a communication module 10, a memory 20, and a processor 30. In the electronic device, the processor 30 is connected to the memory 20 and the communication module 10 respectively, and a computer program is stored in the memory 20. The computer program is executed by the processor 30 at the same time. When the computer program is executed, the steps of the above method embodiment are implemented.

[0169] The communication module 10 can be connected to an external communication device through a network. The communication module 10 can receive requests from an external communication device, and can also send requests, instructions and information to the external communication device, which can be other electronic devices, servers or Internet of Things devices, such as televisions, etc.

[0170] The memory 20 can be used to store software programs and various data. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as generating a third texture map corresponding to the third face image), etc.; the data storage area can include a database, and the data storage area can store data or information created according to the use of the system, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0171] The processor 30 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, so as to monitor the electronic device as a whole. The processor 30 may include one or more processing units; optionally, the processor 30 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 30.

[0172] although Figure 5 Although not shown, the electronic device may further include a circuit control module, which is used to connect to a power source to ensure the normal operation of other components. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0173] The present invention also provides a computer-readable storage medium on which a computer program is stored. The computer-readable storage medium may be Figure 5 The memory 20 in the electronic device may also be at least one of a ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, and an optical disk. The computer-readable storage medium includes a number of instructions for enabling a terminal device with a processor (which may be a television, a car, a mobile phone, a computer, a server, a terminal, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0174] In the present invention, the terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0176] Although the embodiments of the present invention have been shown and described above, the scope of protection of the present invention is not limited thereto. It is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify and replace the above embodiments within the scope of the present invention, and these changes, modifications and replacements should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for generating a texture, characterized in that: The method comprises: Generate a second face image with uneven illumination based on the first face image with balanced illumination, and train an initial illumination balance model using the first face image and the second face image to obtain a trained illumination balance model; Receiving a third facial image and generating a third texture map corresponding to the third facial image; Acquire the trained illumination balance model, and use the third face image and the third texture map as inputs of the trained illumination balance model; The trained illumination balance model is run to obtain an illumination-balanced target texture map.

2. The method for generating a texture according to claim 1, wherein: The steps of generating a second face image with uneven illumination based on the first face image with balanced illumination, and training an initial illumination balance model through the first face image and the second face image to obtain a trained illumination balance model include: Acquire a first face image with balanced lighting from a database, and perform a relighting operation on the first face image to obtain a second face image with uneven lighting; Generate a first texture map corresponding to the first face image and a second texture map corresponding to the second face image, and construct a training sample using the first texture map, the second face image and the second texture map; An initial illumination balance model is obtained, and the initial illumination balance model is trained using the training samples to obtain a trained illumination balance model.

3. The method for generating a texture according to claim 2, wherein: The step of acquiring a first face image with balanced illumination from a database comprises: For each fourth face image in the database, calculating a brightness mean value of the fourth face image; Determine whether the brightness mean is greater than a preset brightness threshold; If the brightness mean is greater than a preset brightness threshold, the fourth face image corresponding to the brightness mean is used as the first face image.

4. The method for generating a texture according to claim 3, wherein: The step of calculating the brightness mean of the fourth face image comprises: Calculating the brightness value corresponding to each pixel in the fourth face image respectively; The brightness mean is obtained by calculating the average value of each brightness value.

5. The method for generating a texture according to claim 3, wherein: The step of calculating the brightness mean of the fourth face image comprises: Calculating the brightness value corresponding to each pixel in the fourth face image respectively; The standard deviation of each brightness value is calculated to obtain the brightness mean value.

6. The method for generating a texture according to claim 2, wherein: The step of training the initial illumination balance model by using the training samples comprises: Using the second face image and the second texture map as inputs of the initial illumination balance model, and obtaining a fourth texture map through the initial illumination balance model; Obtain a preset loss function, and bring the first texture map and the fourth texture map into the preset loss function to calculate the adversarial loss; The initial illumination balance model is optimized through the adversarial loss.

7. The method for generating a texture according to claim 1, wherein: The step of generating a third texture map corresponding to the third face image comprises: Performing a reconstruction operation on the third facial image to obtain a first three-dimensional facial model corresponding to the third facial image; The third texture map is obtained by performing two-dimensional mapping on the pixels in the first three-dimensional face model.

8. A map generating device, characterized in that: The map generating device comprises: A first training module is used to generate a second face image with uneven illumination based on the first face image with balanced illumination, and train an initial illumination balance model using the first face image and the second face image to obtain a trained illumination balance model; A first generating module, configured to receive a third facial image and generate a third texture map corresponding to the third facial image; A first acquisition module is used to acquire a trained illumination balance model, and use the third face image and the third texture map as inputs of the trained illumination balance model; The first execution module is used to run the trained illumination balance model to obtain an illumination balanced target texture map.

9. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the map generation method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the texture generation method according to any one of claims 1 to 7 are implemented.

Images