Face data enhancement method and device, computer device, and storage medium

Abstract

The application relates to a face data enhancement method, acquires a face 2D image, performs three-dimensional reconstruction on the face 2D image, obtains a face three-dimensional model, performs texture fitting and illumination rendering on the face three-dimensional model, and the illumination rendering is performed based on a preset virtual scene light source in the process, the preset virtual scene light source can set illumination intensity and illumination direction according to actual needs, so that the result of the illumination rendering is adapted to different real application scenes, a face three-dimensional model adapted to a rich scene is obtained, the rendered face three-dimensional model is re-projected to the face 2D image, a face region and a background region carried in the face 2D image are fused, and an enhanced face 2D image is obtained, the face 2D image carries rich and diverse face data, face training data set can be effectively expanded, and accurate face three-dimensional reconstruction can be supported.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face recognition, in particular to a face data enhancement method and device, computer equipment and a storage medium. BACKGROUND

[0002] Face, as one of the most important biometric features, covers rich facial features and information differences, and has been widely used in various recognition scenarios, which can be used for identity confirmation and information transmission. The current face recognition scheme mostly trains neural networks with massive face data, and has achieved very high accuracy in general face recognition scenarios, and has been widely used in products. However, face recognition in actual scenarios faces many challenges, such as face size, blur, noise, and light. In scenarios with these interference factors, the quality of face data is damaged, and the recognition accuracy is affected to varying degrees, which limits the application scenarios. Therefore, it is necessary to enable the trained face recognition neural network to better adapt to different scenarios, which can greatly improve the accuracy of face recognition in general scenarios.

[0003] Data augmentation is one of the general methods for enhancing the generalization performance of models in current deep learning, which is used to adapt to different scenarios. By preprocessing the training data, simulating real-world face lighting, blur, noise, and other conditions, the model training generalization is better, and it can better adapt to different scenarios. The current research on face data enhancement mainly focuses on face size, rotation, noise, and other aspects, and there is little research on light in different scenarios. The lighting scene in the real world is very complex, and there are different directions and intensities of light, such as backlight, dark light, and side light. The existing light processing algorithm enhances the global or local light of the image, but rarely considers different light direction scenarios.

[0004] Three-dimensional reconstruction technology is a very important method that can restore three-dimensional information of a scene, and has been widely used in face three-dimensional reconstruction and other fields. In the process of face three-dimensional reconstruction, a large amount of face data is needed. However, in general cases, the collected face data is limited. Therefore, in order to ensure that the face three-dimensional reconstruction can obtain an accurate face model, there is an urgent need for a face data enhancement scheme. SUMMARY

[0005] Therefore, it is necessary to provide a face data enhancement method, device, computer equipment and storage medium supporting accurate face three-dimensional reconstruction to solve the above technical problems.

[0006] A face data enhancement method, the method comprising:

[0007] obtaining a face 2D image;

[0008] performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model;

[0009] performing texture mapping on the face three-dimensional model and light rendering processing based on a preset virtual scene light source to obtain a rendered face three-dimensional model;

[0010] projecting the rendered face three-dimensional model to the face 2D image, and fusing a face region and a background region in the projected face 2D image to obtain an enhanced face 2D image.

[0011] In one of the embodiments, performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model comprises:

[0012] performing three-dimensional reconstruction on the face 2D image by a 3DMM model to obtain a face three-dimensional model.

[0013] In one of the embodiments, performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model comprises:

[0014] obtaining key points and boundary points of the face 2D image to generate an optimized point set;

[0015] querying three-dimensional coordinate information corresponding to the key points and three-dimensional coordinate information corresponding to the optimized point set in the 3D model;

[0016] solving initial transformation parameters from a three-dimensional space to a two-dimensional plane according to the three-dimensional coordinate information corresponding to the key points;

[0017] solving target transformation parameters from the three-dimensional space to the two-dimensional plane according to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimized point set;

[0018] constructing a face three-dimensional model corresponding to the face 2D image according to the target transformation parameters.

[0019] In one of the embodiments, performing texture mapping on the face three-dimensional model and light rendering processing based on a preset virtual scene light source to obtain a rendered face three-dimensional model comprises:

[0020] extracting texture information in the face 2D image;

[0021] filling the texture information into the face three-dimensional model by a bilinear interpolation method;

[0022] performing lightening on the face three-dimensional model with the filled texture information by a preset virtual scene light source to obtain a rendered face three-dimensional model.

[0023] In one of the embodiments, the preset virtual scene light source is a white parallel light source; the lighted face three-dimensional model after the texture information filling is lighted by using the preset virtual scene light source, and a rendered face three-dimensional model is obtained, which includes:

[0024] According to the position of the preset virtual scene light source in the three-dimensional space and the intensity in three directions of the three-dimensional space, a lighted point in the face three-dimensional model after the texture information filling is calculated;

[0025] According to the texture information of the lighted point, the reflectivity corresponding to the lighted point is obtained;

[0026] According to the reflectivity and the light intensity of the preset virtual scene light source in three directions of the three-dimensional space, the pixel value corresponding to the lighted point is obtained;

[0027] According to the pixel value corresponding to the lighted point, the rendered face three-dimensional model is obtained.

[0028] In one of the embodiments, the rendered face three-dimensional model is re-projected to the face 2D image, and the face region and the background region in the re-projected face 2D image are fused to obtain an enhanced face 2D image, which includes:

[0029] The rendered face three-dimensional model is re-projected to the face 2D image to obtain a projected face 2D image;

[0030] The face region and the background region in the projected face 2D image are fused by using a Poisson fusion method to obtain an enhanced face 2D image.

[0031] In one of the embodiments, re-projecting the rendered face three-dimensional model to the face 2D image includes:

[0032] The spatial transformation relationship corresponding to the three-dimensional space to the two-dimensional plane in the three-dimensional reconstruction process of the face 2D image is obtained;

[0033] According to the spatial transformation relationship, the rendered face three-dimensional model is re-projected to the face 2D image.

[0034] A face data enhancement device, the device includes:

[0035] A 2D image acquisition module is configured to acquire a face 2D image;

[0036] A three-dimensional reconstruction module is configured to perform three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model;

[0037] A rendering module is configured to perform texture mapping and light rendering processing based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model;

[0038] The enhancement module is configured to re-project the rendered face three-dimensional model to the face 2D image, and fuse the face region and the background region in the re-projected face 2D image to obtain an enhanced face 2D image.

[0039] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0040] obtaining a face 2D image;

[0041] performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model;

[0042] performing texture mapping and light rendering based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model;

[0043] re-projecting the rendered face three-dimensional model to the face 2D image, and fusing the face region and the background region in the re-projected face 2D image to obtain an enhanced face 2D image.

[0044] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0045] obtaining a face 2D image;

[0046] performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model;

[0047] performing texture mapping and light rendering based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model;

[0048] re-projecting the rendered face three-dimensional model to the face 2D image, and fusing the face region and the background region in the re-projected face 2D image to obtain an enhanced face 2D image.

[0049] The face data enhancement method and device, the computer device and the storage medium, obtain a face 2D image, perform three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model, perform texture mapping and light rendering on the face three-dimensional model, and the light rendering is performed based on a preset virtual scene light source in the light rendering process. The preset virtual scene light source can set the light intensity and the light direction according to actual needs, so that the result of the light rendering is adapted to different real application scenarios, and a face three-dimensional model adapted to a rich scene is obtained. The rendered face three-dimensional model is re-projected to the face 2D image, the face region and the background region in the face 2D image are fused, and an enhanced face 2D image is obtained. The enhanced face 2D image carries rich and diverse face data, which can effectively expand the face training data set, and can support accurate face three-dimensional reconstruction. Attached Figure Description

[0050] Figure 1 This is an application environment diagram of a face data enhancement method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a face data enhancement method in one embodiment;

[0052] Figure 3 This is a flowchart illustrating a face data enhancement method in another embodiment;

[0053] Figure 4 This is a schematic diagram of the face data augmentation process in one application example;

[0054] Figure 5 This is a structural block diagram of a face data enhancement device in one embodiment;

[0055] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] The face data enhancement method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 uploads the acquired 2D face image to server 104. Server 104 acquires the 2D face image, performs 3D reconstruction on the 2D face image to obtain a 3D face model; performs texture fitting and lighting rendering based on a preset virtual scene light source on the 3D face model to obtain a rendered 3D face model; reprojects the rendered 3D face model onto the 2D face image, and merges the face region with the background region in the reprojected 2D face image to obtain an enhanced 2D face image. Furthermore, server 104 can also use the enhanced 2D face image to perform 3D face reconstruction to construct a 3D face model. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0058] In one embodiment, such as Figure 2 As shown, a face data augmentation method is provided, which is then applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0059] S100: Obtain a face 2D image.

[0060] The face 2D image can be directly collected, or obtained by uploading after being collected by an external terminal, or downloaded from a third party (for example, downloaded from the Internet). In actual application, a smart phone can capture a face 2D image of a person A, and upload the face 2D image to a server, so that the server obtains the face 2D image of the person A.

[0061] S200: Perform three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model.

[0062] According to the matching of the key points and the boundary information in the face 2D image and the initial open-source three-dimensional model, three-dimensional reconstruction is performed to obtain a face three-dimensional model. Specifically, the initial open-source three-dimensional model can use the currently available open-source three-dimensional model, for example, a 3DMM model can be used. The server matches and optimizes the 3DMM model according to the key points and the boundary information carried in the obtained face 2D image, performs three-dimensional reconstruction, and obtains a reconstructed face three-dimensional model.

[0063] S300: Perform texture mapping and light rendering processing based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model.

[0064] After the face three-dimensional model and the corresponding two-dimensional plane and three-dimensional space transformation relationship are reconstructed, the texture (color) information of the two-dimensional image can be mapped to the three-dimensional model, that is, the face three-dimensional model is subjected to texture mapping. In addition, light rendering processing based on a preset virtual scene light source is performed to obtain a rendered face three-dimensional model. The preset virtual scene light source is a virtual scene light source that is preset in advance. The light intensity and light direction thereof can be set according to actual needs to simulate a light source in a certain scene. The texture-mapped face three-dimensional model is subjected to light rendering processing based on the preset virtual scene light source to obtain a rendered face three-dimensional model. Optionally, the texture-mapped face three-dimensional model can be subjected to multiple light rendering processing by setting and changing the light intensity and light direction to obtain face three-dimensional model data rendered under different light intensity and light direction. In this way, face three-dimensional model data corresponding to the light source irradiating the face (lighting) in different application scenarios in actual application can be fully simulated.

[0065] S400: Project the rendered face three-dimensional model to the face 2D image, and fuse the face region and the background region in the re-projected face 2D image to obtain an enhanced face 2D image.

[0066] After the light rendering generates the face three-dimensional model, it is necessary to further generate the two-dimensional image after light rendering to enhance the face 2D image data and support the subsequent accurate reconstruction of the face three-dimensional model. Here, the rendered face three-dimensional model is re-projected into the face 2D image to obtain the re-projected face 2D image, which contains the face region re-projected from the three-dimensional space to the two-dimensional plane and the background region in the face 2D image (original image) obtained in S100. Due to the accuracy of the space transformation parameter solving and the brightness difference caused by the light rendering, the face region in the face 2D image generated by the re-projection cannot be well fitted with the background region, so it is necessary to further fuse the face region and the background region to obtain the enhanced face 2D image.

[0067] In one of the embodiments, re-projecting the rendered face three-dimensional model into the face 2D image comprises:

[0068] Obtaining the space transformation relationship corresponding to the three-dimensional space to the two-dimensional plane in the three-dimensional reconstruction process of the face 2D image; and re-projecting the rendered face three-dimensional model into the face 2D image according to the space transformation relationship.

[0069] Here, the rendered face three-dimensional model is re-projected into the face 2D image, which can be understood as the inverse process of the three-dimensional model reconstruction in S200, that is, in S200, the key points and boundary points in the face 2D image are matched to the initial open source three-dimensional model to construct the face three-dimensional model, that is, to realize the conversion between the two-dimensional plane and the three-dimensional space. Here, the data in the face three-dimensional model is re-projected into the face 2D image after texture fitting and light rendering, realizing the conversion from the three-dimensional space to the two-dimensional plane. The space transformation parameters corresponding to the two space conversions can be the same.

[0070] Further, due to the accuracy of the space transformation parameter solving and the brightness difference caused by the light rendering, the face region in the face 2D image generated by the re-projection often cannot be well fitted with the background region in the original image (the face 2D image obtained in S100). Here, the face region in the re-projected face 2D image is fused with the background region, effectively fusing the face region and the background region. Optionally, this fusion process can select Poisson fusion or other fusion methods, such as feather fusion. The fusion effect of the feather fusion may be worse than that of the Poisson fusion, but the fusion speed is faster.

[0071] The face data enhancement method obtains a face 2D image, performs three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model, performs texture fitting and illumination rendering on the face three-dimensional model, and performs illumination rendering based on a preset virtual scene light source in the illumination rendering process. The preset virtual scene light source can set the illumination intensity and the illumination direction according to actual needs, so that the result of the illumination rendering is adapted to different real application scenarios, and a face three-dimensional model adapted to a rich scene is obtained. The rendered face three-dimensional model is re-projected to the face 2D image, the face region and the background region in the face 2D image are fused, and an enhanced face 2D image is obtained. The face data carried in the enhanced face 2D image is rich and diverse, which can effectively expand the face training data set, and can support accurate face three-dimensional reconstruction.

[0072] In one of the embodiments, the three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model comprises:

[0073] The three-dimensional reconstruction on the face 2D image by the 3DMM model to obtain a face three-dimensional model.

[0074] The 3DMM is an open-source face standard model. By constructing matching points of a two-dimensional image and a 3DMM model and establishing an optimization target, a face three-dimensional model corresponding to a face in the two-dimensional image can be generated by using the 3DMM model.

[0075] As shown in Figure 3 In one of the embodiments, S200 comprises:

[0076] S210: Obtain key points and boundary points of the face 2D image, and generate an optimization point set.

[0077] The feature points existing in the face 2D image mainly include key points and boundary points. The key points can be detected by using a face key point detection algorithm, and generally 68 key points of a face can be detected by using the face key point detection algorithm. The boundary points are points obtained by boundary extraction on the face 2D image. Further, the key points are key data for representing a face, and a face three-dimensional model can be accurately constructed based on the key points. However, the generalization of the face three-dimensional model obtained by the key points is poor. In order to enrich the data amount of the face three-dimensional model, the boundary points are also obtained here. In subsequent processing, the initial transformation parameters of a two-dimensional plane and a three-dimensional space are obtained by using the key points, and then the target transformation parameters of the two-dimensional plane and the three-dimensional space are further optimized and solved based on the optimization point set of the key points + boundary points and the initial transformation parameters, so that an accurate and good generalization face three-dimensional model is constructed.

[0078] S220: Query three-dimensional coordinate information corresponding to the key points in the 3D model and three-dimensional coordinate information corresponding to the optimization point set.

[0079] Based on the overall processing concept described, the three-dimensional coordinate information corresponding to the key points in the 3D model and the three-dimensional coordinate information corresponding to the optimized point set are first queried. Specifically, the 3D model here is an initial open source 3D model, for example, a 3DMM model, which itself provides the three-dimensional coordinate information of these key points and the three-dimensional coordinate information corresponding to the optimized point set, so that the correspondence between the key points and the optimized point set in the two-dimensional plane (the face 2D image) and the three-dimensional space (the 3D model) is obtained.

[0080] S230: According to the three-dimensional coordinate information corresponding to the key points, the initial transformation parameters from the three-dimensional space to the two-dimensional plane are solved.

[0081] Based on the correspondence of the key points, the initial transformation parameters from the three-dimensional space to the two-dimensional plane are solved. This solving process specifically includes the following steps: querying the three-dimensional coordinate information corresponding to the key points in the 3D model, obtaining the correspondence of the key points projected from the three-dimensional space to the two-dimensional plane, constructing an Euclidean distance function between the corresponding points in the 3D model and the two-dimensional plane according to the key points, the three-dimensional coordinate information corresponding to the key points and the correspondence, and solving the initial transformation parameters from the three-dimensional space to the two-dimensional plane under the minimum Euclidean distance function.

[0082] In actual application, based on the three-dimensional coordinate information corresponding to the key points, the three-dimensional space to two-dimensional plane correspondence of the key points is obtained, and a distance constraint is constructed for the projection of the three-dimensional points of the 3D model onto the two-dimensional plane and the corresponding points in the two-dimensional plane. In detail, the three-dimensional coordinates of the ith face key point on the 3D model are defined as X i , the corresponding two-dimensional point coordinates on the two-dimensional image (face 2D image) are x i , the three-dimensional to two-dimensional space transformation parameters are the rotation matrix R, the translation matrix T and the scaling coefficient s, and the shape coefficient a of the 3D model. In the 3D model, the spatial position of each point is represented as the product of the principal component P and the coefficient a plus the mean value f, so the point X i can be represented as P i a+f i . P and f are given in the 3D model. The projection process of the three-dimensional point to the two-dimensional point is denoted as SOP, and the parameters are R, T, s, so that the obtained two-dimensional point coordinates can be represented as SOP[P i a+f i , R, T, s], and an optimization objective is constructed for the Euclidean distance E lmk between the two-dimensional coordinates of the 3D model after projection and the corresponding points in the two-dimensional image (face 2D image), and the expression is:

[0083]

[0084] In the above formula (1), L is the number of corresponding points. Solving the minimization (1) can obtain a set of initial solutions a0, R0, T0, s0, i.e. a set of initial transformation parameters of three-dimensional points to two-dimensional points.

[0085] S240: According to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimization point set, solving the target transformation parameters of the three-dimensional space to the two-dimensional plane.

[0086] In addition, in order to increase the diversity of data, so that the final obtained face three-dimensional model has better generalization, the face boundary in the face 2D image is also extracted here, and the same processing method as described above for the key points is adopted. The same processing is performed on the optimization point set with the boundary points to obtain the corresponding relationship between the new three-dimensional points and the two-dimensional points. Then, the initial transformation parameters are taken as the initial solutions to solve the above formula (1) corresponding to the optimization point set again to obtain the target transformation parameters, thereby obtaining the spatial transformation corresponding relationship between the three-dimensional model and the two-dimensional image.

[0087] S250: According to the target transformation parameters, constructing the face three-dimensional model corresponding to the face 2D image.

[0088] The target transformation parameters represent the accurate spatial transformation corresponding relationship between the three-dimensional model and the two-dimensional image, and based on the data, the face three-dimensional model can be accurately constructed.

[0089] As shown in the following figure, in one embodiment, S300 includes: Figure 3

[0090] S320: Extracting texture information in the face 2D image;

[0091] S340: Filling the texture information into the face three-dimensional model by bilinear interpolation method;

[0092] S360: Lighting the face three-dimensional model filled with texture information by using a preset virtual scene light source to obtain a rendered face three-dimensional model.

[0093] After obtaining the reconstructed face three-dimensional model and the corresponding spatial transformation relationship, the texture (color) information of the two-dimensional image can be mapped to the three-dimensional model. Since the coordinate points calculated by the mapping relationship are usually floating-point numbers, they cannot be corresponding to integer coordinate positions, therefore, the bilinear interpolation method is adopted to obtain the texture information of the three-dimensional points. The specific method is that if the texture information of a three-dimensional point X i is to be filled, it is first mapped to a two-dimensional image by the SOP process to obtain a set of floating-point coordinates. According to the position of the floating-point coordinates, the neighborhood four pixel points and their weights (the distance of the floating-point part of the floating-point coordinates from the integer point is the weight) are determined, and the color information of the current position is interpolated, that is, the corresponding three-dimensional point X​i Color information of the bilinear interpolation is expressed as:

[0094] f = w 11 x P 11 + w 12 x P 12 + w 21 x P 21 + w 22 x P 22 (2)

[0095] In the above formula (2), the subscripts 11, 12, 21 and 22 respectively represent the upper left corner, the upper right corner, the lower left corner and the lower right corner points corresponding to the floating point coordinates, w is the weight calculated from the floating point coordinate position, P is the pixel value of the corresponding point, and f is the result calculated by the interpolation. After completing the texture information filling of the three-dimensional model, a pre-set virtual scene light source is obtained, in which the light intensity and the light direction are set according to actual needs, the three-dimensional model is lighted based on the set light intensity and light direction, and a rendered face three-dimensional model is obtained. Here, the light is lighted based on the pre-set virtual scene light source, the data is enhanced through the light, and different light directions are provided for face recognition, so that an enhanced face 2D image can be finally obtained.

[0096] In one embodiment, the pre-set virtual scene light source is a white parallel light source; the face three-dimensional model after the texture information filling is lighted by using the pre-set virtual scene light source, and a rendered face three-dimensional model is obtained, which includes:

[0097] According to the position of the pre-set virtual scene light source in the three-dimensional space and the intensity in three directions of the three-dimensional space, the illuminated points in the face three-dimensional model after the texture information filling are calculated; according to the texture information of the illuminated points, the reflectivity corresponding to the illuminated points is obtained; according to the reflectivity and the light intensity of the pre-set virtual scene light source in three directions of the three-dimensional space, the pixel value corresponding to the illuminated points is obtained; and according to the pixel value corresponding to the illuminated points, the rendered face three-dimensional model is obtained.

[0098] In the process of lightening, the intensity, direction of the light source and the shape and surface reflectivity of the face model need to be considered at the same time. In this embodiment, the pre-set virtual scene light source is a white parallel light source, so the texture information can be directly used as the reflectivity of the model. The position of the light source in the three-dimensional space is set as [X, Y, Z], and the intensity in three directions is s = [s1, s2, s3]. In this way, it can be calculated which points in the three-dimensional model can be illuminated by the light source, and the pixel value after illumination is calculated through the reflectivity and the light intensity. The calculation formula is

[0099]

[0100] Wherein P' is the pixel value after lighting, P is the original pixel value, is the unit normal vector of the plane where the current three-dimensional point is located.

[0101] In practical applications, the overall flow of the face data enhancement method of the present application is as shown in Figure 4 The input is a face 2D image, which is matched and optimized with the 3DMM model according to the key points and boundary information, to obtain a reconstructed face three-dimensional model. After the texture information is fitted, different angles and intensities of light rendering are performed, and then the three-dimensional model is projected back to a two-dimensional image. Finally, the face region and the background region are fused by image fusion to generate the final lighted image, realizing face 2D image enhancement.

[0102] Overall, the face data enhancement method of the present application has the following characteristics and technical advantages in practical applications:

[0103] (1) Focus on solving the problem of face light data enhancement, and propose a scheme that can generate face images of any lighting scene. Data enhancement can effectively expand the face training dataset, making the trained model more robust and able to adapt to various real scenes. The lighting conditions of real scenes are very complex, and the scheme of the present application can effectively generate data of various lighting scenes.

[0104] (2) A method based on virtual scene light source is proposed, which sets a light source with a specific position and intensity to light the reconstructed three-dimensional model, and obtains the pixel value after light enhancement through reflectivity relationship calculation.

[0105] (3) Combine the three-dimensional reconstruction method of 3DMM and Poisson fusion to effectively fuse the rendered face image, making the generated image more natural.

[0106] It should be understood that although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0107] As Figure 5 shown, the present application also provides a face data enhancement device, the device comprising:

[0108] 2D image acquisition module 100, used to acquire a face 2D image;

[0109] Three-dimensional reconstruction module 200, used to perform three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model;

[0110] Rendering module 300, used to perform texture mapping and illumination rendering processing on the face three-dimensional model based on a preset virtual scene light source to obtain a rendered face three-dimensional model;

[0111] Enhancement module 400, used to project the rendered face three-dimensional model to the face 2D image, and fuse the face region and the background region in the projected face 2D image to obtain an enhanced face 2D image.

[0112] The above face data enhancement device acquires a face 2D image, performs three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model, performs texture mapping and illumination rendering on the face three-dimensional model, and the illumination rendering is performed based on a preset virtual scene light source, the preset virtual scene light source can set the illumination intensity and the illumination direction according to actual needs, so that the result of the illumination rendering adapts to different real application scenarios, and a face three-dimensional model that adapts to a rich scene is obtained, the rendered face three-dimensional model is projected to the face 2D image, and the face region and the background region in the projected face 2D image are fused to obtain an enhanced face 2D image, the enhanced face 2D image carries rich and diverse face data, which can effectively expand a face training data set, and accurate face three-dimensional reconstruction can be supported.

[0113] In one of the embodiments, the three-dimensional reconstruction module 200 is further used to perform three-dimensional reconstruction on the face 2D image by a 3DMM model to obtain the face three-dimensional model.

[0114] In one of the embodiments, the three-dimensional reconstruction module 200 is further used to acquire key points and boundary points of the face 2D image, generate an optimized point set, query three-dimensional coordinate information corresponding to the key points in the 3D model and three-dimensional coordinate information corresponding to the optimized point set, solve initial transformation parameters from the three-dimensional space to the two-dimensional plane according to the three-dimensional coordinate information corresponding to the key points, solve target transformation parameters from the three-dimensional space to the two-dimensional plane according to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimized point set, and construct the face three-dimensional model corresponding to the face 2D image according to the target transformation parameters.

[0115] In one of the embodiments, the rendering module 300 is further used to extract texture information in the face 2D image, fill the texture information into the face three-dimensional model by a bilinear interpolation method, and light the face three-dimensional model filled with the texture information by the preset virtual scene light source to obtain the rendered face three-dimensional model.

[0116] In one of the embodiments, the preset virtual scene light source is a white parallel light source; the rendering module 300 is further configured to calculate the illuminated points in the three-dimensional face model after the texture information is filled according to the position of the preset virtual scene light source in the three-dimensional space and the intensity of the preset virtual scene light source in three directions of the three-dimensional space; obtain the reflectivity corresponding to the illuminated points according to the texture information of the illuminated points; obtain the pixel value corresponding to the illuminated points according to the reflectivity and the intensity of the preset virtual scene light source in the three directions of the three-dimensional space; and obtain the rendered three-dimensional face model according to the pixel value corresponding to the illuminated points.

[0117] In one of the embodiments, the enhancement module 400 is further configured to project the rendered three-dimensional face model to the face 2D image to obtain a projected face 2D image; and fuse the face region and the background region in the projected face 2D image by using a Poisson blending method to obtain an enhanced face 2D image.

[0118] In one of the embodiments, the enhancement module 400 is further configured to obtain a spatial transformation relationship corresponding to the three-dimensional space to the two-dimensional plane in the three-dimensional reconstruction process of the face 2D image; and project the rendered three-dimensional face model to the face 2D image according to the spatial transformation relationship.

[0119] The specific embodiments of the face data enhancement apparatus can refer to the embodiments of the face data enhancement method described above, and will not be described herein. Each module in the face data enhancement apparatus described above can be realized by software, hardware, and a combination thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0120] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 6 The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data related to the preset virtual scene light source. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a face data enhancement method.

[0121] Those skilled in the art can understand that Figure 6The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0123] obtaining a face 2D image;

[0124] performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model;

[0125] performing texture mapping and illumination rendering processing based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model;

[0126] reprojecting the rendered face three-dimensional model to the face 2D image, and fusing the face region and the background region in the reprojected face 2D image to obtain an enhanced face 2D image.

[0127] In one embodiment, the processor further implements the following steps when executing the computer program:

[0128] performing three-dimensional reconstruction on the face 2D image by a 3DMM model to obtain a face three-dimensional model.

[0129] In one embodiment, the processor further implements the following steps when executing the computer program:

[0130] obtaining key points and boundary points of the face 2D image to generate an optimized point set, querying three-dimensional coordinate information corresponding to the key points in the 3D model and three-dimensional coordinate information corresponding to the optimized point set, solving initial transformation parameters from the three-dimensional space to the two-dimensional plane according to the three-dimensional coordinate information corresponding to the key points, solving target transformation parameters from the three-dimensional space to the two-dimensional plane according to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimized point set, and constructing a face three-dimensional model corresponding to the face 2D image according to the target transformation parameters.

[0131] In one embodiment, the processor further implements the following steps when executing the computer program:

[0132] extracting texture information in the face 2D image, filling the texture information into the face three-dimensional model by a bilinear interpolation method, and performing lighting on the face three-dimensional model with the filled texture information by a preset virtual scene light source to obtain a rendered face three-dimensional model.

[0133] In one of the embodiments, the preset virtual scene light source is a white parallel light source; when the processor executes the computer program, the following steps are further implemented:

[0134] According to the position of the preset virtual scene light source in the three-dimensional space and the intensity of the preset virtual scene light source in three directions of the three-dimensional space, the illuminated point in the three-dimensional model of the face after filling of the texture information is calculated; according to the texture information of the illuminated point, the reflectivity corresponding to the illuminated point is obtained; according to the reflectivity and the intensity of the preset virtual scene light source in the three directions of the three-dimensional space, the pixel value corresponding to the illuminated point is obtained; and according to the pixel value corresponding to the illuminated point, the three-dimensional model of the face after rendering is obtained.

[0135] In one of the embodiments, when the processor executes the computer program, the following steps are further implemented:

[0136] The three-dimensional model of the face after rendering is re-projected to the two-dimensional image of the face to obtain a projected two-dimensional image of the face; and the face region and the background region in the projected two-dimensional image of the face are fused in a Poisson blending manner to obtain an enhanced two-dimensional image of the face.

[0137] In one of the embodiments, when the processor executes the computer program, the following steps are further implemented:

[0138] The spatial transformation relationship corresponding to the three-dimensional space to the two-dimensional plane in the three-dimensional reconstruction process of the two-dimensional image of the face is obtained; and the three-dimensional model of the face after rendering is re-projected to the two-dimensional image of the face according to the spatial transformation relationship.

[0139] In one of the embodiments, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0140] The two-dimensional image of the face is obtained;

[0141] The three-dimensional model of the face is obtained by performing three-dimensional reconstruction on the two-dimensional image of the face;

[0142] The three-dimensional model of the face is subjected to texture mapping and illumination rendering processing based on a preset virtual scene light source to obtain a three-dimensional model of the face after rendering;

[0143] The three-dimensional model of the face after rendering is re-projected to the two-dimensional image of the face, and the face region and the background region in the re-projected two-dimensional image of the face are fused to obtain an enhanced two-dimensional image of the face.

[0144] In one of the embodiments, when the computer program is executed by the processor, the following steps are further implemented:

[0145] The three-dimensional model of the face is obtained by performing three-dimensional reconstruction on the two-dimensional image of the face through a 3DMM model.

[0146] In one embodiment, the computer program which is executed by the processor further implements the following steps:

[0147] Obtaining the key points and the boundary points of the face 2D image, generating an optimized point set; querying the three-dimensional coordinate information corresponding to the key points in the 3D model and the three-dimensional coordinate information corresponding to the optimized point set; solving the initial transformation parameters from the three-dimensional space to the two-dimensional plane according to the three-dimensional coordinate information corresponding to the key points; solving the target transformation parameters from the three-dimensional space to the two-dimensional plane according to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimized point set; constructing the face three-dimensional model corresponding to the face 2D image according to the target transformation parameters.

[0148] In one embodiment, the computer program which is executed by the processor further implements the following steps:

[0149] Extracting the texture information in the face 2D image; filling the texture information into the face three-dimensional model by using the bilinear interpolation method; lighting the face three-dimensional model after filling the texture information by using a preset virtual scene light source, obtaining the rendered face three-dimensional model.

[0150] In one of the embodiments, the preset virtual scene light source is a white parallel light source; the computer program which is executed by the processor further implements the following steps:

[0151] According to the position of the preset virtual scene light source in the three-dimensional space and the intensity in three directions of the three-dimensional space, calculating the illuminated points in the face three-dimensional model after filling the texture information; according to the texture information of the illuminated points, obtaining the reflectivity corresponding to the illuminated points; according to the reflectivity and the light intensity in three directions of the three-dimensional space of the preset virtual scene light source, obtaining the pixel value corresponding to the illuminated points; according to the pixel value corresponding to the illuminated points, obtaining the rendered face three-dimensional model.

[0152] In one embodiment, the computer program which is executed by the processor further implements the following steps:

[0153] Re-projecting the rendered face three-dimensional model to the face 2D image, obtaining the projected face 2D image; fusing the face region and the background region in the projected face 2D image by using the Poisson fusion method, obtaining the enhanced face 2D image.

[0154] In one embodiment, the computer program which is executed by the processor further implements the following steps:

[0155] Obtaining the spatial transformation relationship corresponding to the three-dimensional space to the two-dimensional plane in the process of three-dimensional reconstruction of the face 2D image; re-projecting the rendered face three-dimensional model to the face 2D image according to the spatial transformation relationship.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0157] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of each technical feature in the above embodiments are not described, but as long as the combination of these technical features does not exist, it should be considered as the scope of the present application.

[0158] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A face data enhancement method, characterized in that, The method comprises: acquiring a face 2D image; performing three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model; performing texture mapping and light rendering processing based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model; the preset virtual scene light source is used to simulate a light source in an actual demand scene; projecting the rendered face three-dimensional model to the face 2D image, and fusing a face region and a background region in the projected face 2D image to obtain an enhanced face 2D image; the texture mapping and light rendering processing based on the preset virtual scene light source on the face three-dimensional model to obtain the rendered face three-dimensional model comprises: extracting texture information in the face 2D image; filling the texture information into the face three-dimensional model through a bilinear interpolation method; and performing lightening on the face three-dimensional model after the texture information is filled by using the preset virtual scene light source to obtain the rendered face three-dimensional model; the preset virtual scene light source is a white parallel light source; and the lightening on the face three-dimensional model after the texture information is filled by using the preset virtual scene light source to obtain the rendered face three-dimensional model comprises: calculating an illuminated point in the face three-dimensional model after the texture information is filled according to a position of the preset virtual scene light source in a three-dimensional space and intensities of the preset virtual scene light source in three directions of the three-dimensional space; obtaining reflectivity corresponding to the illuminated point according to texture information of the illuminated point; obtaining a pixel value corresponding to the illuminated point according to the reflectivity and the light intensities of the preset virtual scene light source in the three directions of the three-dimensional space; and obtaining the rendered face three-dimensional model according to the pixel value corresponding to the illuminated point.

2. The method of claim 1, wherein, the three-dimensional reconstruction on the face 2D image to obtain the face three-dimensional model comprises: performing three-dimensional reconstruction on the face 2D image by using a 3DMM model to obtain the face three-dimensional model.

3. The method of claim 1, wherein, the three-dimensional reconstruction on the face 2D image to obtain the face three-dimensional model comprises: obtaining key points and boundary points of the face 2D image to generate an optimized point set; querying three-dimensional coordinate information corresponding to the key points and three-dimensional coordinate information corresponding to the optimized point set in a 3D model; solving initial transformation parameters from a three-dimensional space to a two-dimensional plane according to the three-dimensional coordinate information corresponding to the key points; solving target transformation parameters from the three-dimensional space to the two-dimensional plane according to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimized point set; constructing a face three-dimensional model corresponding to the face 2D image according to the target transformation parameters.

4. The method of claim 1, wherein, the projection of the rendered face three-dimensional model to the face 2D image and the fusion of a face region and a background region in the projected face 2D image to obtain the enhanced face 2D image comprises: projecting the rendered face three-dimensional model to the face 2D image to obtain a projected face 2D image; fusing a face region and a background region in the projected face 2D image by using a Poisson fusion mode to obtain the enhanced face 2D image.

5. The method of claim 4, wherein, The re-projecting the rendered face three-dimensional model to the face 2D image comprises: obtaining a spatial transformation relationship from a three-dimensional space to a two-dimensional plane in a three-dimensional reconstruction process of the face 2D image; re-projecting the rendered face three-dimensional model to the face 2D image according to the spatial transformation relationship.

6. A face data augmentation apparatus, comprising: The device comprises: a 2D image obtaining module configured to obtain a face 2D image; a three-dimensional reconstruction module configured to perform three-dimensional reconstruction on the face 2D image to obtain a face three-dimensional model; a rendering module configured to perform texture mapping and illumination rendering processing based on a preset virtual scene light source on the face three-dimensional model to obtain a rendered face three-dimensional model; the preset virtual scene light source is used to simulate a light source in an actual demand scene; an enhancement module configured to re-project the rendered face three-dimensional model to the face 2D image and fuse a face region and a background region in the re-projected face 2D image to obtain an enhanced face 2D image; the rendering module is further configured to extract texture information in the face 2D image, fill the texture information into the face three-dimensional model by using a bilinear interpolation method, and perform lighting on the face three-dimensional model with the filled texture information by using the preset virtual scene light source to obtain the rendered face three-dimensional model; the rendering module is further configured to calculate an illuminated point in the face three-dimensional model with the filled texture information according to a position of the preset virtual scene light source in a three-dimensional space and intensities of the preset virtual scene light source in three directions of the three-dimensional space, obtain a reflectivity corresponding to the illuminated point according to texture information of the illuminated point, obtain a pixel value corresponding to the illuminated point according to the reflectivity and the intensities of the preset virtual scene light source in the three directions of the three-dimensional space, and obtain the rendered face three-dimensional model according to the pixel value corresponding to the illuminated point; the preset virtual scene light source is a white parallel light source.

7. The apparatus of claim 6, wherein, The three-dimensional reconstruction module is further configured to perform three-dimensional reconstruction on the face 2D image by using a 3DMM model to obtain the face three-dimensional model.

8. The apparatus of claim 6, wherein, The three-dimensional reconstruction module is further configured to obtain key points and boundary points of the face 2D image to generate an optimized point set, query three-dimensional coordinate information corresponding to the key points in the 3D model and three-dimensional coordinate information corresponding to the optimized point set, solve initial transformation parameters from a three-dimensional space to a two-dimensional plane according to the three-dimensional coordinate information corresponding to the key points, solve target transformation parameters from the three-dimensional space to the two-dimensional plane according to the initial transformation parameters and the three-dimensional coordinate information corresponding to the optimized point set, and construct the face three-dimensional model corresponding to the face 2D image according to the target transformation parameters. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor implements the steps of the method in any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method in any one of claims 1 to 5.

Images