Face reconstruction and occluded region identification method, device and equipment and storage medium

Abstract

Embodiments of the application disclose a face reconstruction and occlusion region identification method, device and equipment and a storage medium. The method comprises: inputting a face image into a first network structure, outputting a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure; performing three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information; performing rendering and mapping processing on the three-dimensional face information to generate two-dimensional face information containing network segmentation data; obtaining a deep feature map extracted by the first network structure, and constructing graph structure information based on the deep feature map and the two-dimensional face information; inputting the graph structure information into a second network structure, and outputting an occlusion region corresponding to the face image through the second network structure. The scheme optimizes overall resource deployment, can meet application scenarios with many parallel tasks and high real-time requirements, and has higher accuracy in occlusion region determination.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for face reconstruction and occlusion region recognition. Background Technology

[0002] With the development and popularization of virtual reality technology, people are no longer satisfied with ordinary two-dimensional interaction and have an urgent need for applications in three-dimensional space such as 3D face reshaping and 3D facial sculpting. Currently, most 3D face reshaping and stylization applications are deployed in live streaming and social media scenarios, requiring high real-time performance and reconstruction quality. Meanwhile, because the actual production environment is much more complex than imagined, input data does not always contain complete faces or may contain self-occlusion, object occlusion, etc., therefore the model needs to be robust to diverse input data. Furthermore, it is often necessary to determine the actual occlusion area to perform a series of post-processing operations.

[0003] In related technologies, convolutional neural networks are typically used to extract 3D facial features and segment 2D faces separately for occlusion region determination. This requires running two separate tasks for prediction, thus demanding more resources and failing to meet the needs of applications with many parallel tasks and high real-time requirements. Furthermore, this approach to occlusion region determination has poor robustness. Moreover, most existing methods for occlusion region determination rely on traditional semantic segmentation tasks, resulting in low accuracy in occlusion region determination. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for face reconstruction and occlusion region recognition, which solves the problems in related technologies, optimizes the overall resource deployment, can meet the application scenarios with many parallel tasks and high real-time requirements, and has higher accuracy in occlusion region determination.

[0005] In a first aspect, embodiments of this application provide a method for face reconstruction and occlusion region recognition, the method comprising:

[0006] A face image is input into a first network structure, and the first network structure outputs a reconstruction parameter vector of the face image during 3D reconstruction.

[0007] Based on the reconstruction parameter vector, the face image is reconstructed in three dimensions to generate three-dimensional face information;

[0008] The three-dimensional face information is rendered and mapped to generate two-dimensional face information containing network segmentation data;

[0009] Obtain the depth feature map extracted by the first network structure, and construct graph structure information based on the depth feature map and the two-dimensional face information;

[0010] The graph structure information is input into the second network structure, and the occlusion area corresponding to the face image is output through the second network structure.

[0011] Secondly, embodiments of this application also provide a face reconstruction and occlusion region recognition device, including:

[0012] The parameter vector generation module is configured to input a face image into a first network structure and output the reconstruction parameter vector of the face image during three-dimensional reconstruction through the first network structure.

[0013] The three-dimensional information determination module is configured to perform three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information;

[0014] The rendering and mapping module is configured to render and map the three-dimensional face information to generate two-dimensional face information containing network segmentation data;

[0015] The graph information construction module is configured to obtain the depth feature map extracted by the first network structure, and construct graph structure information based on the depth feature map and the two-dimensional face information;

[0016] The occlusion region determination module is configured to input the graph structure information into a second network structure and output the occlusion region corresponding to the face image through the second network structure.

[0017] Thirdly, embodiments of this application also provide a face reconstruction and occlusion region recognition device, the device comprising:

[0018] One or more processors;

[0019] Storage device for storing one or more programs.

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the face reconstruction and occlusion region recognition method described in the embodiments of this application.

[0021] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the face reconstruction and occlusion region recognition method described in embodiments of this application.

[0022] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the face reconstruction and occlusion region recognition method described in embodiments of this application.

[0023] In this embodiment, a face image is input into a first network structure, which outputs a reconstruction parameter vector for the face image during 3D reconstruction. Based on this parameter vector, 3D face information is generated through 3D reconstruction of the face image. The 3D face information is then rendered and mapped to generate 2D face information containing network segmentation data. Further, a depth feature map extracted by the first network structure is obtained. Based on the depth feature map and the 2D face information, graph structure information is constructed and input into a second network structure. The second network structure outputs the occlusion region corresponding to the face image. This solution provides a face reconstruction and occlusion region recognition method that merges face reconstruction and occlusion region recognition into a single model. During occlusion region recognition, relevant data from the face reconstruction process is utilized, modeling the occlusion problem on a 2D plane as a 3D face occlusion recognition, which is more consistent with human perception. Furthermore, by utilizing relevant 3D face data, it is easier to uncover hidden related features, resulting in more accurate occlusion region recognition results. Meanwhile, it fully utilizes the common features of face reconstruction and occlusion region recognition tasks, compressing the model as much as possible while enabling the two tasks to promote each other, optimizing the overall resource deployment, and meeting the application scenarios with many parallel tasks and high real-time requirements, with higher accuracy in occlusion region determination. Attached Figure Description

[0024] Figure 1 A flowchart illustrating a face reconstruction and occlusion region recognition method provided in this application embodiment;

[0025] Figure 2 A flowchart illustrating a method for 3D reconstruction of face images based on reconstruction parameter vectors, provided in this application embodiment;

[0026] Figure 3 A schematic diagram of face image reconstruction and processing provided in an embodiment of this application;

[0027] Figure 4 A flowchart illustrating a method for constructing graph structure information based on depth feature maps and two-dimensional face information, provided in this application embodiment;

[0028] Figure 5 This is a schematic diagram illustrating special effects processing of an image, provided as an embodiment of this application.

[0029] Figure 6 A structural block diagram of a face reconstruction and occlusion region recognition device provided in this application embodiment;

[0030] Figure 7 This is a schematic diagram of the structure of a face reconstruction and occlusion area recognition device provided in an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0033] The face reconstruction and occlusion region recognition method provided in this application embodiment can be applied to various scenarios requiring 3D face reconstruction, and can accurately determine whether the current face is occluded and the corresponding occlusion region. Specifically, it can be used to process face images in short videos and live streams to achieve face reconstruction and occlusion region recognition. After the occlusion region recognition is completed, further beautification and other special effects processing can be applied based on the occlusion region recognition results to ensure the effectiveness of the effects.

[0034] Figure 1 A flowchart of a face reconstruction and occlusion region recognition method provided in this application embodiment specifically includes the following steps:

[0035] Step S101: Input the face image into the first network structure, and output the reconstruction parameter vector of the face image during three-dimensional reconstruction through the first network structure.

[0036] In one embodiment, the face image is an image containing a face region, which may be, for example, an image captured by a camera or a user-input image. Optionally, before inputting the face image into the first network structure, a face detector can perform relevant face detection and alignment correction on the face image.

[0037] In one embodiment, the first network structure can be a convolutional neural network, such as the MobileNet-v3 network structure (a lightweight convolutional neural network, the third generation of the MobileNet series), the VGG network structure, or a backbone network structure like ResNet. Here, the first network structure is a pre-trained network structure, which takes a 2D image as input during training, passes it through a series of convolutional layers, and outputs a reconstruction parameter vector for 3D face reconstruction. Optionally, the first network structure uses a weakly supervised reconstruction loss function during training.

[0038] In one embodiment, after a face image is input into a first network structure, the reconstruction parameter vector output by the first network structure exemplarily includes: a face feature vector, a face expression vector, and a three-dimensional face coefficient vector. It also includes other vectors used for three-dimensional reconstruction, such as illumination vectors, reflectivity vectors, pose vectors, and translation vectors.

[0039] Step S102: Based on the reconstruction parameter vector, perform three-dimensional reconstruction of the face image to generate three-dimensional face information.

[0040] In one embodiment, after obtaining the reconstruction parameter vector corresponding to the input face image through a first network structure, 3D reconstruction of the face image is performed based on the reconstruction parameter vector to obtain 3D face information. An exemplary 3D reconstruction process is as follows: Figure 2 As shown, Figure 2 A flowchart of a method for 3D reconstruction of face images based on reconstruction parameter vectors provided in this application embodiment specifically includes:

[0041] Step S1021: Reconstruct the three-dimensional face point cloud based on the face feature vector, face expression vector, three-dimensional face coefficient vector and preset average face shape information to obtain point cloud reconstruction information.

[0042] In one embodiment, 3D face reconstruction includes the reconstruction of a 3D face point cloud and the generation of face texture information. Optionally, during the reconstruction of the 3D face point cloud, reconstruction is performed based on the face feature vector, face expression vector, 3D face coefficient vector, and preset average face shape information from the determined reconstruction parameter vector.

[0043] For example, let B be the facial feature vector. id The facial expression vector is denoted as B. exp The three-dimensional face coefficient vector includes α, β, and δ, and the preset average face shape information is denoted as . For example, when determining the point cloud reconstruction information S, the following formula is used to calculate it:

[0044]

[0045] Step S1022: Reconstruct face texture based on the three-dimensional face coefficient vector and the preset average face texture information and face base information to obtain face texture information.

[0046] In one embodiment, when generating facial texture information, it is based on the three-dimensional facial coefficient vector in the aforementioned determined reconstruction parameter vector, as well as the acquired facial base information and the preset average facial texture information. Optionally, the facial base information can use the base information in a publicly available facial model.

[0047] For example, a three-dimensional face coefficient vector includes α, β, and δ, and the face basis information is denoted as B. t The preset average face texture information is denoted as The facial texture information T is calculated using the following formula:

[0048]

[0049] Step S1023: Generate 3D face information based on point cloud reconstruction information and face texture information.

[0050] In one embodiment, after obtaining point cloud reconstruction information and facial texture information, the final 3D facial information is generated based on the point cloud reconstruction information and facial texture information. For example, the 3D facial information can be obtained by superimposing and fitting the point cloud reconstruction information and facial texture information.

[0051] Step S103: Render and map the three-dimensional face information to generate two-dimensional face information containing network segmentation data.

[0052] In one embodiment, after obtaining 3D face information, further rendering and mapping processing is performed to generate 2D face information, wherein the generated 2D face information includes network segmentation data. Optionally, the rendering process can be to render the 3D face information into a 2D face image using a renderer, or to perform rendering processing using a set rendering model or other rendering algorithms; the mapping process can be: based on the topological structure in the 3D face information, mapping the 3D face point pairs in the 3D face information to the 2D face image. For example, as shown... Figure 3 As shown, Figure 3 This is a schematic diagram of a face image reconstruction and processing provided in an embodiment of this application, wherein the original input face image is I. T The image obtained after 3D reconstruction is V R Regarding the image V R A two-dimensional face image can be obtained by using a differential renderer. RAt this point, combined with the reconstructed image V R The topological structure projects the triangular patches of 1293 three-dimensional face point pairs onto the two-dimensional face image I. R Generate 2D face information containing network segmentation data in Tr R .

[0053] Step S104: Obtain the depth feature map extracted by the first network structure, and construct the graph structure information based on the depth feature map and the two-dimensional face information.

[0054] In one embodiment, when identifying occluded regions, a depth feature map extracted from the first network structure during 3D reconstruction is used. Based on this depth feature map and the 2D face information generated in step S103, image structure information is constructed. Subsequently, occluded regions are further identified based on this image structure information. The depth feature map is the feature of the layer obtained after calculation of the convolutional layers in the first network structure.

[0055] Optionally, the method of constructing graph structure information using the depth feature map and two-dimensional face information is exemplified as follows: Figure 4 As shown, Figure 4 A flowchart illustrating a method for constructing graph structure information based on depth feature maps and two-dimensional face information, provided in this application embodiment, specifically includes:

[0056] Step S1041: Based on the network segmentation data in the two-dimensional face information, the depth feature map is segmented to obtain the triangular face segmentation result.

[0057] Specifically, when segmenting the depth feature map, the same segmentation strategy used in network segmentation of 2D face information is employed. Further, the segmentation process involves adjusting the size of the depth feature map to be consistent with the size of the 2D face image, ensuring they are the same size. Then, the same segmentation strategy is applied to obtain the triangular facet segmentation result.

[0058] Step S1042: Based on the vertex connectivity and point-to-point distances in the triangular facet segmentation results, construct the adjacency matrix to obtain graph structure information.

[0059] After obtaining the triangular facet segmentation results, an adjacency matrix is ​​constructed using the vertex connectivity of the 3D face mesh and the point-to-point distances after projection onto the 2D plane, thus completing the construction of graph structure information. For example, the adjacency matrix A is constructed as follows: ij Taking ∈NxN as an example, where N represents the number of vertices in the 3D face point cloud, 1293 vertices are used in one embodiment. The specific method for constructing the adjacency matrix based on vertex connectivity and point-to-point distance is as follows:

[0060] Aij =C ij *D ij

[0061]

[0062] Among them, C ij This represents whether two vertices are connected; the value is 0 if they are not connected, and 1 otherwise. ij This represents the point-to-point distance between two vertices after projection onto a two-dimensional plane.

[0063] Step S105: Input the graph structure information into the second network structure, and output the occlusion area corresponding to the face image through the second network structure.

[0064] In one embodiment, the second network structure can be a graph convolutional neural network (GNN). The graph structure information generated in step S104 is input into the second network structure, and the second network structure outputs the occlusion region corresponding to the face image. Optionally, the 3D face vertices in the graph structure information can be classified using a graph convolutional neural network, and the occlusion region corresponding to the face image can be output based on the classification result. Optionally, the framework of the second network structure mainly uses a graph attention network, which integrates and classifies the 3D face vertices and finally outputs a segmentation mask M of the occlusion region. R .

[0065] Optionally, this second network structure uses a supervised segmentation loss function during training, such as the Dice loss function, to supervise the training of the occluded mask and the real mask. Experiments show that it can obtain more accurate results for segmentation mask prediction compared with the traditional cross-entropy loss function.

[0066] As described above, the scheme involves inputting a face image into a first network structure, which outputs a reconstruction parameter vector for the face image during 3D reconstruction. Based on this parameter vector, 3D face information is generated through 3D reconstruction. This 3D face information is then rendered and mapped to generate 2D face information containing network segmentation data. Further, the depth feature map extracted by the first network structure is obtained. Based on the depth feature map and the 2D face information, a graph structure is constructed and input into a second network structure. The second network structure then outputs the occlusion region corresponding to the face image. This scheme combines face reconstruction and occlusion region recognition into a single model. In occlusion region recognition, relevant data from the face reconstruction process is utilized, modeling the occlusion problem on a 2D plane as a 3D face occlusion recognition problem. This is more consistent with human perception. Furthermore, by utilizing relevant 3D face data, it is easier to uncover hidden related features, resulting in more accurate occlusion region recognition results. Meanwhile, it fully utilizes the common features of face reconstruction and occlusion region recognition tasks, compressing the model as much as possible while enabling the two tasks to promote each other, optimizing the overall resource deployment, and meeting the application scenarios with many parallel tasks and high real-time requirements, with higher accuracy in occlusion region determination.

[0067] The above solution integrates face reconstruction and occlusion region recognition into a single model for processing. It simultaneously extracts features from two related tasks, thereby addressing the pain point that traditional solutions cannot uncover the hidden related features between the two tasks. This approach compresses the model size and achieves better reconstruction and segmentation results.

[0068] Based on the above scheme, after outputting the occlusion region corresponding to the face image through the second network structure, the method further includes: performing special effects rendering processing based on the face image, two-dimensional face information, and the occlusion region, and displaying the processing result. That is, after performing 3D reconstruction and determining the occlusion region, various special effects processing can be further applied based on this. For example, such as... Figure 5 As shown, Figure 5 This is a schematic diagram illustrating special effects processing of an image, provided as an embodiment of this application. Wherein, a face image I... T The original input image, image M R Image I is the output image after occlusion region recognition. R To obtain the final image I, which is the image corresponding to the rendered and mapped 2D face information, the three processes are combined. F Optionally, during the special effects rendering process, the occluded area can be rendered using the original image's occluder, while the facial image portion outside the occluded area can be rendered based on 3D facial information.

[0069] Among them, image M R The black areas represent the unoccluded regions, indicating that this area was not occluded in the original image, and the reconstructed image I should be rendered in this area during the final rendering. R In contrast, image M R The white areas in the image represent regions of the original input image that are occluded; these reconstructed images should not be rendered during the final rendering process. This is because 3D face reconstruction will always reconstruct a complete face regardless of whether it is occluded. Therefore, for similar... Figure 5 In the example, the self-occlusion of the middle hand and the occlusion by sunglasses will only result in reconstruction as shown in image I. R The image shows the entire face reconstructed without any occlusions. If a post-processing workflow involving 3D makeup is present, since the reconstructed 3D face is complete, the entire 3D makeup material will be preserved, without considering the presence of occlusions and masking any obstructed areas. Therefore, to address this common issue, one embodiment utilizes the predicted image M... R The black areas are used to identify occlusions. For the occluded parts, the final rendered image displays the input image. For the parts obscured by sunglasses and hands, the final rendered image I... F The image used is I T The sunglasses and hands in the image are rendered, thus avoiding the problem of failing to render occluders or incorrectly rendering 3D makeup onto occluders if there is 3D makeup post-processing.

[0070] Figure 6 This is a structural block diagram of a face reconstruction and occlusion region recognition device provided in an embodiment of this application. The device is used to execute the face reconstruction and occlusion region recognition method provided in the above embodiments, and has corresponding functional modules and beneficial effects for executing the method. Figure 6 As shown, the device specifically includes: a parameter vector generation module 101, a 3D information determination module 102, a rendering mapping module 103, a graph information construction module 104, and an occlusion region determination module 105, wherein...

[0071] The parameter vector generation module 101 is configured to input a face image into a first network structure and output the reconstruction parameter vector of the face image during three-dimensional reconstruction through the first network structure.

[0072] The three-dimensional information determination module 102 is configured to perform three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information;

[0073] The rendering mapping module 103 is configured to render and map the three-dimensional face information to generate two-dimensional face information containing network segmentation data;

[0074] The graph information construction module 104 is configured to obtain the depth feature map extracted by the first network structure, and construct graph structure information based on the depth feature map and the two-dimensional face information.

[0075] The occlusion region determination module 105 is configured to input the graph structure information into a second network structure and output the occlusion region corresponding to the face image through the second network structure.

[0076] As described above, the face image is input into a first network structure, which outputs a reconstruction parameter vector for 3D face image reconstruction. Based on this parameter vector, 3D face information is generated through 3D face image reconstruction. This 3D face information is then rendered and mapped to generate 2D face information containing network segmentation data. Further, the depth feature map extracted by the first network structure is obtained. Based on the depth feature map and the 2D face information, graph structure information is constructed and input into a second network structure. The second network structure then outputs the occlusion region corresponding to the face image. This face reconstruction and occlusion region recognition method combines face reconstruction and occlusion region recognition into a single model. In occlusion region recognition, relevant data from the face reconstruction process is utilized, modeling the occlusion problem on the 2D plane as a 3D face occlusion recognition problem, which is more consistent with human perception. Furthermore, by utilizing relevant 3D face data, it is easier to uncover hidden related features, resulting in more accurate occlusion region recognition results. Meanwhile, it fully utilizes the common features of face reconstruction and occlusion region recognition tasks, compressing the model as much as possible while enabling the two tasks to promote each other, optimizing the overall resource deployment, and meeting the application scenarios with many parallel tasks and high real-time requirements, with higher accuracy in occlusion region determination.

[0077] In one possible embodiment, the reconstructed parameter vector includes a face feature vector, a face expression vector, and a three-dimensional face coefficient vector, and the three-dimensional information determination module 102 is configured as follows:

[0078] Based on the facial feature vector, the facial expression vector, the three-dimensional facial coefficient vector, and the preset average facial shape information, a three-dimensional facial point cloud is reconstructed to obtain point cloud reconstruction information.

[0079] Based on the three-dimensional face coefficient vector and the preset average face texture information and face base information, face texture is reconstructed to obtain face texture information;

[0080] Three-dimensional face information is generated based on the point cloud reconstruction information and the face texture information.

[0081] In one possible embodiment, the rendering mapping module 103 is configured as follows:

[0082] A two-dimensional face image is obtained by rendering the three-dimensional face information using a renderer;

[0083] Based on the topological structure in the three-dimensional face information, the three-dimensional point pair data of the face in the three-dimensional face information is mapped to the two-dimensional face image to generate two-dimensional face information containing network segmentation data.

[0084] In one possible embodiment, the graph information construction module 104 is configured as follows:

[0085] Based on the network segmentation data in the two-dimensional face information, the depth feature map is segmented to obtain a triangular facet segmentation result.

[0086] The graph structure information is obtained by constructing an adjacency matrix based on the vertex connectivity and point-to-point distances in the triangular facet segmentation results.

[0087] In one possible embodiment, the occlusion area determination module 105 is configured as follows:

[0088] The graph convolutional neural network is used to classify the 3D face vertices in the graph structure information;

[0089] The occlusion region corresponding to the face image is output based on the classification result.

[0090] In one possible embodiment, the device further includes a special effects processing module configured to:

[0091] After the occlusion region corresponding to the face image is output through the second network structure, special effects rendering is performed based on the face image, the two-dimensional face information, and the occlusion region, and the processing result is displayed.

[0092] In one possible embodiment, the special effects processing module is configured as follows:

[0093] The occluded area is rendered using the original image occluder, while the facial image portion outside the occluded area is rendered based on the 3D facial information.

[0094] Figure 7 This is a schematic diagram of the structure of a face reconstruction and occlusion region recognition device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of processors 201 in the device can be one or more. Figure 7 Taking a processor 201 as an example; the processor 201, memory 202, input device 203, and output device 204 in the device can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, the memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the face reconstruction and occlusion region recognition method in this embodiment. The processor 201 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 202, thereby realizing the aforementioned face reconstruction and occlusion region recognition method. The input device 203 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 204 may include a display screen or other display device.

[0095] This application embodiment also provides a non-volatile storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a face reconstruction and occlusion region recognition method described in the above embodiments, wherein the instructions include:

[0096] A face image is input into a first network structure, and the first network structure outputs a reconstruction parameter vector of the face image during 3D reconstruction.

[0097] Based on the reconstruction parameter vector, the face image is reconstructed in three dimensions to generate three-dimensional face information;

[0098] The three-dimensional face information is rendered and mapped to generate two-dimensional face information containing network segmentation data;

[0099] Obtain the depth feature map extracted by the first network structure, and construct graph structure information based on the depth feature map and the two-dimensional face information;

[0100] The graph structure information is input into the second network structure, and the occlusion area corresponding to the face image is output through the second network structure.

[0101] It is worth noting that in the embodiments of the face reconstruction and occlusion area recognition device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0102] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can perform the face reconstruction and occlusion region recognition methods described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media.

Claims

1. A method of face reconstruction and occluded region identification, characterized in that, The method comprises the following steps: inputting a face image into a first network structure, and outputting a reconstruction parameter vector of the face image in three-dimensional reconstruction by the first network structure; performing three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information; performing rendering and mapping processing on the three-dimensional face information to generate two-dimensional face information containing network segmentation data, wherein the rendering and mapping processing comprises the following steps: rendering the three-dimensional face information by a renderer to obtain a two-dimensional face image, and mapping face three-dimensional point pair data in the three-dimensional face information to the two-dimensional face image based on a topological structure in the three-dimensional face information to generate two-dimensional face information containing network segmentation data; obtaining a deep feature map extracted by the first network structure, and constructing graph structure information based on the deep feature map and the two-dimensional face information; inputting the graph structure information into a second network structure, and outputting an occlusion area corresponding to the face image by the second network structure.

2. The face reconstruction and occlusion region identification method of claim 1, wherein, The reconstruction parameter vector comprises a face feature vector, a face expression vector and a three-dimensional face coefficient vector, and the three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information comprises the following steps: reconstructing a three-dimensional face point cloud based on the face feature vector, the face expression vector, the three-dimensional face coefficient vector and preset average face shape information to obtain point cloud reconstruction information; reconstructing a face texture based on the three-dimensional face coefficient vector and preset average face texture information and face base information to obtain face texture information; generating three-dimensional face information based on the point cloud reconstruction information and the face texture information.

3. The face reconstruction and occlusion region identification method according to any one of claims 1-2, characterized in that, The construction of the graph structure information based on the deep feature map and the two-dimensional face information comprises the following steps: segmenting the deep feature map based on network segmentation data in the two-dimensional face information to obtain a triangular facet segmentation result; constructing an adjacency matrix based on vertex connectivity and point pair distance in the triangular facet segmentation result to obtain graph structure information.

4. The face reconstruction and occlusion region identification method according to any one of claims 1-2, characterized in that, The second network structure comprises a graph convolutional neural network, and the outputting of the occlusion area corresponding to the face image by the second network structure comprises the following steps: classifying three-dimensional face vertices in the graph structure information by the graph convolutional neural network; outputting the occlusion area corresponding to the face image based on a classification result.

5. The face reconstruction and occlusion region identification method according to any one of claims 1-2, characterized in that, After the outputting of the occlusion area corresponding to the face image by the second network structure, the method further comprises the following steps: performing special effect rendering processing based on the face image, the two-dimensional face information and the occlusion area, and displaying a processing result.

6. The face reconstruction and occlusion region identification method of claim 5, wherein, The special effect rendering processing based on the face image, the two-dimensional face information and the occlusion area comprises the following steps: performing rendering on the occlusion area using an original image occluder, and performing rendering on a face image part outside the occlusion area based on the three-dimensional face information.

7. A device for face reconstruction and occluded region identification, characterized in that, The method comprises the following steps: a parameter vector generation module configured to input a face image into a first network structure, and output a reconstruction parameter vector of the face image in three-dimensional reconstruction by the first network structure; a three-dimensional information determination module configured to generate three-dimensional face information based on three-dimensional reconstruction of the face image according to the reconstruction parameter vector; a rendering and mapping module configured to generate two-dimensional face information containing network segmentation data by rendering and mapping the three-dimensional face information, and specifically configured to: render the three-dimensional face information by a renderer to obtain a two-dimensional face image, and map face three-dimensional point pair data in the three-dimensional face information to the two-dimensional face image based on a topology in the three-dimensional face information to generate two-dimensional face information containing network segmentation data; a graph information construction module configured to obtain a deep feature map extracted by the first network structure, and construct graph structure information based on the deep feature map and the two-dimensional face information; an occlusion region determination module configured to input the graph structure information into a second network structure, and output an occlusion region corresponding to the face image by the second network structure.

8. A face reconstruction and occluded region identification device, the device comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the face reconstruction and occlusion region identification method in any one of claims 1-6.

9. A non-transitory storage medium storing computer-executable instructions that, when executed by a computer processor, perform the face reconstruction and occlusion region identification method in any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, the computer program is executed by the processor to implement the face reconstruction and occlusion region identification method in any one of claims 1-6.

Images