3D face reconstruction method and system based on occlusion segmentation

Abstract

An embodiment of the present application provides a 3D face reconstruction method and system based on occlusion segmentation. The technical solution provided in the embodiment of the present application inputs a target face image into a pre-constructed parameter prediction model, which includes an image feature extractor and an image segmentation decoder. The parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until the associated loss functions of the image feature extractor and the image segmentation decoder reach a predetermined state. Then, the parameter prediction model outputs target face reconstruction parameters and target face occlusion regions of the target face image, and performs 3D face reconstruction post-processing based on the face reconstruction parameters and face occlusion segmentation regions. Using the above technical solutions can reduce the computational resource consumption caused by model deployment, reduce model redundancy, compress model calculations, and improve 3D face reconstruction efficiency.

Description

[Technical Field]

[0001] This application claims priority from Chinese Patent Application No. 202211286327.0, filed with the State Intellectual Property Office of China on October 20, 2022, the entire contents of which are incorporated herein by reference.

[0002] FIELD Embodiments of the present application relate to the field of computer technology, and more particularly to a method and system for 3D face reconstruction based on occlusion segmentation. [Background technology]

[0003] Currently, 3D facial reconstruction technology is widely applied in fields such as film and television, games, medicine, and live social media. For example, in the live social media field, a user's 2D facial image is acquired and 3D facial reconstruction technology is used to restore the user's 3D facial information (expression, texture), enabling functions such as 3D facial beautification and 3D makeup. In actual application scenarios, the user's 2D facial image does not always contain the entire face as expected and may be occluded by limbs or objects. Therefore, facial occlusion segmentation is required to locate the occluded area. Therefore, after 3D facial reconstruction, post-processing is performed based on the 3D facial reconstruction results and the facial occlusion segmentation results to ensure the implementation of functions such as 3D facial beautification and 3D makeup.

[0004] However, in the application scenario of 3D face reconstruction in related technologies, the 3D face reconstruction model and the face occlusion segmentation model are deployed independently, and the 3D face reconstruction and the occlusion area segmentation are processed as two independent tasks. Due to the limited computing power of the deployment platform, deploying multiple models at the same time will occupy too many computing resources, increase the computing stress of the platform, and affect the execution of the platform's computing tasks. Summary of the Invention [Means for solving the problem]

[0005] In the embodiments of the present application, a 3D face reconstruction method and system based on occlusion segmentation is provided, which reduces the computational resource consumption caused by model deployment in 3D face reconstruction application scenarios, compresses the model calculation amount, and solves the technical problem of excessive computational resource consumption in 3D face reconstruction application scenarios.

[0006] In a first aspect, an embodiment of the present application provides a 3D face reconstruction method based on occlusion segmentation, and the 3D face reconstruction method based on occlusion segmentation includes: Input a target face image into a pre-constructed parameter prediction model, the parameter prediction model including an image feature extractor and an image segmentation decoder, and train the parameter prediction model based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until associated loss functions of the image feature extractor and the image segmentation decoder reach a set state; outputting target face reconstruction parameters and target face occlusion regions of the target face image through the parameter prediction model, and performing 3D face reconstruction post-processing according to the face reconstruction parameters and the face occlusion segmentation regions.

[0007] In a second aspect, an embodiment of the present application provides a 3D face reconstruction system based on occlusion segmentation, the 3D face reconstruction system based on occlusion segmentation includes an input module and an output module; the input module is configured to input a target face image into a pre-constructed parameter prediction model, the parameter prediction model including an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until associated loss functions of the image feature extractor and the image segmentation decoder reach a set state; The output module is configured to output target face reconstruction parameters and target face occlusion regions of the target face image according to the parameter prediction model, and perform 3D face reconstruction post-processing according to the face reconstruction parameters and the face occlusion segmentation regions.

[0008] In a third aspect, an embodiment of the present application provides a 3D face reconstruction device based on occlusion segmentation, the 3D face reconstruction device based on occlusion segmentation comprising: a memory and one or more processors; the memory is configured to store one or more programs; The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the occlusion segmentation based 3D face reconstruction method of the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions being arranged, when executed by a computer processor, to perform the occlusion segmentation based 3D face reconstruction method as in the first aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product including instructions that, when executed by a computer or processor, cause the computer or processor to perform the occlusion segmentation based 3D face reconstruction method as in the first aspect.

[0011] In an embodiment of the present application, a target face image is input into a pre-constructed parameter prediction model, which includes an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until the associated loss functions of the image feature extractor and the image segmentation decoder reach a predetermined state, and then target face reconstruction parameters and target face occlusion regions of the target face image are output by the parameter prediction model, and 3D face reconstruction post-processing is performed based on the face reconstruction parameters and the face occlusion segmentation regions. By employing the above technical means to train the parameter prediction model, which includes the image feature extractor and the image segmentation decoder, until the associated loss functions of the image feature extractor and the image segmentation decoder reach a predetermined state, the parameter prediction model can incorporate 3D face reconstruction and face occlusion segmentation functions, thereby reducing the computational resource consumption due to model deployment, reducing model redundancy, compressing model calculations, and improving 3D face reconstruction efficiency.

[0012] In addition, the embodiments of the present application can adaptively allocate the computational load of the parameter prediction model by customizing the parameter dimension of the target face reconstruction parameters and the number of channels of the image segmentation decoder, and can adapt the parameter prediction model to deployment environments supported by different computing power. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a flowchart of a 3D face reconstruction method based on occlusion segmentation provided in an embodiment of the present application; [Figure 2] 1 is a flowchart illustrating a parameter prediction model training process according to an embodiment of the present application. [Figure 3] FIG. 1 is a sample input / output schematic diagram of a parameter prediction model in an embodiment of the present application. [Figure 4] 1 is a flowchart of image pre-processing in an embodiment of the present application. [Figure 5] 1 is a prediction flowchart of a parameter prediction model in an embodiment of the present application. [Figure 6]10 is a flowchart of a target face image processing according to an embodiment of the present invention. [Figure 7] FIG. 1 is a structural schematic diagram of a 3D face reconstruction system based on occlusion segmentation provided in an embodiment of the present application. [Figure 8] FIG. 1 is a structural schematic diagram of a 3D face reconstruction device based on occlusion division provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0014] To clarify the objectives, technical solutions, and advantages of the present application, specific embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein do not limit the present application, but are used only to explain the present application. For convenience of explanation, the drawings show only parts relevant to the present application, rather than all of the contents. Before describing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods illustrated in flowcharts. Although the flowcharts show operations (or steps) as sequential processes, many of the operations can occur in parallel, simultaneously, or be performed simultaneously. The order of the operations can be rearranged. The process can be terminated upon completion of the operation, but may further include additional steps not shown in the drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0015] The 3D face reconstruction method based on occlusion segmentation provided herein aims to train a parameter prediction model that combines an image feature extractor and an image segmentation decoder, enabling the parameter prediction model to combine the functions of 3D face reconstruction and facial occlusion region segmentation. This reduces the computational resource consumption caused by model deployment and compresses the model calculation volume. In traditional 3D face reconstruction scenarios, to avoid facial occlusion affecting the 3D face post-processing function during 3D face reconstruction, an independent facial occlusion segmentation model is deployed to predict facial occlusion regions, and 3D face post-processing is then performed based on the located facial occlusion regions. Because the 3D face reconstruction model and the facial occlusion segmentation model are deployed independently, redundant image processing steps exist between them. Separating 3D face reconstruction and occlusion region processing into two independent tasks increases the computing stress of the platform and affects the performance of other tasks on the platform. Based on this, the embodiments of the present application provide a solution to the technical problem of excessive computational resource occupation in the current 3D face reconstruction application scene by processing 3D face reconstruction and occluded area segmentation as two independent tasks.

[0016] Working Example:

[0017] 1 shows a flowchart of a 3D face reconstruction method based on occlusion segmentation provided in an embodiment of the present application, which can be performed by a 3D face reconstruction device based on occlusion segmentation, which can be realized in a software and / or hardware manner, and which can be configured with two or more physical entities, or can be configured with one physical entity. Generally, the 3D face reconstruction device based on occlusion segmentation can be a processing device such as a computer, a mobile phone, a tablet, an image processing server, etc.

[0018] The following description will be given taking the occlusion segmentation-based 3D face reconstruction device as an example of executing the occlusion segmentation-based 3D face reconstruction method. Referring to Figure 1, the occlusion segmentation-based 3D face reconstruction method specifically includes:

[0019] S110: input a target face image into a pre-constructed parameter prediction model, the parameter prediction model including an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until associated loss functions of the image feature extractor and the image segmentation decoder reach a set state.

[0020] In an embodiment of the present application, when performing 3D face reconstruction, a 2D face image ready to be generated as a 3D face image is input into a pre-constructed parameter prediction model, and the 2D face image is defined as a target face image. The parameter prediction model predicts the 3D face reconstruction parameters and facial occlusion areas of the target face image, and defines them as target face reconstruction parameters and target facial occlusion areas. The parameter prediction model combines an image feature extractor and an image segmentation decoder and is trained based on the associated loss function between them, thereby enabling parallel processing of 3D face reconstruction and occlusion area segmentation tasks, improving model processing efficiency and mutually promoting effectiveness.

[0021] Prior to this, the parameter prediction model is pre-trained so that the parameter prediction model can perform the 3D face reconstruction and occlusion region segmentation tasks. Referring to Figure 2, the training process of the parameter prediction model is as follows: S1001, a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions are taken as training samples; S1002, training a parameter prediction model according to the training sample, outputting corresponding face prediction keypoint information by an image feature extractor, outputting face prediction occlusion segmentation region by an image segmentation decoder, and performing 3D face reconstruction according to the face prediction keypoint information to generate a 3D face prediction image; S1003, taking the 3D face prediction image, the face prediction keypoint information, and the face prediction occlusion segmentation region as prediction samples, calculating the related loss function of the image feature extractor and the image segmentation decoder according to the training samples and the prediction samples, and completing the training process of the parameter prediction model when the related loss function reaches a set state; Includes:

[0022] When training a parameter prediction model in an embodiment of the present application, a plurality of facial pictures are input as facial training pictures, facial keypoint information of each facial training picture is obtained, and a facial occlusion segmentation region of the facial training picture is determined by a pre-trained facial region segmentation model, and the above-mentioned plurality of facial training pictures, the facial keypoint information of the facial training pictures, and the facial occlusion segmentation region are used as training samples to train the image feature extractor and image segmentation decoder of the parameter prediction model.

[0023] Here, a facial training picture is input to an image feature extractor, which outputs corresponding facial prediction keypoint information. The facial prediction keypoint information is then input to a 3D facial reconstruction model to perform 3D facial reconstruction, and a 3D facial model is determined based on the facial prediction keypoint information. A differentiable renderer projects the 3D facial model onto a 2D plane and renders it into a 2D image, thereby obtaining a corresponding predicted rendered image, i.e., a 3D facial prediction image. Meanwhile, the image feature extractor extracts image features based on the facial training picture, which are then input to an image segmentation decoder to perform facial image segmentation, i.e., output a facial prediction occlusion segmentation region. The parameter prediction model uses the 3D facial prediction image, facial prediction keypoint information, and facial prediction occlusion segmentation region as prediction samples, calculates associated loss functions for the image feature extractor and image segmentation decoder based on the training sample and the predicted sample, and determines that the parameter prediction model training is complete when the associated loss function reaches a set state.

[0024] It should be noted that for face training pictures in the training samples, the face region of the picture may include occlusion regions or may not include face occlusion regions, and training the parameter prediction model on face training pictures with different occlusion situations can improve the stability and reliability of model prediction.

[0025] The embodiment of this application aims to gradually make the predicted samples closer to the training originals by designing a related loss function, and when the related loss function is in a set state, the similarity between the training samples and the predicted samples meets the model prediction standard and can be applied to 3D face reconstruction.

[0026] For example, as shown in FIG. 3, face training picture I T , face keypoint information of face training picture Im T and the face occlusion segmentation region M T Based on the training process of the above model, the corresponding 3D face prediction image I R , face prediction keypoint information Im R and the face predicted occlusion segmentation region M R The 3D face prediction image I R reconstructs only the unoccluded parts of the face, and does not reconstruct the corresponding occluded parts in the case of occlusions. As shown in Figure 3, the predicted 3D face image I R does not reconstruct the eye-occluding sunglasses, which can avoid the interference of the face occlusion area with the reconstructed 3D face and optimize the subsequent 3D face reconstruction post-processing effect.

[0027] In an embodiment, the associated loss functions of the parameter prediction model include a segmentation loss function, a segmentation scaling loss function, and a face reconstruction loss function, where the segmentation loss function is used to measure the difference between the face occluded segmentation region and the corresponding face predicted occluded segmentation region, the segmentation scaling loss function is used to make a scaling adjustment to the face predicted occluded segmentation region, and the face reconstruction loss functionis used to measure the difference between the face training picture and the corresponding 3D face prediction image.

[0028] In this embodiment, a segmentation scaling loss function is introduced to combine the loss function related to face segmentation and the loss function related to face reconstruction, establishing a connection between 3D face reconstruction and face occlusion segmentation. This makes the prediction of face reconstruction parameters more stable in the presence of occlusion, while also making face occlusion segmentation more accurate. The model training is completed so that the image feature extractor and the image segmentation decoder mutually promote each other.

[0029] The partition scaling loss function also includes a partition region expansion function and a partition region shrinkage function, where the partition region expansion function is used to expand the face predicted occlusion partition region and the partition region shrinkage function is used to shrink the face predicted occlusion partition region.

[0030] Specifically, the split loss function is

number

[0031] The split scaling loss function is

number

[0032] where:

number

number

number

[0033] Combining the above formulas (1) to (6), the cross-entropy loss can be used to ensure the basic contour of the face prediction occlusion segmentation region, while the segmentation scaling loss function can be used to refine the face prediction occlusion segmentation region. Furthermore, the segmentation scaling loss function can be used to establish a connection with the face reconstruction part, and at the same time, when adding the face prediction occlusion segmentation region, the perceptual layer of face reconstruction and pixel-level error can be taken into account, so that the 3D face reconstruction and face occlusion segmentation tasks can be performed in parallel, and the effects of both can be mutually enhanced.

[0034] The face reconstruction loss function is

number

[0035] where Equation (7) is the face training picture I T and 3D face prediction image I R and the pixel level should be similar in the unoccluded part. T and face prediction keypoint information Im R and should be as similar as possible. Equation (9) expresses that the face training picture I T and 3D face prediction image I R represents that the model perception should be similar.

[0036] When the parameter prediction model is trained based on the associated loss functions of the image feature extractor and the image segmentation decoder until the associated loss functions reach a set state, as shown in the associated loss function formulas (1)-(9) above, when they converge to the set value, it indicates that the training of the parameter prediction model is completed and the prediction result of the parameter prediction model has reached the expected standard.

[0037] In this embodiment, 3D face reconstruction and face segmentation tasks are combined to realize simultaneous output of 3D face reconstruction parameters and face occlusion regions in one model, thereby reducing the overhead of model deployment. Furthermore, by combining three types of loss functions, namely, segmentation loss function, segmentation scaling loss function, and face reconstruction loss function, the model learns the inherent connection between 3D face reconstruction and face occlusion segmentation tasks, which can greatly reduce model redundancy and allow the use of smaller models to complete the 3D face reconstruction and face occlusion segmentation tasks. This reduces the model computational complexity and improves model processing efficiency.

[0038] In an embodiment, when performing 3D face reconstruction and face occlusion segmentation of a target face image based on the already constructed parameter prediction model, the embodiment of the present application performs preprocessing on the target face image, and inputs the preprocessed target face image into the parameter prediction model to perform 3D face reconstruction and face occlusion segmentation.

[0039] Referring to FIG. 4, the pre-processing process of the target face image is as follows: S1101, aligning a target face image based on a face keypoint detector and a template face keypoint, and obtaining elongation and translation parameters of the target face image; S1102, cropping the target face image based on the elongation and translation parameters to fit the target face image to standard face dimensions; Includes:

[0040] The preprocessing of the target face image mainly involves filtering and calibrating the picture data to be input into the parameter prediction model. A facial keypoint detector is used to align the template facial keypoints with the target face image to obtain the stretching and translation parameters of the preprocessed target face image. The target face image is then processed and cropped using the corresponding parameters to conform to standard facial dimensions, thereby facilitating the subsequent use of the parameter prediction model. It can be understood that different target face images have different dimensions of facial regions in the images. Therefore, to ensure the prediction effect of the parameter prediction model, the parameter prediction model needs to standardize and process the target face image to adjust the facial region of the target face image to standard facial dimensions.

[0041] Thereafter, for the target face image for which preprocessing has been completed, parameter prediction is performed based on the parameter prediction model for which pre-training has been completed.

[0042] S120: outputting target face reconstruction parameters and a target face occlusion region of the target face image according to the parameter prediction model; and performing 3D face reconstruction post-processing according to the target face reconstruction parameters and the target face occlusion region.

[0043] The parameter prediction model of the present embodiment receives the preprocessed target face image as input, and simultaneously outputs the corresponding target face reconstruction parameters and target face occlusion regions through model prediction.

[0044] Specifically, in the parameter prediction model, a target face image is input to an image feature extractor to obtain a corresponding feature map, and the feature map is integrated to obtain target face reconstruction parameters. The feature map is input to an image segmentation decoder, and the image segmentation decoder performs image segmentation to obtain the target face occlusion area.

[0045] The overall framework of the parameter prediction model is shown in Figure 5. In this embodiment, an improved lightweight mobilenet-v3 network is used as an image-level feature extractor to learn the complete 3D facial structure geometry from image pixels, making it more suitable for deployment on mobile end devices. At the same time, this embodiment uses a lightweight image segmentation decoder, LR-ASPP, connected to the image-level feature extractor to efficiently extract deep features and detailed information, achieving efficient image segmentation. Figure 5 shows the detailed structure of the parameter prediction model in this embodiment. The parameter prediction model takes a preprocessed target face image as input, passes through a series of bneck blocks to obtain a series of yellow feature maps, and finally integrates the extracted features through 1x1 convolution to output a parameter prediction vector, i.e., the target face reconstruction parameters.

[0046] The core component of the image-level feature extractor is the bneck module, which mainly implements channel-separable convolution, SE channel attention mechanism, and residual connection. Channel-separable convolution allows the model to use fewer parameters and achieve better feature extraction results. SE channel attention mechanism is used to adjust the weights of each channel, and residual connection allows the model to better combine high- and low-level features, laying the foundation for the model to learn 3D face parameters.

[0047] In addition, in this embodiment, an image-level feature extractor capable of capturing 3D facial features is connected to the image segmentation decoder LR-ASPP, which performs the face segmentation task, to simultaneously output target face reconstruction parameters and target face occlusion segmentation regions. The image segmentation decoder LR-ASPP takes 56x56 and 7x7 feature maps as input, and performs further feature recalibration on the high-level feature map (7x7) using an SE channel attention mechanism. Then, high- and low-resolution features are classified using 1x1 convolution, respectively, and then blended. Multi-level blended features are used for learning to achieve accurate segmentation of the moving end image. Finally, the target face occlusion segmentation region is obtained.

[0048] As one option, the parameter dimensions of the target face reconstruction parameters output by the image feature extractor and the number of channels of the image segmentation decoder correspond to the model computational power allocation of the parameter prediction model. The dimensions of the target face reconstruction parameters ultimately output by the present embodiment can be freely defined by the user, and the user can freely specify the required model size and effect in combination during the training phase. The final output dimensions are equal to the sum of identity (face ID) + expression (facial expression) + albedo (facial texture) + lighting (27 dimensions) + pose (3D) + translation (3D).

[0049] Additionally, a single width parameter can control the computational complexity of the entire model structure, which in turn controls the number of channels in the entire model. Actual measurements show that when width = 0.5, the entire model can be compressed to 20 MFLOPS. Face reconstruction and occlusion segmentation on the evaluation set performed very well, demonstrating that this method can be deployed on a variety of low-end devices and meet the needs of modules with different computational power configurations. Based on the specific needs of the actual application scenario and deployment environment, the parameter dimensions of identity (face ID), expression (facial expression), and albedo (facial texture), as well as the width parameter, can be used to control the computational complexity of the final model, allowing for the generation of customized models tailored to actual needs and improving the flexibility of model design.

[0050] Referring to steps a1-a5 provided in Figure 6, based on the above parameter prediction model, the preprocessed target face image is input into the parameter prediction model, so that image features are obtained by the image feature extractor of the parameter prediction model, and the image features are on the one hand used to generate target face reconstruction parameters, and on the other hand input into the image segmentation decoder to generate target face occlusion areas, and further output the target face reconstruction parameters and target face occlusion areas for 3D face reconstruction post-processing to complete parameter prediction.

[0051] In an embodiment, when performing 3D face reconstruction post-processing, the embodiment of the present application performs 3D face reconstruction based on target face reconstruction parameters to generate a target 3D face model, the target 3D face model includes a target 3D face shape and a target 3D face texture, and for occluded areas in the target 3D face model, rendering is performed using a target face image based on the target face occluded areas, and for non-occluded areas in the target 3D face model, rendering is performed using a target material.

[0052] Based on the target face reconstruction parameters, the pre-generated face model bases are combined to reconstruct a 3D face shape and a 3D face texture, thereby generating a target 3D face model.

[0053] The formula for constructing the target 3D face model is:

number

[0054] where S is the 3D face shape and T is the 3D face texture,

number

number

number

number

[0055] Other parameters output by the parameter prediction model, such as posture and translation parameters, can be used to perform posture correction on the reconstructed 3D face model, and lighting parameters can be used to process spherical harmony on the reconstructed face texture, making the results more lifelike and detailed.

[0056] The 3D face reconstruction post-processing of the embodiment of the present application will be described using 3D makeup in a live broadcast scene as an example. During live broadcast 3D makeup, 3D makeup materials can be pasted based on the 3D face model reconstructed from the user's face image, and the final rendering texture is calculated based on the target face occlusion area predicted by the parameter prediction model. For the occluded area, the original image (and target face image) captured by the camera can be selected for rendering, and for the non-occluded area, the 3D makeup material can be rendered according to normal logic. This achieves the goal of "using the original image captured by the camera to render the occluded area and using the reconstructed 3D makeup to render the non-occluded area," allowing the user to enjoy the visual effect of beautiful makeup while avoiding the makeup effect from floating on occlusions and optimizing the 3D makeup effect.

[0057] In actual application scenarios, the 3D face reconstruction method of the embodiments of the present application can be used in any 3D face reconstruction application scenario that processes inputs that may involve occlusion in real time, such as live broadcasting, 3D beauty makeup in conference scenes, 3D special effects, medical plastic modeling, etc. The embodiments of the present application fix limitations on specific application environment steps, and detailed descriptions are omitted here.

[0058] As described above, a target face image is input into a pre-constructed parameter prediction model, which includes an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until the associated loss functions of the image feature extractor and the image segmentation decoder reach a set state, and then target face reconstruction parameters and target face occlusion regions of the target face image are output by the parameter prediction model, and 3D face reconstruction post-processing is performed based on the face reconstruction parameters and the face occlusion segmentation regions. By employing the above technical means to train the parameter prediction model, which includes the image feature extractor and the image segmentation decoder, until the associated loss functions of the image feature extractor and the image segmentation decoder reach a set state, the parameter prediction model can combine 3D face reconstruction and face occlusion segmentation functions, thereby reducing the computational resource occupation due to model deployment, reducing model redundancy, compressing model calculations, and improving 3D face reconstruction efficiency.

[0059] In addition, the embodiments of the present application can adaptively allocate the computational load of the parameter prediction model by customizing the parameter dimension of the target face reconstruction parameters and the number of channels of the image segmentation decoder, and can adapt the parameter prediction model to deployment environments supported by different computing power.

[0060] Based on the above embodiment, Fig. 7 is a structural schematic diagram of the 3D face reconstruction system based on occlusion segmentation provided in the present application. Referring to Fig. 7, the 3D face reconstruction system based on occlusion segmentation provided in the present embodiment specifically includes an input module and an output module.

[0061] Here, the input module 21 is configured to input a target face image into a pre-constructed parameter prediction model, which includes an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until the associated loss functions of the image feature extractor and the image segmentation decoder reach a set state.

[0062] The output module 22 is configured to output target face reconstruction parameters and target face occlusion regions of the target face image according to the parameter prediction model, and perform 3D face reconstruction post-processing according to the face reconstruction parameters and the face occlusion segmentation regions.

[0063] Specifically, the training process of the parameter prediction model is as follows: Taking a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions as training samples; According to the training sample, training a parameter prediction model, outputting corresponding face prediction keypoint information by an image feature extractor, outputting face prediction occlusion segmentation region by an image segmentation decoder, and performing 3D face reconstruction according to the face prediction keypoint information to generate a 3D face prediction image; The method includes: taking the 3D face prediction image, the face prediction keypoint information, and the face prediction occlusion segmentation region as prediction samples; calculating the associated loss functions of the image feature extractor and the image segmentation decoder based on the training samples and the prediction samples; and completing the training process of the parameter prediction model when the associated loss functions reach a set state.

[0064] Here, the related loss functions include a segmentation loss function, a segmentation scaling loss function, and a face reconstruction loss function, where the segmentation loss function is used to measure the difference between the face occluded segmented region and the corresponding face predicted occluded segmented region, the segmentation scaling loss function is used to make a scaling adjustment to the face predicted occluded segmented region, and the face reconstruction loss function is used to measure the difference between the face training picture and the corresponding 3D face prediction image.

[0065] The segmentation scaling loss function includes a segmentation region expansion function and a segmentation region shrinkage function, where the segmentation region expansion function is used to expand the face predicted occluded segmentation region and the segmentation region shrinkage function is used to shrink the face predicted occluded segmentation region.

[0066] Specifically, the input module 21 is configured to input the target face image into an image feature extractor, obtain a corresponding feature map, integrate the feature map to obtain target face reconstruction parameters, input the feature map into an image segmentation decoder, perform image segmentation through the image segmentation decoder, and obtain the target face occlusion region.

[0067] The parameter dimension of the target face reconstruction parameters output by the image feature extractor and the number of channels of the image segmentation decoder correspond to the model computational power allocation of the parameter prediction model.

[0068] Specifically, before inputting the target face image into the pre-constructed parameter prediction model, Aligning a target face image based on the face keypoint detector and the template face keypoints to obtain elongation and translation parameters of the target face image; Cropping the target facial image based on the elongation and translation parameters to fit the target facial image to standard facial dimensions; Includes:

[0069] Specifically, the output module 22 is configured to perform 3D face reconstruction based on the target face reconstruction parameters and generate a target 3D face model, the target 3D face model including a target 3D face shape and a target 3D face texture, and to render occluded areas in the target 3D face model using a target face image based on the target face occluded areas, and to render non-occluded areas in the target 3D face model using a target material.

[0070] As described above, a target face image is input into a pre-constructed parameter prediction model, which includes an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until the associated loss functions of the image feature extractor and the image segmentation decoder reach a set state, and then target face reconstruction parameters and target face occlusion regions of the target face image are output by the parameter prediction model, and 3D face reconstruction post-processing is performed based on the face reconstruction parameters and the face occlusion segmentation regions. By employing the above technical means to train the parameter prediction model, which includes the image feature extractor and the image segmentation decoder, until the associated loss functions of the image feature extractor and the image segmentation decoder reach a set state, the parameter prediction model can incorporate 3D face reconstruction and face occlusion segmentation functions, thereby reducing the computational resource consumption caused by model deployment, reducing model redundancy, compressing model calculations, and improving 3D face reconstruction efficiency.

[0071] In addition, the embodiments of the present application can adaptively allocate the computational load of the parameter prediction model by customizing the parameter dimension of the target face reconstruction parameters and the number of channels of the image segmentation decoder, and can adapt the parameter prediction model to deployment environments supported by different computing power.

[0072] The 3D face reconstruction system based on occlusion segmentation provided in the embodiments of the present application is configured to perform the 3D face reconstruction method based on occlusion segmentation provided in the above embodiments, and can have corresponding functions and beneficial effects.

[0073] Based on the above practical example, an embodiment of the present application further provides a 3D face reconstruction device based on occlusion segmentation. Referring to FIG. 8 , the 3D face reconstruction device based on occlusion segmentation includes a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory 32 is a computer-readable storage medium configured to store software programs, computer-executable programs, and modules (e.g., program instructions / modules corresponding to the 3D face reconstruction method based on occlusion segmentation described in any embodiment of the present application (e.g., the input module and output module in the 3D face reconstruction system based on occlusion segmentation)). The communication module 33 is configured to perform data transfer. The processor 31 executes the software programs, instructions, and modules stored in the memory to perform various function applications and data processing of the device, i.e., to realize the above 3D face reconstruction method based on occlusion segmentation. The input device 34 can be configured to receive input numeric or character information and key signal inputs related to user configuration and function control of the device. The output device 35 can include a display device such as a display. The occlusion segmentation-based 3D face reconstruction device provided above is configured to perform the occlusion segmentation-based 3D face reconstruction method provided in the above embodiments, and can have corresponding functions and beneficial effects.

[0074] Based on the above embodiment, the present invention further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a computer processor, are configured to perform a 3D face reconstruction method based on occlusion segmentation, and the storage medium may be any type of memory device or storage device. Of course, the computer-executable instructions stored in the computer-readable storage medium provided in the embodiment of the present invention are not limited to the 3D face reconstruction method based on occlusion segmentation, and can also perform related operations in the 3D face reconstruction method based on occlusion segmentation provided in any embodiment of the present invention.

[0075] Based on the above embodiments, the embodiments of the present application further provide a computer program product, wherein the technical solution of the present application, essentially or in part contributing to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, the computer program product being stored in a storage medium and including several instructions, causing a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the 3D face reconstruction method based on occlusion segmentation of each embodiment of the present application. [Explanation of symbols]

[0076] 21 Input Module 22 Output Module 31 processors 32 memory 33 Communication Module 34 Input Devices 35 Output Device

Claims

1. Input a target face image into a pre-constructed parameter prediction model, the parameter prediction model including an image feature extractor and an image segmentation decoder, and train the parameter prediction model based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until associated loss functions of the image feature extractor and the image segmentation decoder reach a set state; outputting target face reconstruction parameters and a target face occlusion region of the target face image by the parameter prediction model, and performing 3D face reconstruction post-processing based on the target face reconstruction parameters and the target face occlusion region; Including, A 3D face reconstruction method based on occlusion segmentation.

2. The training process of the parameter prediction model includes: A plurality of the face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions are used as training samples; training the parameter prediction model based on the training sample, outputting corresponding face prediction keypoint information through the image feature extractor, outputting face prediction occlusion segmentation regions through the image segmentation decoder, and performing 3D face reconstruction based on the face prediction keypoint information to generate a 3D face prediction image; Using the 3D face prediction image, the face prediction keypoint information, and the face prediction occlusion segmentation region as prediction samples, calculate related loss functions of the image feature extractor and the image segmentation decoder based on the training samples and the prediction samples, and complete the training process of the parameter prediction model when the related loss function reaches a set state; The occlusion segmentation based 3D face reconstruction method according to claim 1.

3. The associated loss functions include a partitioning loss function, a partitioning scaling loss function, and a face reconstruction loss function; the segmentation loss function is used to measure the difference between the face occlusion segmentation region and the corresponding face prediction occlusion segmentation region; the segmentation scaling loss function is used to perform scaling adjustments on the face prediction occlusion segmentation regions; the face reconstruction loss function is used to measure the difference between the face training picture and the corresponding 3D face prediction image; The occlusion segmentation based 3D face reconstruction method according to claim 2.

4. the segmentation scaling loss function includes a segmentation region expansion function and a segmentation region shrinkage function, the segmentation region expansion function is used to expand the face prediction occluded segmentation region, and the segmentation region shrinkage function is used to shrink the face prediction occluded segmentation region; The occlusion segmentation based 3D face reconstruction method according to claim 3.

5. outputting target face reconstruction parameters and target face occlusion regions of the target face image according to the parameter prediction model; inputting the target face image into the image feature extractor to obtain a corresponding feature map; integrating the feature maps to obtain target face reconstruction parameters; inputting the feature map into the image segmentation decoder; performing image segmentation by the image segmentation decoder; and obtaining a target face occlusion region; The method for 3D face reconstruction based on occlusion segmentation according to any one of claims 1 to 4.

6. the parameter dimension of the target face reconstruction parameters output by the image feature extractor and the number of channels of the image segmentation decoder correspond to the model computational power allocation of the parameter prediction model; The method for 3D face reconstruction based on occlusion segmentation according to any one of claims 1 to 4.

7. Before inputting the target face image into the pre-constructed parameter prediction model, aligning the target face image based on a face keypoint detector and template face keypoints to obtain elongation and translation parameters of the target face image; Cropping the target facial image based on the elongation and translation parameters to fit the target facial image to standard facial dimensions; Including, The method for 3D face reconstruction based on occlusion segmentation according to any one of claims 1 to 4.

8. performing 3D face reconstruction post-processing based on the target face reconstruction parameters and the target face occlusion region; performing 3D face reconstruction based on the target face reconstruction parameters to generate a target 3D face model, the target 3D face model including a target 3D face shape and a target 3D face texture; performing rendering using the target face image for an occluded region in the target three-dimensional face model based on the target face occluded region, and performing rendering using target material for an unoccluded region in the target three-dimensional face model; Including, The method for 3D face reconstruction based on occlusion segmentation according to any one of claims 1 to 4.

9. an input module and an output module, the input module is configured to input a target face image into a pre-constructed parameter prediction model, the parameter prediction model including an image feature extractor and an image segmentation decoder, and the parameter prediction model is trained based on a plurality of face training pictures, face keypoint information of the face training pictures, and face occlusion segmentation regions until associated loss functions of the image feature extractor and the image segmentation decoder reach a set state; the output module is configured to output target face reconstruction parameters and target face occlusion regions of the target face image according to the parameter prediction model, and perform three-dimensional face reconstruction post-processing according to the target face reconstruction parameters and the target face occlusion segmentation regions; A 3D face reconstruction system based on occlusion segmentation.

10. a memory configured to store one or more programs; and one or more processors; The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the 3D face reconstruction method based on occlusion segmentation according to any one of claims 1 to 4. A 3D face reconstruction device based on occlusion segmentation.

11. computer-executable instructions are stored; The computer-executable instructions, when executed by a computer processor, are arranged to perform the occlusion segmentation based 3D face reconstruction method according to any one of claims 1 to 4. A computer-readable storage medium.

12. When executed by a computer or processor, causes the computer or processor to perform a 3D face reconstruction method based on occlusion segmentation according to any one of claims 1 to 4. Computer program.

Images