Method and device for estimating face optical flow based on neural non-rigid registration

Abstract

The present invention discloses a method and device for estimating facial optical flow based on neural non-rigid registration, which can quickly, accurately and with high quality estimate the facial optical flow between two frames of face RGB images before and after input. With the help of differentiable non-rigid registration, a loss function is designed, and the parameters of the optical flow estimation network and the optical flow confidence estimation network are reversely optimized by optimizing the loss function. The depth and facial structure priors provided by the non-rigid registration process promote the optical flow estimation network to generate more accurate optical flow. In order to stabilize the network training process, the initial correspondence obtained by 3D face reconstruction and key point detection is used to guide the non-rigid registration process. The face parsing prior is further introduced to improve the accuracy of optical flow estimation by providing the network with facial semantic information.

Description

Technical Field

[0001] The present invention belongs to the technical field of face image processing, and in particular relates to a face optical flow estimation method and device based on neural non-rigid registration. Background Art

[0002] With the development of image processing technology in recent years, the estimation of optical flow between monocular face video frames is widely used in applications such as 3D face reconstruction, facial expression recognition and inter-frame interpolation. How to calculate accurate and robust optical flow has become a widely concerned issue.

[0003] In the past, there were three main methods for optical flow estimation: the traditional method based on optimization (represented by the Lucas-Kanade algorithm), based on the assumption of constant brightness and similarity of neighborhood optical flow, the corresponding energy function is constructed, and the optical flow is obtained by iterative solution using a suitable optimization algorithm. In order to solve the problem of optical flow estimation in fast motion, a method is proposed to use a pyramid structure to solve the optical flow layer by layer. This type of traditional method is still widely used in some simple scenes (unchanged brightness, etc.). However, the traditional method based on optimization is not robust and has a slow solution speed for pictures taken in some complex scenes (brightness changes, too fast motion). The optical flow estimation method based on supervised learning obtains a network that maps from the previous and next two frames of pictures to the optical flow through a large amount of labeled or synthetic data and supervised training. Compared with the traditional method based on optimization, this method does not need to solve complex optimization problems for each input picture, and does not need to post-process the final result separately, which saves manpower and time. However, since this method requires "black box" training of many pairs of frame pairs-optical flow data, its final effect and stability are still difficult to guarantee. This method relies on large-scale labeled data, but the labeling of optical flow data is extremely expensive, and the network trained based on synthetic data cannot be effectively generalized to real data, which lacks practicality. The optical flow estimation method based on unsupervised learning is to change the supervised loss function in the supervised learning method into an unsupervised loss function. This type of function mainly measures the brightness (or local structure) difference of corresponding points in two pictures after the current optical flow calculation. This method gets rid of the dependence on real optical flow data and has better application prospects.

[0004] The task of facial optical flow estimation faces the following challenges: 1) The face contains large areas of similar textures, and optical flow estimation is prone to fall into local optimal solutions; 2) The face movement is complex (including changes in facial posture and various facial expressions). There are two main solutions to this problem. One is to input a facial video sequence, embed the face's motion trajectory into a low-dimensional space, and then jointly optimize the multi-frame optical flow. Although it can effectively preserve the continuity of facial structure and motion, this type of method is extremely time-consuming and requires the entire video to be input. Another method focuses on constructing a facial optical flow dataset in various ways, and then performing supervised training based on the dataset. For example, there is a method to construct optical flow through 3D face reconstruction based on 3DMM, but the accuracy of the optical flow constructed in this way is limited by the expression ability of 3DMM. There are also methods to construct optical flow through non-rigid registration of point clouds or registration of images, but without further post-processing, the accuracy of this type of optical flow cannot be guaranteed. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and provide a method and device for estimating facial optical flow based on neural non-rigid registration, which can quickly and robustly estimate facial optical flow without relying on a data set with optical flow annotations.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present specification, a method for estimating face optical flow based on neural non-rigid registration is provided, the method comprising:

[0008] Construct a training data set: collect RGB-D face video sequences, select several frame pairs, perform face analysis, 3D face reconstruction, and face key point detection on each frame, and construct a deformation map for non-rigid registration for the first frame of each frame pair to form a training data set;

[0009] Construct an optical flow estimation network and an optical flow confidence estimation network, use the outputs of the two networks together for differentiable 3D point cloud non-rigid registration, design a loss function, and reversely optimize the network parameters of the two networks by optimizing the loss function on the training dataset;

[0010] Input the source frame source RGB face image and the target frame target RGB face image into the trained optical flow estimation network to obtain the optical flow from the source frame source to the target frame target, that is, the movement of each pixel. By estimating the optical flow of any two frames in the video, it can assist in realizing tasks such as 3D face reconstruction and facial expression recognition.

[0011] Furthermore, in the process of constructing the training data set,

[0012] The face analysis specifically includes: using any face analysis technology based on a neural network to obtain a segmented image that conforms to corresponding semantics, wherein the segmentation elements are: background area, facial skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, mouth, left ear, and right ear;

[0013] The 3D face reconstruction uses any face reconstruction technology based on a parametric face model and a neural network;

[0014] The facial key point detection uses any facial key point detection technology based on a neural network.

[0015] Furthermore, the input of the optical flow estimation network is the two frames of RGB images of the face, and the output is the optical flow from the source frame source to the target frame target and the optical flow features; the input of the optical flow confidence estimation network is the two frames of RGB images of the face, the optical flow from the source frame source to the target frame target and the optical flow features output by the optical flow estimation network, and the output is the confidence of the optical flow.

[0016] Furthermore, the optical flow estimation network may adopt: an optical flow learning network PWC-Net based on a pyramid structure, and an optical flow learning network RAFT based on a cyclic network structure.

[0017] Furthermore, the optical flow confidence estimation network is a convolutional neural network model, including an input layer, a convolutional layer, a nonlinear layer and an output layer.

[0018] Furthermore, the input of the non-rigid registration is optimized, specifically: the vertices of the 3D face reconstruction result are sampled, and the initial correspondence between the source frame and the target frame provided by the sampling result and the face key point detection result replaces the optical flow of the corresponding pixels in the output result of the optical flow estimation network, and at the same time, the optical flow confidence of the corresponding pixels in the output result of the optical flow confidence estimation network is improved, so that the non-rigid registration process is more robust, thereby stabilizing the training process.

[0019] Furthermore, a differentiable 3D point cloud non-rigid registration is performed based on the optical flow, optical flow confidence, source frame point cloud, target frame point cloud, and source frame deformation map. Three energy functions are constructed to fit the 2D correspondence provided by the optical flow, fit the corresponding depth correspondence, and constrain the smoothness of the deformation of the deformation map, respectively. The three energy functions are weighted combined and optimized to obtain the deformation from the source frame to the target frame, and the deformation is represented by a set of rotations and translations on the deformation map.

[0020] Furthermore, the loss function includes four items: alignment loss acting on non-rigid registration results, photometric loss acting on the output of the optical flow estimation network, face parsing consistency loss acting on the output of the optical flow estimation network, and confidence regularization loss acting on the output of the optical flow confidence estimation network; the four items are weighted as the total loss function.

[0021] Furthermore, the alignment loss can directly measure the 3D alignment effect, such as chamfer distance loss; or the aligned source frame point cloud and the original target frame point cloud can be rendered onto a 2D plane first, and then the difference between the 2D renderings can be measured.

[0022] Furthermore, the formula for the face parsing consistency loss is as follows:

[0023]

[0024] Where N ly Represents the number of all optical flows including the rough optical flow output by the optical flow estimation network. In PWC-Net based on the pyramid structure, each layer of the pyramid outputs an optical flow, and the number of optical flows is the number of pyramid layers. In RAFT based on the cyclic network structure, each cycle outputs an optical flow, and the number of optical flows is the number of cycles. N cls represents the number of segmentation elements of the face analysis. The segmentation elements are usually: background area, facial skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, mouth, left ear, right ear, so N cls =12;m 1j Represents the heat map of the jth segmentation element in the face parsing result of the source frame face image I1, Represents the heat map of the j-th segmentation element in the face parsing result of the target frame face image I2 after alignment with the i-th optical flow output by the optical flow estimation network.

[0025] Furthermore, the formula of the confidence regularization loss is as follows:

[0026]

[0027] Where W bw Represents the confidence of the reverse optical flow from the target frame face image I2 to the source frame face image I1 predicted by the optical flow confidence estimation network; Represents the set of occluded pixels (the pixels whose corresponding points in the target frame calculated according to the reverse optical flow are outside the face of the source frame are defined as occluded pixels); For collection The complement of .

[0028] According to the second aspect of this specification, a facial optical flow estimation device based on neural non-rigid registration is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and wherein when the processor executes the executable code, it is used to implement the facial optical flow estimation method based on neural non-rigid registration as described in the first aspect.

[0029] The beneficial effects of the present invention are:

[0030] 1. The facial optical flow estimation method based on neural non-rigid registration does not require labeled facial optical flow data, so it can be trained using easily collected facial RGB-D video sequences.

[0031] 2. After training, input two frames of RGB images into the trained optical flow estimation network to estimate the optical flow from the source frame source face RGB image to the target frame target face RGB image.

[0032] 3. Use differentiable non-rigid registration to assist the self-supervised learning of the optical flow estimation network and the optical flow confidence estimation network, providing depth and face structure priors so that the generated optical flow conforms to the motion properties of the face.

[0033] 4. Introducing three types of face priors: face analysis results, 3D face reconstruction results, and face key point detection results, to stabilize the network training process and improve the accuracy of optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0035] Figure 1 is a flowchart of a method for estimating face optical flow based on neural non-rigid registration provided by an exemplary embodiment;

[0036] Figure 2 It is a schematic diagram of the working process of a method for estimating face optical flow based on neural non-rigid registration provided by an exemplary embodiment;

[0037] Figure 3 is a structural diagram of an optical flow confidence estimation network provided by an exemplary embodiment;

[0038] Figure 4 It is a structural diagram of a facial optical flow estimation device based on neural non-rigid registration provided by an exemplary embodiment. DETAILED DESCRIPTION

[0039] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0040] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0041] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0042] In the field of face optical flow estimation, traditional optimization methods have good effects, but they require strict control of the shooting environment and are slow to solve for high-resolution images; network models based on supervised learning require large-scale annotated data sets, which are often difficult to obtain, and networks trained on synthetic data sets are difficult to generalize to real data sets. To this end, an embodiment of the present invention provides a method for estimating face optical flow based on neural non-rigid registration, such as Figure 1 , 2 As shown, it mainly includes the following steps:

[0043] Step 1: Construct a training data set. Use iPhone X to collect 948 RGB-D face video sequences with a resolution of 640*480. Randomly select 50,000 frame pairs from the 948 RGB-D face video sequences, perform face analysis, 3D face reconstruction, and face key point detection on each frame, and construct a deformation map for the first frame of each frame pair to form a training data set; the face analysis results are used to segment the face foreground and background and provide face semantic priors; the 3D face reconstruction results and face key point detection results are used to optimize the input of the non-rigid registration process and stabilize network training; the deformation map is used to solve the non-rigid registration.

[0044] Specifically, the segmentation elements of face analysis are: background area, facial skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, mouth, left ear, and right ear.

[0045] 3D face reconstruction adopts a 3D face reconstruction method based on 3DMM. The 3D faces reconstructed from different face images have the same topological structure, that is, they have 34,650 vertices and the same adjacency relationship, and each vertex has its own definite semantics.

[0046] The facial key point detection uses a conventional 64 key point detector.

[0047] Step 2: construct an optical flow estimation network and an optical flow confidence estimation network, use the 3D face reconstruction results and the face key point detection results to optimize the outputs of the optical flow estimation network and the optical flow confidence estimation network, use the optimized outputs together for differentiable 3D point cloud non-rigid registration, design a loss function, and reversely optimize the network parameters of the two networks by optimizing the loss function on the training data set.

[0048] In one embodiment, the optical flow estimation network uses a pyramid-structured optical flow learning network PWC-Net, which inputs the face RGB images of the previous and next frames with the background filtered out according to the face analysis results, and outputs the optical flow and optical flow features from the source frame source to the target frame target. The feature dimension of the optical flow feature is 565 dimensions and the resolution is 160*120. The optical flow is added to the pixel coordinate tensor to obtain the pixel correspondence between the previous and next frames.

[0049] In one embodiment, the optical flow confidence estimation network is a convolutional neural network model, including an input layer, a convolutional layer, a nonlinear layer and an output layer. Figure 3 This is an example of the structure of the optical flow confidence estimation network, where the numbers in brackets are the parameters of the convolution or deconvolution, for example Figure 3 The first (32,4,2,1) in (a) indicates a deconvolution with an output dimension of 32, a deconvolution kernel size of 4*4, a stride of 2, and a padding parameter of 1. Figure 3 As shown in (a), first, two deconvolution layers are used to upsample the optical flow features to a resolution of 640*480. Then, the target frame target image is aligned to the source frame source image through the optical flow estimated by the optical flow estimation network. The source frame source image, the aligned target frame target image, and the upsampled optical flow features are concatenated in the feature dimension to obtain a new feature as the input of the main network for optical flow confidence estimation. Figure 3 As shown in (b), the main network structure of optical flow confidence estimation is: convolution layer -> BatchNorm layer -> non-linear layer (ReLU) -> residual block -> residual block -> residual block -> convolution layer -> non-linear layer (Sigmoid). Figure 3 As shown in (c), the residual block structure contains two convolutional layers, two BatchNorm layers and two non-linear layers (ReLU).

[0050] In one embodiment, the output process of optimizing the optical flow estimation network and the optical flow confidence estimation network is specifically as follows: first, 34 key points close to the central axis of the face are selected from the 64 facial key points, and the corresponding 34 key points are respectively extracted from the facial key point detection results of the source frame and the target frame according to the semantic correspondence, forming a set of initial correspondences from the source frame source to the target frame target. Next, 2009 vertices are uniformly sampled from the vertices of the 3D face reconstruction result of the source frame, and the corresponding 2009 sampled vertices are taken from the vertices of the 3D face reconstruction result of the target frame according to the semantic correspondence. The sampled vertices of the source frame and the sampled vertices of the target frame are projected to the 2D plane respectively, and the points that are not visible in the projection are deleted to obtain another set of initial correspondences from the source frame source to the target frame target. Finally, the initial correspondences of the above two sets of related pixels of the source frame and the target frame are used to replace the optical flow of the corresponding pixels in the output result of the optical flow estimation network, and the optical flow confidence of the corresponding pixels in the output result of the optical flow confidence estimation network is increased to 1, so that the non-rigid registration process is more robust, thereby stabilizing the training process.

[0051] Differentiable 3D point cloud non-rigid registration can align the source frame point cloud to the target frame point cloud, input the above optimized optical flow and optical flow confidence, source frame point cloud, target frame point cloud, source frame deformation map, and output the deformation from the source frame point cloud to the target frame point cloud, where the deformation is represented by a set of rotations and translations on the source frame deformation map, where rotations are represented by axis angles, and the rotation and translation at each point are a 3D vector. The energy function of non-rigid registration contains three items, which are used to fit the 2D correspondence provided by the optical flow, fit the corresponding depth correspondence, and constrain the smoothness of the deformation of the deformation map.

[0052] In one embodiment, the loss function includes four items, which are used together to update the network parameters, namely:

[0053] 1. Alignment loss: Align the source frame point cloud The original target frame point cloud P2 is rendered onto the 2D image plane to measure the difference between the 2D renderings.

[0054]

[0055] Where D(·) and C(·) represent the depth map and color map obtained by rendering, respectively. is the aligned source frame point cloud, P2 is the original target frame point cloud, and |·| represents a norm.

[0056] 2. Photometric loss: The optical flow from the source frame to the target frame can align the target frame image I2 to the source frame image I1. The photometric loss measures the aligned target frame image. The difference with the original source frame picture I1.

[0057]

[0058] Where N ly Represents the number of all optical flows including the rough optical flow output by the optical flow estimation network. In this embodiment, the optical flow estimation network adopts PWC-Net based on the pyramid structure. Each layer of the pyramid outputs an optical flow. The number of optical flows is the number of pyramid layers. In this embodiment, the pyramid structure has 6 layers, so N ly =6; I1 represents the source frame face picture, Represents the target frame image after optical flow alignment according to the output of the i-th layer of the optical flow estimation network PWC-Net pyramid structure.

[0059] 3. Face parsing consistency loss: The optical flow from the source frame to the target frame can align the face parsing result of the target frame image to the face parsing result of the source frame image. The face parsing consistency loss measures the face parsing result of the aligned target frame image. The difference between the face parsing result m1 and the original source frame image.

[0060]

[0061] Where N ly Represents the number of all optical flows including the rough optical flow output by the optical flow estimation network. This embodiment uses PWCNet based on a 6-layer pyramid structure, so N ly =6; N cls Represents the number of segmentation elements for face analysis, so N cls =12,m 1j Represents the heat map of the jth segmentation element in the face parsing result of the source frame face image I1, Represents the heat map of the j-th segmentation element in the face parsing result of the target frame face image I2 after the optical flow is aligned according to the output of the i-th layer of the optical flow estimation network PWC-Net pyramid structure.

[0062] 4. Confidence Regularization Loss:

[0063]

[0064] Where W bw Represents the confidence of the reverse optical flow from the target frame face image I2 to the source frame face image I1 predicted by the optical flow confidence estimation network. Represents a set of occluded pixels (the pixels whose corresponding points in the target frame calculated according to the reverse optical flow are outside the face of the source frame are defined as occluded pixels). For collection The complement of .

[0065] Finally, the loss function of the training phase is:

[0066] L=λ1L Alig +λ2L P +λ3L Label +λ4L Reg

[0067] Among them, λ1, λ2, λ3, and λ4 are weight parameters. In this embodiment, λ1=1, λ2=5, λ3=1, and λ4=0.1 are set.

[0068] In one embodiment, the model can be trained based on the PyTorch deep learning framework, and multiple groups of data (for example, 4 groups) are read in each time for unsupervised learning. The training ends after multiple training cycles (for example, 50,000 cycles).

[0069] Step 3: After training, input the source frame source RGB face image and the target frame target RGB face image into the trained optical flow estimation network, and output the optical flow from the source frame source to the target frame target, that is, the movement of each pixel.

[0070] By estimating the optical flow of any two frames in the video, we can assist in achieving tasks such as 3D face reconstruction and facial expression recognition. For example, by inputting a face video and the optical flow between video frames, we can fit the face motion trajectory provided by the optical flow between frames, jointly optimize the 3D face shape and camera extrinsics, and finally obtain the 3D face shape, which is the 3D face reconstruction result.

[0071] Compared with the traditional face optical flow estimation method, the above solution of the embodiment of the present invention has the following advantages:

[0072] 1) No large amount of annotated face optical flow datasets are required for training. Only a series of face RGB-D video sequences are needed to train the face optical flow estimation network model.

[0073] 2) Through differentiable non-rigid registration, the depth and facial structure information are better utilized to assist optical flow estimation, which improves the network's robustness to optical flow estimation of faces with large motion.

[0074] 3) The sparse initial correspondence between frames provided by the 3D face reconstruction results and the facial key point detection results is used to replace the value of the optical flow corresponding to the pixel position output by the optical flow estimation network, so as to stabilize the network training process and improve the accuracy of facial optical flow estimation.

[0075] 4) Use face parsing prior to provide face semantic prior to further improve the accuracy of face optical flow estimation.

[0076] Corresponding to the aforementioned embodiment of the method for estimating facial optical flow based on neural non-rigid registration, the present invention also provides an embodiment of a device for estimating facial optical flow based on neural non-rigid registration.

[0077] See also Figure 4 A facial optical flow estimation device based on neural non-rigid registration is provided in an embodiment of the present invention, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the facial optical flow estimation method based on neural non-rigid registration in the above embodiment.

[0078] The embodiment of the facial optical flow estimation device based on neural non-rigid registration of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability where the face optical flow estimation device based on neural non-rigid registration of the present invention is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0079] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0080] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0081] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for estimating facial optical flow based on neural non-rigid registration in the above embodiment is implemented.

[0082] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0086] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0087] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for estimating face optical flow based on neural non-rigid registration, characterized in that: include: Construct a training data set: collect RGB-D face video sequences, select several frame pairs, perform face analysis, 3D face reconstruction, and face key point detection on each frame, and construct a deformation map for non-rigid registration for the first frame of each frame pair to form a training data set; Construct an optical flow estimation network: input the two frames of face RGB images, and output the optical flow and optical flow features from the source frame to the target frame; Construct an optical flow confidence estimation network: input the two frames of face RGB images, the optical flow from the source frame to the target frame and the optical flow features output by the optical flow estimation network, and output the confidence of the optical flow pixel by pixel; Using differentiable 3D point cloud non-rigid registration, a loss function is designed, and the network parameters of the two networks are reversely optimized by optimizing the loss function on the training data set; wherein, the 3D point cloud non-rigid registration is to construct three energy functions according to the optical flow, the optical flow confidence, the source frame point cloud, the target frame point cloud, and the source frame deformation map, respectively fitting the 2D correspondence provided by the optical flow, fitting the corresponding depth correspondence, and constraining the smoothness of the deformation of the deformation map, and performing a weighted combination of the three energy functions and optimizing the solution to obtain the deformation from the source frame to the target frame, and the deformation is represented by a rotation and translation set on the deformation map; The loss function includes four items: alignment loss acting on non-rigid registration results, photometric loss acting on the output of the optical flow estimation network, face parsing consistency loss acting on the output of the optical flow estimation network, and confidence regularization loss acting on the output of the optical flow confidence estimation network; the four items are weighted as the total loss function; the formula of the confidence regularization loss is as follows: Where W bw represents the confidence of the reverse optical flow from the target frame face image I2 to the source frame face image I1 predicted by the optical flow confidence estimation network, represents the set of occluded pixels, For collection The complement of The source frame face RGB image and the target frame face RGB image are input into the trained optical flow estimation network to obtain the optical flow from the source frame to the target frame, that is, the movement of each pixel.

2. According to claim 1, a method for estimating face optical flow based on neural non-rigid registration is characterized in that: The face analysis specifically includes: using the face analysis technology based on neural network to obtain a segmented image that conforms to the corresponding semantics, and the segmentation elements are: background area, facial skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, mouth, left ear, and right ear.

3. According to claim 1, a method for estimating face optical flow based on neural non-rigid registration is characterized in that: The optical flow estimation network adopts a pyramid-structured optical flow learning network PWC-Net or a cyclic network-structured optical flow learning network RAFT.

4. According to claim 1, a method for estimating face optical flow based on neural non-rigid registration is characterized in that: The optical flow confidence estimation network is a convolutional neural network model, including an input layer, a convolutional layer, a nonlinear layer and an output layer.

5. According to claim 1, a method for estimating face optical flow based on neural non-rigid registration is characterized in that: The input of non-rigid registration is optimized, specifically: the vertices of the 3D face reconstruction result are sampled, and the initial correspondence between the source frame and the target frame provided by the sampling result and the face key point detection result is used to replace the optical flow of the corresponding pixels in the output result of the optical flow estimation network, while improving the optical flow confidence of the corresponding pixels in the output result of the optical flow confidence estimation network.

6. The method for estimating face optical flow based on neural non-rigid registration according to claim 1, characterized in that: The alignment loss directly measures the 3D alignment effect, or first renders the aligned source frame point cloud and the original target frame point cloud onto a 2D plane, and then measures the difference between the 2D renderings. The formula for the face parsing consistency loss is as follows: Where N ly Represents the number of all optical flows output by the optical flow estimation network, including the rough optical flow. N cls represents the number of segmentation elements of the face analysis, m 1j Represents the heat map of the jth segmentation element in the face parsing result of the source frame face image I1, Represents the heat map of the j-th segmentation element in the face parsing result of the target frame face image I2 after alignment with the i-th optical flow output by the optical flow estimation network.

7. A face optical flow estimation device based on neural non-rigid registration, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the face optical flow estimation method based on neural non-rigid registration as described in any one of claims 1 to 6.

Images